About half of all optical observations collected via spaceborne satellitesare affected by haze or clouds. Consequently, cloud coverage affects the remotesensing practitioner's capabilities of a continuous and seamless monitoring ofour planet. This work addresses the challenge of optical satellite imagereconstruction and cloud removal by proposing a novel multi-modal andmulti-temporal data set called SEN12MS-CR-TS. We propose two modelshighlighting the benefits and use cases of SEN12MS-CR-TS: First, a multi-modalmulti-temporal 3D-Convolution Neural Network that predicts a cloud-free imagefrom a sequence of cloudy optical and radar images. Second, asequence-to-sequence translation model that predicts a cloud-free time seriesfrom a cloud-covered time series. Both approaches are evaluated experimentally,with their respective models trained and tested on SEN12MS-CR-TS. The conductedexperiments highlight the contribution of our data set to the remote sensingcommunity as well as the benefits of multi-modal and multi-temporal informationto reconstruct noisy information. Our data set is available athttps://patrickTUM.github.io/cloud_removal