.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "generated/gallery/dataloaders/loader_solo.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_generated_gallery_dataloaders_loader_solo.py: ========================== Dataloaders: Solar Orbiter ========================== This example shows how to preprocess Solar Orbiter EUI data to obtain ML-ready data using the ITI tool. .. GENERATED FROM PYTHON SOURCE LINES 7-17 .. code-block:: Python import glob from itipy.evaluation.util import * from itipy.data.editor import solo_norm from itipy.translate import * from itipy.data.dataset import get_intersecting_files, EUIDataset, HRIDataset from sunpy.map import Map base_path = os.getcwd() .. GENERATED FROM PYTHON SOURCE LINES 18-19 As the first step, we need to download the data. We make use of our publicly available dataset which allows the users to play around with a subset of the data available without downloading the entire database. .. GENERATED FROM PYTHON SOURCE LINES 19-21 .. code-block:: Python download_gcp_bucket('iti-dataset', base_path+'/iti-testset/') .. rst-class:: sphx-glr-script-out .. code-block:: none Failed to download configs/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/configs/' Downloaded configs/proba2_to_sdo_colab.yaml to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/configs/proba2_to_sdo_colab.yaml. Failed to download models/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/' Downloaded models/aia_to_hri_v0_1.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/aia_to_hri_v0_1.pt. Downloaded models/fsi_to_aia_v0_3.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/fsi_to_aia_v0_3.pt. Downloaded models/swap_to_aia_v0_4.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/swap_to_aia_v0_4.pt. Failed to download proba2/174/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/' Downloaded proba2/174/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-11T00:00:00.fits. Downloaded proba2/174/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-11T12:00:00.fits. Downloaded proba2/174/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-14T00:00:00.fits. Downloaded proba2/174/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-15T00:00:00.fits. Downloaded proba2/174/2021-11-16T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-16T00:00:00.fits. Failed to download sdo/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/' Failed to download sdo/171/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/' Downloaded sdo/171/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-11T00:00:00.fits. Downloaded sdo/171/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-11T12:00:00.fits. Downloaded sdo/171/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-14T00:00:00.fits. Downloaded sdo/171/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-15T00:00:00.fits. Downloaded sdo/171/2021-11-16T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-16T00:00:00.fits. Failed to download sdo/304/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/' Downloaded sdo/304/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-11T00:00:00.fits. Downloaded sdo/304/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-11T12:00:00.fits. Downloaded sdo/304/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-14T00:00:00.fits. Downloaded sdo/304/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-15T00:00:00.fits. Downloaded sdo/304/2021-11-16T00:38:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-16T00:38:00.fits. Downloaded soho/171/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/171/2007-12-13T07:19:35.fits. Downloaded soho/195/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/195/2007-12-13T07:19:35.fits. Downloaded soho/284/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/284/2007-12-13T07:19:35.fits. Downloaded soho/304/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/304/2007-12-13T07:19:35.fits. Downloaded soho/mag/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/mag/2007-12-13T07:19:35.fits. Downloaded solo/.DS_Store to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/.DS_Store. Downloaded solo/eui-fsi174-image/2021-11-11T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-11T00:00:45.fits. Downloaded solo/eui-fsi174-image/2021-11-11T12:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-11T12:00:45.fits. Downloaded solo/eui-fsi174-image/2021-11-14T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-14T00:00:45.fits. Downloaded solo/eui-fsi174-image/2021-11-15T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-15T00:00:45.fits. Downloaded solo/eui-fsi174-image/2021-11-16T00:30:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-16T00:30:45.fits. Failed to download solo/eui-fsi304-image/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/' Downloaded solo/eui-fsi304-image/2021-11-11T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-11T00:00:45.fits. Downloaded solo/eui-fsi304-image/2021-11-11T12:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-11T12:00:45.fits. Downloaded solo/eui-fsi304-image/2021-11-14T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-14T00:00:45.fits. Downloaded solo/eui-fsi304-image/2021-11-15T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-15T00:00:45.fits. Downloaded solo/eui-fsi304-image/2021-11-16T00:30:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-16T00:30:45.fits. Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150905235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150905235_V04.fits. Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150910235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150910235_V04.fits. Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150915235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150915235_V04.fits. Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150920235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150920235_V04.fits. Downloaded stereo/171/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/171/2016-12-28T00:14:00.fits. Downloaded stereo/195/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/195/2016-12-28T00:14:00.fits. Downloaded stereo/284/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/284/2016-12-28T00:14:00.fits. Downloaded stereo/304/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/304/2016-12-28T00:14:00.fits. .. GENERATED FROM PYTHON SOURCE LINES 22-23 The EUI instrument is equipped with the Full Sun Imager (FSI) and the High-resolution Imager (HRI). We can glob the files for each instrument separately. .. GENERATED FROM PYTHON SOURCE LINES 23-26 .. code-block:: Python fsi_files = get_intersecting_files(base_path+'/iti-testset/solo', ['eui-fsi174-image', 'eui-fsi304-image']) hri_files = sorted(glob.glob(base_path+'/iti-testset/solo/eui-hrieuv174-image/*.fits', recursive=True)) .. GENERATED FROM PYTHON SOURCE LINES 27-29 To preprocess the data, we use Editor classes. These classes allow to apply different operations on the data, such as normalization, cropping etc. The Editor classes are stacked and applied sequentially to the data. .. GENERATED FROM PYTHON SOURCE LINES 29-39 .. code-block:: Python # For Solar Orbiter/EUI we: # - load the `.fits` files as SunPy maps # - crop the observations to 1.1 solar radii # - transform the maps to data arrays # - normalize the data to an interval between [-1, 1] # - reshape to channel first format [channel, height, width] fsi_dataset = EUIDataset(fsi_files) hri_dataset = HRIDataset(hri_files) .. GENERATED FROM PYTHON SOURCE LINES 40-41 We can now compare the original data with the preprocessed ML-ready ITI data. Here we load the FSI and HRI files as SunPy maps. .. GENERATED FROM PYTHON SOURCE LINES 41-46 .. code-block:: Python fsi174_maps = [Map(f) for f in fsi_files[0]] fsi304_maps = [Map(f) for f in fsi_files[1]] hri_maps = [Map(f) for f in hri_files] .. GENERATED FROM PYTHON SOURCE LINES 47-48 Plot the FSI observations with the ITI data .. GENERATED FROM PYTHON SOURCE LINES 48-64 .. code-block:: Python fig, axs = plt.subplots(2, 2, subplot_kw={'projection': fsi174_maps[0]}, figsize=(40, 30), dpi=100) fsi174_maps[0].plot(axes=axs[0, 0]) axs[0, 1].imshow(fsi_dataset[0][0], cmap='sdoaia171', vmin=-1, vmax=1, origin='lower') fsi304_maps[0].plot(axes=axs[1, 0]) axs[1, 1].imshow(fsi_dataset[0][1], cmap='sdoaia304', vmin=-1, vmax=1, origin='lower') axs[0, 0].set_axis_off() axs[0, 1].set_axis_off() axs[1, 0].set_axis_off() axs[1, 1].set_axis_off() axs[0, 0].set_title('Original FSI 174 Å', fontsize=50) axs[0, 1].set_title('ITI 174 Å', fontsize=50) axs[1, 0].set_title('Original FSI 304 Å', fontsize=50) axs[1, 1].set_title('ITI 304 Å', fontsize=50) plt.show() .. image-sg:: /generated/gallery/dataloaders/images/sphx_glr_loader_solo_001.png :alt: Original FSI 174 Å, ITI 174 Å, Original FSI 304 Å, ITI 304 Å :srcset: /generated/gallery/dataloaders/images/sphx_glr_loader_solo_001.png :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 65-66 Plot the HRI observations with the ITI data .. GENERATED FROM PYTHON SOURCE LINES 66-74 .. code-block:: Python fig, axs = plt.subplots(1, 2, subplot_kw={'projection': hri_maps[0]}, figsize=(20, 10), dpi=100) hri_maps[0].plot(axes=axs[0]) axs[1].imshow(hri_dataset[0][0], cmap='sdoaia171', vmin=-1, vmax=1, origin='lower') axs[0].set_axis_off() axs[1].set_axis_off() axs[0].set_title('Original HRI 174 Å', fontsize=50) axs[1].set_title('ITI 174 Å', fontsize=50) plt.show() .. image-sg:: /generated/gallery/dataloaders/images/sphx_glr_loader_solo_002.png :alt: Original HRI 174 Å, ITI 174 Å :srcset: /generated/gallery/dataloaders/images/sphx_glr_loader_solo_002.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 55.704 seconds) .. _sphx_glr_download_generated_gallery_dataloaders_loader_solo.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: loader_solo.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: loader_solo.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: loader_solo.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_