.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "generated/gallery/dataloaders/loader_proba2.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_generated_gallery_dataloaders_loader_proba2.py: =================== Dataloaders: PROBA2 =================== This example shows how to preprocess PROBA2 data to obtain ML-ready data using the ITI tool. .. GENERATED FROM PYTHON SOURCE LINES 7-17 .. code-block:: Python import glob from itipy.evaluation.util import * from itipy.data.editor import solo_norm from itipy.translate import * from itipy.data.dataset import get_intersecting_files, SWAPDataset from sunpy.map import Map base_path = os.getcwd() .. GENERATED FROM PYTHON SOURCE LINES 18-19 As the first step, we need to download the data. We make use of our publicly available dataset which allows the users to play around with a subset of the data available without downloading the entire database. .. GENERATED FROM PYTHON SOURCE LINES 19-21 .. code-block:: Python download_gcp_bucket('iti-dataset', base_path+'/iti-testset/') .. rst-class:: sphx-glr-script-out .. code-block:: none Failed to download configs/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/configs/' Downloaded configs/proba2_to_sdo_colab.yaml to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/configs/proba2_to_sdo_colab.yaml. Failed to download models/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/' Downloaded models/aia_to_hri_v0_1.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/aia_to_hri_v0_1.pt. Downloaded models/fsi_to_aia_v0_3.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/fsi_to_aia_v0_3.pt. Downloaded models/swap_to_aia_v0_4.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/swap_to_aia_v0_4.pt. Failed to download proba2/174/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/' Downloaded proba2/174/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-11T00:00:00.fits. Downloaded proba2/174/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-11T12:00:00.fits. Downloaded proba2/174/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-14T00:00:00.fits. Downloaded proba2/174/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-15T00:00:00.fits. Downloaded proba2/174/2021-11-16T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-16T00:00:00.fits. Failed to download sdo/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/' Failed to download sdo/171/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/' Downloaded sdo/171/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-11T00:00:00.fits. Downloaded sdo/171/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-11T12:00:00.fits. Downloaded sdo/171/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-14T00:00:00.fits. Downloaded sdo/171/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-15T00:00:00.fits. Downloaded sdo/171/2021-11-16T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-16T00:00:00.fits. Failed to download sdo/304/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/' Downloaded sdo/304/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-11T00:00:00.fits. Downloaded sdo/304/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-11T12:00:00.fits. Downloaded sdo/304/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-14T00:00:00.fits. Downloaded sdo/304/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-15T00:00:00.fits. Downloaded sdo/304/2021-11-16T00:38:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-16T00:38:00.fits. Downloaded soho/171/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/171/2007-12-13T07:19:35.fits. Downloaded soho/195/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/195/2007-12-13T07:19:35.fits. Downloaded soho/284/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/284/2007-12-13T07:19:35.fits. Downloaded soho/304/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/304/2007-12-13T07:19:35.fits. Downloaded soho/mag/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/mag/2007-12-13T07:19:35.fits. Downloaded solo/.DS_Store to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/.DS_Store. Downloaded solo/eui-fsi174-image/2021-11-11T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-11T00:00:45.fits. Downloaded solo/eui-fsi174-image/2021-11-11T12:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-11T12:00:45.fits. Downloaded solo/eui-fsi174-image/2021-11-14T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-14T00:00:45.fits. Downloaded solo/eui-fsi174-image/2021-11-15T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-15T00:00:45.fits. Downloaded solo/eui-fsi174-image/2021-11-16T00:30:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-16T00:30:45.fits. Failed to download solo/eui-fsi304-image/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/' Downloaded solo/eui-fsi304-image/2021-11-11T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-11T00:00:45.fits. Downloaded solo/eui-fsi304-image/2021-11-11T12:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-11T12:00:45.fits. Downloaded solo/eui-fsi304-image/2021-11-14T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-14T00:00:45.fits. Downloaded solo/eui-fsi304-image/2021-11-15T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-15T00:00:45.fits. Downloaded solo/eui-fsi304-image/2021-11-16T00:30:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-16T00:30:45.fits. Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150905235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150905235_V04.fits. Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150910235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150910235_V04.fits. Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150915235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150915235_V04.fits. Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150920235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150920235_V04.fits. Downloaded stereo/171/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/171/2016-12-28T00:14:00.fits. Downloaded stereo/195/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/195/2016-12-28T00:14:00.fits. Downloaded stereo/284/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/284/2016-12-28T00:14:00.fits. Downloaded stereo/304/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/304/2016-12-28T00:14:00.fits. .. GENERATED FROM PYTHON SOURCE LINES 22-23 Glob the downloaded files and sort them by date. .. GENERATED FROM PYTHON SOURCE LINES 23-25 .. code-block:: Python swap_files = sorted(glob.glob(base_path+'/iti-testset/proba2/*/*.fits', recursive=True)) .. GENERATED FROM PYTHON SOURCE LINES 26-34 To preprocess the data, we use Editor classes. These classes allow to apply different operations on the data, such as normalization, cropping etc. The Editor classes are stacked and applied sequentially to the data. For PROBA2/SWAP we: - load the `.fits` files as SunPy maps - crop the observations to 1.1 solar radii - transform the maps to data arrays - normalize the data to an interval between [-1, 1] - reshape to channel first format [channel, height, width] .. GENERATED FROM PYTHON SOURCE LINES 34-36 .. code-block:: Python swap_dataset = SWAPDataset(swap_files) .. GENERATED FROM PYTHON SOURCE LINES 37-38 We can now compare the original data with the preprocessed ML-ready ITI data. Here we load the aia files as SunPy maps and visualize them with the ITI data. .. GENERATED FROM PYTHON SOURCE LINES 38-50 .. code-block:: Python swap_maps = [Map(f) for f in swap_files] fig, axs = plt.subplots(1, 2, subplot_kw={'projection': swap_maps[0]}, figsize=(20, 10), dpi=100) swap_maps[0].plot(axes=axs[0]) axs[1].imshow(swap_dataset[0][0], cmap='sdoaia171', vmin=-1, vmax=1, origin='lower') axs[0].set_axis_off() axs[1].set_axis_off() axs[0].set_title('Original SWAP 174 Å', fontsize=50) axs[1].set_title('ITI 174 Å', fontsize=50) plt.show() .. image-sg:: /generated/gallery/dataloaders/images/sphx_glr_loader_proba2_001.png :alt: Original SWAP 174 Å, ITI 174 Å :srcset: /generated/gallery/dataloaders/images/sphx_glr_loader_proba2_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase] .. rst-class:: sphx-glr-timing **Total running time of the script:** (0 minutes 6.901 seconds) .. _sphx_glr_download_generated_gallery_dataloaders_loader_proba2.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: loader_proba2.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: loader_proba2.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: loader_proba2.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_