Dataloaders: PROBA2

This example shows how to preprocess PROBA2 data to obtain ML-ready data using the ITI tool.

import glob

from itipy.evaluation.util import *
from itipy.data.editor import solo_norm
from itipy.translate import *
from itipy.data.dataset import get_intersecting_files, SWAPDataset
from sunpy.map import Map

base_path = os.getcwd()

As the first step, we need to download the data. We make use of our publicly available dataset which allows the users to play around with a subset of the data available without downloading the entire database.

download_gcp_bucket('iti-dataset', base_path+'/iti-testset/')

Failed to download configs/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/configs/'
Downloaded configs/proba2_to_sdo_colab.yaml to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/configs/proba2_to_sdo_colab.yaml.
Failed to download models/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/'
Downloaded models/aia_to_hri_v0_1.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/aia_to_hri_v0_1.pt.
Downloaded models/fsi_to_aia_v0_3.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/fsi_to_aia_v0_3.pt.
Downloaded models/swap_to_aia_v0_4.pt to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/models/swap_to_aia_v0_4.pt.
Failed to download proba2/174/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/'
Downloaded proba2/174/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-11T00:00:00.fits.
Downloaded proba2/174/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-11T12:00:00.fits.
Downloaded proba2/174/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-14T00:00:00.fits.
Downloaded proba2/174/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-15T00:00:00.fits.
Downloaded proba2/174/2021-11-16T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/proba2/174/2021-11-16T00:00:00.fits.
Failed to download sdo/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/'
Failed to download sdo/171/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/'
Downloaded sdo/171/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-11T00:00:00.fits.
Downloaded sdo/171/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-11T12:00:00.fits.
Downloaded sdo/171/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-14T00:00:00.fits.
Downloaded sdo/171/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-15T00:00:00.fits.
Downloaded sdo/171/2021-11-16T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/171/2021-11-16T00:00:00.fits.
Failed to download sdo/304/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/'
Downloaded sdo/304/2021-11-11T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-11T00:00:00.fits.
Downloaded sdo/304/2021-11-11T12:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-11T12:00:00.fits.
Downloaded sdo/304/2021-11-14T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-14T00:00:00.fits.
Downloaded sdo/304/2021-11-15T00:00:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-15T00:00:00.fits.
Downloaded sdo/304/2021-11-16T00:38:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/sdo/304/2021-11-16T00:38:00.fits.
Downloaded soho/171/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/171/2007-12-13T07:19:35.fits.
Downloaded soho/195/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/195/2007-12-13T07:19:35.fits.
Downloaded soho/284/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/284/2007-12-13T07:19:35.fits.
Downloaded soho/304/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/304/2007-12-13T07:19:35.fits.
Downloaded soho/mag/2007-12-13T07:19:35.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/soho/mag/2007-12-13T07:19:35.fits.
Downloaded solo/.DS_Store to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/.DS_Store.
Downloaded solo/eui-fsi174-image/2021-11-11T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-11T00:00:45.fits.
Downloaded solo/eui-fsi174-image/2021-11-11T12:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-11T12:00:45.fits.
Downloaded solo/eui-fsi174-image/2021-11-14T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-14T00:00:45.fits.
Downloaded solo/eui-fsi174-image/2021-11-15T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-15T00:00:45.fits.
Downloaded solo/eui-fsi174-image/2021-11-16T00:30:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi174-image/2021-11-16T00:30:45.fits.
Failed to download solo/eui-fsi304-image/ due to exception: [Errno 21] Is a directory: '/home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/'
Downloaded solo/eui-fsi304-image/2021-11-11T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-11T00:00:45.fits.
Downloaded solo/eui-fsi304-image/2021-11-11T12:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-11T12:00:45.fits.
Downloaded solo/eui-fsi304-image/2021-11-14T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-14T00:00:45.fits.
Downloaded solo/eui-fsi304-image/2021-11-15T00:00:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-15T00:00:45.fits.
Downloaded solo/eui-fsi304-image/2021-11-16T00:30:45.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-fsi304-image/2021-11-16T00:30:45.fits.
Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150905235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150905235_V04.fits.
Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150910235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150910235_V04.fits.
Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150915235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150915235_V04.fits.
Downloaded solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150920235_V04.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/solo/eui-hrieuv174-image/solo_L2_eui-hrieuv174-image_20210220T150920235_V04.fits.
Downloaded stereo/171/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/171/2016-12-28T00:14:00.fits.
Downloaded stereo/195/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/195/2016-12-28T00:14:00.fits.
Downloaded stereo/284/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/284/2016-12-28T00:14:00.fits.
Downloaded stereo/304/2016-12-28T00:14:00.fits to /home/docs/checkouts/readthedocs.org/user_builds/iti-documentation/checkouts/latest/examples_docs/dataloaders/iti-testset/stereo/304/2016-12-28T00:14:00.fits.

Glob the downloaded files and sort them by date.

swap_files = sorted(glob.glob(base_path+'/iti-testset/proba2/*/*.fits', recursive=True))

To preprocess the data, we use Editor classes. These classes allow to apply different operations on the data, such as normalization, cropping etc. The Editor classes are stacked and applied sequentially to the data.

For PROBA2/SWAP we:

• load the .fits files as SunPy maps

• crop the observations to 1.1 solar radii

• transform the maps to data arrays

• normalize the data to an interval between [-1, 1]

• reshape to channel first format [channel, height, width]

swap_dataset = SWAPDataset(swap_files)

We can now compare the original data with the preprocessed ML-ready ITI data. Here we load the aia files as SunPy maps and visualize them with the ITI data.

swap_maps = [Map(f) for f in swap_files]


fig, axs = plt.subplots(1, 2, subplot_kw={'projection': swap_maps[0]}, figsize=(20, 10), dpi=100)
swap_maps[0].plot(axes=axs[0])
axs[1].imshow(swap_dataset[0][0], cmap='sdoaia171', vmin=-1, vmax=1, origin='lower')
axs[0].set_axis_off()
axs[1].set_axis_off()
axs[0].set_title('Original SWAP 174 Å', fontsize=50)
axs[1].set_title('ITI 174 Å', fontsize=50)
plt.show()

INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]
INFO: Missing metadata for solar radius: assuming the standard radius of the photosphere. [sunpy.map.mapbase]