Preprocessing

The second module after the data download module is the preprocessing module. This module is responsible for the preprocessing of the data to be used for neural network applications.

The preprocessing steps are built on Editor classes which can be stacked together to obtain ML-ready datasets.

Editors

All preprocessing editors are stored in itipy.data.editor. To give you and idea of the available editors, here are the most common ones:

Before we start with the preprocessing, let’s first load the data as a SunPy map.

After loading the data, we can start with the preprocessing. In order to scale all observations to the same size, we use the itipy.data.editor.NormalizeRadiusEditor. This crops the observations 1.1 solar radii and scales them to a fixed resolution. To change the cropped radius to a different value, the padding_factor needs to be adjusted.

The next step is to transform the SunPy map to a numpy array. This is done by the itipy.data.editor.MapToDataEditor.

Depending on the instruments and the activation function of the neural network we need to normalize the data. This is done by the itipy.data.editor.NormalizeEditor.

The normalization depends on the instruments used and can be adjusted accordingly.

The last step is to reshape the data according to the neural network architecture. The itipy.data.editor.ReshapeDataEditor is used for this.

Instrument specific preprocessing

In some cases, we need to preprocess the data according to the instrument used. For example, the AIA instrument needs to be corrected for instrument degradation. This is done by the itipy.data.editor.AIAPrepEditor.

For more information on the individual editors available, please look up the itipy.data.editor module on GitHub.