"After specifying an input image (here, Fe bcc [100] alongside the pixel/Angstrom relation), the following code can be used to analyze it via AI-STEM, employing a pretrained model (which is also used in the [the AI-STEM paper](https://doi.org/10.48550/arXiv.2303.12702))."
"***Note 1***: The neural-network classifier is a *Bayesian* neural network, i.e., its predictions are not deterministic but rather probabilistic - modelling the uncertainty in the predictions, which may arise due to experimental noise and/or insufficiency of the model parameters. In particular, we employ [Monte Carlo dropout](https://proceedings.mlr.press/v48/gal16.html?trk=public_post_comment-text) to obtain an uncertainty estimate. In practice, several forward passes (here T=100) are calculated to estimate the uncertainty in the predictions (referred to as 'Performing forward pass n/100' in the cell below). In particular, this suffices to identify the expected bulk symmetry and detect the interface as regions of high uncertainty (as quantified by mutual information). We refer to [the AI-STEM paper](https://doi.org/10.48550/arXiv.2303.12702) for more details (in particular the section 'The Bayesian classification model' and Supplementary Figure S4 for more details on how to choose the number of forward passes).\n",
"\n",
"***Note 2***: The model is trained on a specific pixel/angstrom relation. Specifically, the model is trained to classify local windows of size 12 Angstrom, where 1 pixel corresponds to 0.12 angstrom. Thus a window size of 12 Angstrom corresponds to 100 pixels in the simulation settings that we employed for creating the training set. If a different resolution is employed, we recommend to adapt the window size (and this is also done in the above code): given the pixel-to-Angstrom relation, calculate how much pixels correspond to 12 Angstrom and use this as window size. Alternatively, you may rescale the whole image (up/downsampling, e.g., via cv2.resize) such that the resolutions of your input image match the training resolution and then simply use a 100 pixels window size. "
"The above calculations provide a quickstart, hiding the individual steps performed in the function 'predict'in from ai4stem.utils.utils_prediction. More detailed explanations are provided in the following."
]
},
{
"cell_type": "markdown",
"id": "f6acc9fa",
"metadata": {},
"source": [
"# Step-by-step explanations"
]
},
{
"cell_type": "markdown",
"id": "6b6793cb",
"metadata": {},
"source": [
"First we load the image:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6c93d963",
"metadata": {},
"outputs": [],
"source": [
"input_image = load_example_image()\n",
"image_name = 'Fe_bcc'\n",
"plt.imshow(input_image, cmap='gray')\n",
"plt.show()"
]
},
{
"cell_type": "markdown",
"id": "826aebaa",
"metadata": {},
"source": [
"Next, we need the pixel/Anstrom relation. Moreover, the window size and stride employed in the AI-STEM algorithm has to be specified - here the stride is set to a rather coarse value, while more detailed resolution of structural transitions across defects (such as interfaces) may be achieved by decreasing the stride:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e33deed8",
"metadata": {
"ExecuteTime": {
"end_time": "2023-02-15T09:12:03.793233Z",
"start_time": "2023-02-15T09:12:03.775703Z"
}
},
"outputs": [],
"source": [
"pixel_to_angstrom = 0.1245\n",
"window_size = 12.\n",
"stride_size = [36, 36]"
]
},
{
"cell_type": "markdown",
"id": "f114b1f2",
"metadata": {},
"source": [
"As remarked in the quickstart section, the model is trained on a specific pixel/angstrom relation and window size, so we need to take this into account - where here we do not rescale the image but adapt the window size such that a 12 Angstrom size is obtained:"
"for idx, local_image in enumerate(sliced_images[:selection_size]):\n",
" axs[idx].imshow(local_image, cmap='gray')"
]
},
{
"cell_type": "markdown",
"id": "55aeef16",
"metadata": {},
"source": [
"Next, we calculate the FFT-HAADF descriptor for each of the local images, where one particular setting, the application of a threshold to the calculated FFT is discussed:"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "2122b419",
"metadata": {},
"outputs": [],
"source": [
"# FFT HAADF descriptor settings\n",
"# Threshold parameter; given FFT spectrum normalized\n",
"# to [0, 1], cut off at 0.1 to reduce low-frequency \n",
"# contributions; default is is to use this setting.\n",
"In the function 'predict_with_uncertainty', other uncertainty quantifiers are calculated as well, while in this work, we explored the use of mutual information.\n",
"\n",
"Predictions and uncertainty estimates are not yet in the right shape, they are simply a 1D array and thus we reshape:"
After specifying an input image (here, Fe bcc [100] alongside the pixel/Angstrom relation), the following code can be used to analyze it via AI-STEM, employing a pretrained model (which is also used in the [the AI-STEM paper](https://doi.org/10.48550/arXiv.2303.12702)).
***Note 1***: The neural-network classifier is a *Bayesian* neural network, i.e., its predictions are not deterministic but rather probabilistic - modelling the uncertainty in the predictions, which may arise due to experimental noise and/or insufficiency of the model parameters. In particular, we employ [Monte Carlo dropout](https://proceedings.mlr.press/v48/gal16.html?trk=public_post_comment-text) to obtain an uncertainty estimate. In practice, several forward passes (here T=100) are calculated to estimate the uncertainty in the predictions (referred to as 'Performing forward pass n/100' in the cell below). In particular, this suffices to identify the expected bulk symmetry and detect the interface as regions of high uncertainty (as quantified by mutual information). We refer to [the AI-STEM paper](https://doi.org/10.48550/arXiv.2303.12702) for more details (in particular the section 'The Bayesian classification model' and Supplementary Figure S4 for more details on how to choose the number of forward passes).
***Note 2***: The model is trained on a specific pixel/angstrom relation. Specifically, the model is trained to classify local windows of size 12 Angstrom, where 1 pixel corresponds to 0.12 angstrom. Thus a window size of 12 Angstrom corresponds to 100 pixels in the simulation settings that we employed for creating the training set. If a different resolution is employed, we recommend to adapt the window size (and this is also done in the above code): given the pixel-to-Angstrom relation, calculate how much pixels correspond to 12 Angstrom and use this as window size. Alternatively, you may rescale the whole image (up/downsampling, e.g., via cv2.resize) such that the resolutions of your input image match the training resolution and then simply use a 100 pixels window size.
The above calculations provide a quickstart, hiding the individual steps performed in the function 'predict'in from ai4stem.utils.utils_prediction. More detailed explanations are provided in the following.
%% Cell type:markdown id:f6acc9fa tags:
# Step-by-step explanations
%% Cell type:markdown id:6b6793cb tags:
First we load the image:
%% Cell type:code id:6c93d963 tags:
``` python
input_image=load_example_image()
image_name='Fe_bcc'
plt.imshow(input_image,cmap='gray')
plt.show()
```
%% Cell type:markdown id:826aebaa tags:
Next, we need the pixel/Anstrom relation. Moreover, the window size and stride employed in the AI-STEM algorithm has to be specified - here the stride is set to a rather coarse value, while more detailed resolution of structural transitions across defects (such as interfaces) may be achieved by decreasing the stride:
%% Cell type:code id:e33deed8 tags:
``` python
pixel_to_angstrom=0.1245
window_size=12.
stride_size=[36,36]
```
%% Cell type:markdown id:f114b1f2 tags:
As remarked in the quickstart section, the model is trained on a specific pixel/angstrom relation and window size, so we need to take this into account - where here we do not rescale the image but adapt the window size such that a 12 Angstrom size is obtained:
Next, we calculate the FFT-HAADF descriptor for each of the local images, where one particular setting, the application of a threshold to the calculated FFT is discussed:
%% Cell type:code id:2122b419 tags:
``` python
# FFT HAADF descriptor settings
# Threshold parameter; given FFT spectrum normalized
# to [0, 1], cut off at 0.1 to reduce low-frequency
# contributions; default is is to use this setting.
thresholding=True# very important
fft_descriptors=[]
foriminsliced_images:
fft_desc=calc_fft(im,thresholding=thresholding)
fft_descriptors.append(fft_desc)
```
%% Cell type:markdown id:dc9b3c2d tags:
We may again visualize local fragments and their corresponding 64x64 FFT-HAADF descriptor:
In the function 'predict_with_uncertainty', other uncertainty quantifiers are calculated as well, while in this work, we explored the use of mutual information.
Predictions and uncertainty estimates are not yet in the right shape, they are simply a 1D array and thus we reshape:
