Skip to content
Snippets Groups Projects
Commit 3d29bd9a authored by Markus Kuehbach's avatar Markus Kuehbach
Browse files

Merge branch 'update-leoben-aptyzer' into 'develop'

Bumped APTyzer to newer version

See merge request !136
parents 55bab464 007383c5
No related branches found
No related tags found
1 merge request!136Bumped APTyzer to newer version
Pipeline #168740 passed
%% Cell type:markdown id:88054eb2 tags:
<img src="NOMADOasisLogo.png" alt="Drawing" style="height: 149px;"/>&emsp;&emsp;<img src="FAIRmatNewLogo.png" alt="Drawing" style="height: 149px;"/>
%% Cell type:markdown id:9ecf3020 tags:
# apmtools container cheatsheet
%% Cell type:markdown id:4091c1e2 tags:
The apmtools container offers functionalities to work and perform data analyses<br>
with reconstructed and ranged datasets from atom probe microscopy experiments.<br>
This container includes three tools:
* **APTyzer** by Alexander Reichmann et al. https://github.com/areichm/APTyzer
* **paraprobe-toolbox** by Markus Kühbach et al. https://gitlab.com/paraprobe/paraprobe-toolbox
* **APAV** by Jesse Smith et al. https://gitlab.com/jesseds/apav
Each tool comes shipped with tutorial-style examples.
***
%% Cell type:markdown id:e29f2d4c-199c-4db3-8bab-6a341dc7ea83 tags:
Load a Python library for unpacking ZIP files.
%% Cell type:code id:731941ed-e3b9-4ae1-8372-8211ff900544 tags:
``` python
import zipfile as zp
```
%% Cell type:markdown id:e7afeb96-16a6-4d5a-9986-7aacfc9980f4 tags:
<div class="alert alert-block alert-info">
Having APTyzer, APAV, and paraprobe-toolbox configured in one container comes with the benefit<br>
that one can switch between these tools or create an analysis which couples them together in a<br>
jupyter notebook.
</div>
%% Cell type:markdown id:fd044275-1ce9-4fea-bbe8-9903c3620a18 tags:
<div class="alert alert-block alert-info">
Containers of the Nomad Remote Tools Hub (NORTH) are configured such that they have access to the data in the uploads section of the Oasis instance.<br>
Individual examples which exemplify how to use the tools in the apmtools container may have to be unpacked/decompressed before their first use.
</div>
%% Cell type:markdown id:69238a64 tags:
# Getting started with APTyzer
%% Cell type:markdown id:4aa12615-4734-4802-b6b9-407d6f1797bd tags:
<div class="alert alert-block alert-info">
Computational requirements: Examples with dataset with a few million ions like most used below should be<br>
processable even on a computer with a single core and say at least four GB main memory for docker tasks.
</div>
%% Cell type:markdown id:6e324fa1 tags:
APTyzer is a Python-based tool with a graphical user interface (GUI) which can be used for displaying of a tomographically<br>
reconstructed atom probe dataset thus enabling composition and interfacial excesses analyses. APTyzer can also be used for<br>
preparing the meshing of single interface patches (grain or phase boundaries) via setting control points manually.<br>
These points can be exported and loaded with the paraprobe-toolbox. APTyzer is maintained by Alexander Reichmann,<br>
who is a PhD student with Lorenz Romaner at the Montanuniversität Leoben, Austria.<br>
To use the tool you should navigate into the respective sub-directory.<br>
%% Cell type:markdown id:1aa22278 tags:
### Step 1: Navigate into the respective sub-directory via the explorer panel on the left side
%% Cell type:code id:5fd98af7 tags:
``` python
! cd /home/atom_probe_tools/aptyzer
```
%% Cell type:markdown id:cc1408cc-45ec-40c6-90cb-dcec5f900619 tags:
### Step 2: Download and unpack an example dataset to get started
%% Cell type:code id:ffa2006f tags:
``` python
! cd /home/atom_probe_tools/aptyzer && wget https://www.zenodo.org/record/7908429/files/aut_leoben_leitner.zip
zp.ZipFile("/home/atom_probe_tools/aptyzer/aut_leoben_leitner.zip").extractall(
path="/home/atom_probe_tools/aptyzer", members=None, pwd=None)
```
%% Cell type:markdown id:1d7b6c36-5967-46be-835d-3678c4ef8865 tags:
### Step 3: Start APTyzer by executing the APT_analyzer.ipynb in a new browser tab
%% Cell type:markdown id:65cc8b4d-3f83-49fd-94f8-4309852499de tags:
Open the aptyzer.ipynb via the JupyterLab side bar (on the left side). Clicking through this notebook will start the graphical user interface.
%% Cell type:markdown id:8cea07f1-ccc5-40f4-881a-f9fec8a3f206 tags:
### Step 4: Run the tool using the POS and RRNG with the aut_leoben_leitner example dataset.
%% Cell type:markdown id:051121d5-f318-42a7-81d8-a11349401e4c tags:
There is a detailed tutorial (tutorial.pdf) how to use APTyzer for arbitary datasets. The tool offers export functionalities<br>
for manually selected control points to support building computational geometry models of interfaces.<br>
These control points can be exported to an HDF5 file and this file can be used for example as input for the<br>
paraprobe-nanochem tool from the paraprobe-toolbox.<br>
A respective tutorial how to achieve this is available in the paraprobe-toolbox teaching material<br>
and here specifically the aut_leoben_leitner example. This tutorial will show how to use the<br>
control points and create an automatically meshed model for an interface in the dataset.<br>
***
%% Cell type:markdown id:7bda5aa4 tags:
# Getting started with paraprobe-toolbox
%% Cell type:markdown id:144832aa-b357-406c-84ad-d410d27bd8c0 tags:
<div class="alert alert-block alert-info">
Computational requirements: Examples with dataset with a few million ions like most used below should be processable even on a computer with a<br>
single core and four GB main memory for docker tasks. Having multiple CPU cores can be useful as the tools of the paraprobe-toolbox use<br>
multi-threading for most of the numerical and geometrical analyses.<br>
Making guarantees about the maximum data set sizes (in terms of number of ions) is difficult as it strongly depends on which type of analyses<br>
are performed and how these are parameterized. Noteworthy to mention is that even the largest examples at the time of writing this cheatsheet<br>
which are available in the paraprobe-toolbox were processable with a laptop with 32GB main memory in total. Examples with a few million ions<br>
consumed not more than one to eight GB.<br>
</div>
%% Cell type:markdown id:ec70f493 tags:
The paraprobe-toolbox is a collection of software tools for applying computational geometry tasks on point cloud data<br>
such as tomographic reconstructions of atom probe data to extract and characterize microstructural features.<br>
The tool is developed by <a href="https://arxiv.org/abs/2205.13510">Markus Kühbach et al.</a><br>
The tool is instructed via a jupyter notebook which documents how to chain and script different<br>
analysis steps into a workflow using Python. This can be useful especially for batch processing<br>
on computer clusters.<br>
Tools of the paraprobe-toolbox are chained into computational workflows. Each step uses a different scientific<br>
speciality tool. All these tools have *paraprobe* as a prefix in their executable name.<br>
The tools use CPU parallelization and specific libraries of the computational geometry or other specialists'<br>
communities. The jupyter notebooks enable users to achieve a complete automation of their<br>
data analyses (if this is desired). Internally, each tools keeps track of input and output files via<br>
hashes and time stamps to enable provenance tracking and support repeatable and reproducible research.<br>
All results are openly accessible and documented via so-called <a href="https://fairmat-experimental.github.io/nexus-fairmat-proposal">NeXus application definitions (see the NORTH/apmtools pages).</a><br>
Such workflows can include parameter studies, mesh processing, writing of reports, and creation of plots.<br>
%% Cell type:markdown id:57dd96fe tags:
### Step 1: Enter the paraprobe-toolbox/teaching/example_analyses sub-directory via the explorer panel to the left.
%% Cell type:code id:f6461c64-cbcd-4d0a-8be3-850205382a71 tags:
``` python
# analysis results are stored here:
! ls /home/atom_probe_tools/paraprobe-toolbox/teaching/example_analyses
# measurements are stored here:
! ls /home/atom_probe_tools/paraprobe-toolbox/teaching/example_data
# enter the root/prefix/home directory of paraprobe-toolbox:
! cd /home/atom_probe_tools/paraprobe-toolbox
```
%% Cell type:markdown id:bce1299f-24fb-4ffc-99d3-99aac8921136 tags:
Reconstruction and ranging data (i.e. POS, ePOS, APT, RNG, RRNG, NXS) to be consumed by paraprobe-toolbox<br>
should be stored in *teaching/example_data*. Analyses with relevant jupyter notebooks should be stored<br>
in *teaching/example_analyses*.
The tools use NeXus data schemas and HDF5 files. The specification of these files is documented in the so-called<br>
<a href="https://fairmat-experimental.github.io/nexus-fairmat-proposal">nexus-fairmat-proposal</a> (see specifically the NORTH/apmtools pages).
%% Cell type:markdown id:670cc82a-2ea2-4a48-bded-793b15e2f58f tags:
### Step 2: Unpack the example datasets or use a NeXus/HDF5 file from your uploads section.
%% Cell type:code id:5d4dc19a-6d1d-4e04-ab61-0b40038e8577 tags:
``` python
! cd /home/atom_probe_tools/paraprobe-toolbox/teaching/example_data/usa_portland_wang && tar -xvf usa_portland_wang.tar.gz
! cd /home/atom_probe_tools/paraprobe-toolbox/teaching/example_data/aut_leoben_leitner && tar -xvf aut_leoben_leitner.tar.gz
```
%% Cell type:code id:5c6e219d-1a5d-4417-b157-3745558f6660 tags:
``` python
# additional optional examples are available for testing transcoding of Matlab figure files
! cd /home/atom_probe_tools/paraprobe-toolbox/teaching/example_data/ger_erlangen_felfer && wget https://www.zenodo.org/record/7908429/files/ger_erlangen_felfer_ck10.zip
zp.ZipFile("/home/atom_probe_tools/paraprobe-toolbox/teaching/example_data/ger_erlangen_felfer/ger_erlangen_felfer_ck10.zip").extractall(
path="/home/atom_probe_tools/paraprobe-toolbox/teaching/example_data/ger_erlangen_felfer", members=None, pwd=None)
```
%% Cell type:code id:893e0ef4-90af-4946-8b6c-454f0db2fd89 tags:
``` python
# additional optional examples are available for testing with a very small dataset for efficient debugging
! cd /home/atom_probe_tools/paraprobe-toolbox/teaching/example_data/usa_denton_smith && wget https://www.zenodo.org/record/7908429/files/usa_denton_smith_apav_si.zip
zp.ZipFile("/home/atom_probe_tools/paraprobe-toolbox/teaching/example_data/usa_denton_smith/usa_denton_smith_apav_si.zip").extractall(
path="/home/atom_probe_tools/paraprobe-toolbox/teaching/example_data/usa_denton_smith", members=None, pwd=None)
```
%% Cell type:markdown id:c409e217-e14a-4bba-a07e-e16753288bd8 tags:
Alternatively, NeXus/HDF5 files in your uploads section can also be used. In order to do so, you can move<br>
the respective file(s) into one of the example_data analysis sections inside the container.<br>
This works because the container is configured such that the directory which represents<br>
the uploads section is mounted into the apmtools container and is accessible via config directory.<br>
%% Cell type:code id:22c54c6b-fe0d-4e47-9247-4d5f42761134 tags:
``` python
# ! mv # move the file from the config directory to your desired directory in the teaching/example_data area
```
%% Cell type:markdown id:4fa357bd-7e4a-4c78-afeb-cc4debe34f48 tags:
### Step 3: Start with the unpacked example usa_portland_wang.
%% Cell type:markdown id:8c8865e7-70e9-4309-93c3-5eaf73e2d2e0 tags:
Each example of the paraprobe-toolbox comes with a detailed jupyter notebook which guides through the analysis.<br>
Especially the usa_portland_wang and the aut_leoben_leitner (see connection to APTyzer tool) examples<br>
show the multi-faceted analysis features of the tools in the paraprobe-toolbox.<br>
These examples show how Python and C/C++ applications work together with NeXus, HDF5, and H5Web<br>
to provide tools which can be flexibly be coupled to other open-source atom probe software and<br>
used to complement analyses which have been achieved with commercial software like APSuite/IVAS.<br>
More complex examples with customizable jupyter notebooks are available in the documentation<br>
of the tool https://paraprobe-toolbox.readthedocs.io/en/latest/.<br>
***
%% Cell type:markdown id:80d09ac1-e56d-4923-9ba3-9e869d368a51 tags:
# Getting started with APAV
%% Cell type:markdown id:bfec8daa-59c3-47b9-a0af-8f03afd5c9a6 tags:
<div class="alert alert-block alert-info">
Computational requirements: Examples with dataset with a few million ions like most used below should be processable even<br>
on a computer with a single core and say at least four GB main memory for docker tasks. Having multiple CPU cores can be<br>
useful because APAV uses multi-threading for some costly numerical analyses.
</div>
%% Cell type:markdown id:c10fcef1-e82f-4400-af22-f6df70687faf tags:
APAV (Atom Probe Analysis and Visualization) is a Python package for analysis and visualization of atom probe tomography datasets.<br>
The tool is developed by <a href="https://joss.theoj.org/papers/10.21105/joss.04862">Jesse Smith et al.</a>. Complementary to the design of the paraprobe-toolbox functionalities,<br>
APAV can be chained into workflows via e.g. a jupyter notebook. A particular functional strength and focus of APAV<br>
has been ranging and handling of so-called multi-hit events via a graphical user interface via Python.<br>
APAV has a detailed documentation https://apav.readthedocs.io/en/latest/index.html.
%% Cell type:markdown id:b75f60d5-3151-4986-9222-ddaeb1fa7e5f tags:
<div class="alert alert-block alert-info">
Jesse Smith has also made available example data which can be used for learning the multi-hit event analyses capabilities of APAV.<br>
If these files are not available <a href="https://doi.org/10.5281/zenodo.6794809">via this Zenodo repository (version >9)</a>,<br>
users are requested to inspect the respective file share location from the revision of the APAV JOSS paper<br>
<a href="https://github.com/openjournals/joss-reviews/issues/4862">see the comment from jdasm from November 30, 2022 here</a>.<br>
The respective GBCO dataset relevant here is especially this one R5038_00333-v02.epos.<br>
<a href="https://doi.org/10.1017/S1431927621012794">Scientific details can be found here</a>.
</div>
%% Cell type:markdown id:f88501b0-a4bd-427d-b303-a45a3556ab35 tags:
### Step 1: Enter the apav sub-directory via the explorer panel to the left.
%% Cell type:code id:474644bb-75fd-4514-bab5-8d9d30e46610 tags:
``` python
! ls /home/atom_probe_tools/apav
```
%% Cell type:markdown id:ba59fb6e-2b63-4e95-99f7-75248c2a4f79 tags:
### Step 2: Use available files in community-specific formats from your uploads section/download examples from APAV.
%% Cell type:code id:7c256126-3704-4947-9598-d01707a97323 tags:
``` python
! cd /home/atom_probe_tools/apav && wget https://www.zenodo.org/record/7908429/files/usa_denton_smith_apav_si.zip
zp.ZipFile("/home/atom_probe_tools/apav/usa_denton_smith_apav_si.zip").extractall(
path="/home/atom_probe_tools/apav", members=None, pwd=None)
```
%% Cell type:code id:5d092aaf-663d-4ef3-9ac7-5fc6762d785f tags:
``` python
! cd /home/atom_probe_tools/apav && wget https://www.zenodo.org/record/7908429/files/usa_denton_smith_apav_gbco.zip
zp.ZipFile("/home/atom_probe_tools/apav/usa_denton_smith_apav_gbco.zip").extractall(
path="/home/atom_probe_tools/apav", members=None, pwd=None)
```
%% Cell type:markdown id:f65c621e-05f5-45ca-9a01-4cbd3206bc55 tags:
The APAV documentation details which formats are supported.
%% Cell type:markdown id:9e3fb02c-1706-4864-a27c-06aaadc5c11b tags:
### Step 3: Start a new jupyter-notebook and use it to compose your own analysis workflow.
%% Cell type:markdown id:57c641c2-748c-4c8a-a2f1-4d9428f141a5 tags:
The APAV documentation can support you with starting your own analyses.
***
%% Cell type:markdown id:a27bbf75-8a2a-4df3-8096-f942afc5e656 tags:
# Version, funding, feedback
* **APTyzer** https://github.com/areichm/APTyzer 887b82f
* **APTyzer** https://github.com/areichm/APTyzer e7e50e0
* **paraprobe-toolbox** https://gitlab.com/paraprobe/paraprobe-toolbox e349fd3 (v0.4)
* **APAV** https://pypi.org/project/APAV v1.4.0
* **NeXus** https://fairmat-experimental.github.io/nexus-fairmat-proposal latest
Last revision: Markus Kühbach, 2023/05/08
Last revision: Markus Kühbach, 2023/05/30
<a href="https://www.fairmat-nfdi.eu/fairmat/">FAIRmat</a> is a consortium on research data management which is part of the German NFDI.<br>
The project is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project 460197019.
If you spot issues with this container or you would like to suggest how we can improve the apmtools container:<br>
Please contact the respective maintainer of this container.
%% Cell type:code id:ff8b1235-d4c8-4ce0-9013-d1d87b7191df tags:
``` python
```
......
......@@ -36,7 +36,7 @@ RUN cd ~ \
&& chmod +x nodesource_setup.sh \
&& ./nodesource_setup.sh \
&& apt install -y nodejs=18.16.0-deb-1nodesource1 \
&& apt install -y m4=1.4.18-4 file=1:5.38-4 git=1:2.25.1-1ubuntu3.11 wget=1.20.3-1ubuntu2 libgl1-mesa-dev=21.2.6-0ubuntu0.1~20.04.2 libglu1-mesa-dev=9.0.1-1build1 build-essential=12.8ubuntu1.1 mpich=3.3.2-2build1 libgmp-dev=2:6.2.0+dfsg-4ubuntu0.1 libmpfr-dev=4.0.2-1 libssl-dev=1.1.1f-1ubuntu2.18 hwloc=2.1.0+dfsg-4 \
&& apt install -y m4=1.4.18-4 file=1:5.38-4 git=1:2.25.1-1ubuntu3.11 wget=1.20.3-1ubuntu2 libgl1-mesa-dev=21.2.6-0ubuntu0.1~20.04.2 libglu1-mesa-dev=9.0.1-1build1 build-essential=12.8ubuntu1.1 mpich=3.3.2-2build1 libgmp-dev=2:6.2.0+dfsg-4ubuntu0.1 libmpfr-dev=4.0.2-1 libssl-dev=1.1.1f-1ubuntu2.19 hwloc=2.1.0+dfsg-4 \
&& wget https://repo.anaconda.com/miniconda/Miniconda3-py38_23.1.0-1-Linux-x86_64.sh \
&& mv Miniconda3-py38_23.1.0-1-Linux-x86_64.sh miniconda3-py38_23.1.0-1-Linux-x86_64.sh \
&& chmod +x miniconda3-py38_23.1.0-1-Linux-x86_64.sh \
......@@ -56,7 +56,7 @@ RUN cd ~ \
&& cd /home/atom_probe_tools \
&& git clone https://github.com/areichm/APTyzer.git \
&& cd APTyzer \
&& git checkout 887b82f \
&& git checkout e7e50e038385107cd2fb25453792471b4edd9f8a \
&& cp APTyzer_V_1_2o.ipynb /home/atom_probe_tools/aptyzer/aptyzer.ipynb \
&& cp LICENSE /home/atom_probe_tools/aptyzer/LICENSE \
&& cp README.md /home/atom_probe_tools/aptyzer/README.md \
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment