nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2024-03-18T13:54:14Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1741Resolve "Config does not allow string loglevel like "INFO" anymore"2024-03-18T13:54:14ZMarkus ScheidgenResolve "Config does not allow string loglevel like "INFO" anymore"Closes #1944Closes #1944https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1740Draft: Resolve "pyiron integration"2024-03-18T08:51:38ZAmir GolparvarDraft: Resolve "pyiron integration"Closes #1872Closes #1872Amir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1739Resolve "EditQuantities use global units instead of default units"2024-03-18T08:21:16ZLauri HimanenResolve "EditQuantities use global units instead of default units"Closes #1929Closes #1929https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1737Improving Install plugins docu2024-03-18T12:42:10ZJose PizarroImproving Install plugins docuCloses #1941
@aalbino would you mind reviewing these changes? I just wanted to improve a bit the description on how to add a plugin as a developer. I don't know whether the place for it is the best (because it is under How to > OASIS),...Closes #1941
@aalbino would you mind reviewing these changes? I just wanted to improve a bit the description on how to add a plugin as a developer. I don't know whether the place for it is the best (because it is under How to > OASIS), I just fixed the how to regarding us, developers :slight_smile:
Thanks!Jose PizarroJose Pizarrohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1734Resolve "plotly config dictionary can't be fed in from normalize function"2024-03-15T10:59:52ZDinga WonankeResolve "plotly config dictionary can't be fed in from normalize function"We created a new function create_topology_porosity, which will enable users to easily populate the topology.
In addition, we moved most of the global functions from mof_deconstructor into atomutils, which will facilitate the implementat...We created a new function create_topology_porosity, which will enable users to easily populate the topology.
In addition, we moved most of the global functions from mof_deconstructor into atomutils, which will facilitate the implementation of the wrapping function in the future.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1731Draft: Resolve "Parsing nexus files generates a lot of warnings and errors"2024-03-18T12:04:52ZMarkus ScheidgenDraft: Resolve "Parsing nexus files generates a lot of warnings and errors"Fixed some problems of #1934
- I fixed a bug in the attribute generation. Basically some attributes were missing. But this only solved a small portion of the problems.
- I fixed a bug in the MetainforNormalizer which tried to call thin...Fixed some problems of #1934
- I fixed a bug in the attribute generation. Basically some attributes were missing. But this only solved a small portion of the problems.
- I fixed a bug in the MetainforNormalizer which tried to call things called "normalize" in the schema, but where not actually normalize functions.
- I fixed the error handling a bit to better see what is actually causing the problems.
### The first error type mentioned in #1934
This is fixed with the MetainforNormalizer fix.
### The second error type mentioned in #1934
These are a few separate issues repeating. They are reported while setting an attribute, but they are caused because the parser could not find the quantity or section to put the attribute on. Therefore, they do not seem to be attribute problems. They are indicating a few mapping problems that I cannot resolve!
- string `unit` in `nxdl_path`, where the parser only expects instances of `ET.Element`. This maybe is a problem in `pynxtools.nexus.HandleNexus` which produces all the `nxdl_path`.
```
WARNING nomad.client 2024-03-14T13:46:17 Error while setting attribute.
- exception: Traceback (most recent call last):
File "/Users/markus/Documents/Projects/nomad-fairdi/nomad/parsing/nexus/nexus.py", line 167, in _populate_data
assert isinstance(
AssertionError: nx_path is not a list of ET.Element, one segment is "units:<class 'str'>"
```
- The parsre tries to set `version__attribute` to `nexus.NXentry.COLLECTION:SubSection`, but the nexus mapping does not define attributes on sub sections, I guess this is supposed to go to the section that the sub-section points to?
```
WARNING nomad.client 2024-03-14T13:46:17 Error while setting attribute.
- exception: Traceback (most recent call last):
File "/Users/markus/Documents/Projects/nomad-fairdi/nomad/parsing/nexus/nexus.py", line 194, in _populate_data
assert isinstance(
AssertionError: nexus.NXentry.COLLECTION:SubSection is not a Quantity
```
- `nexus.NXelectronanalyser.NXtransformations` has no suitable quantity for `depends_on__field`
```
WARNING nomad.client 2024-03-14T13:46:18 Error while setting attribute.
- exception: Traceback (most recent call last):
File "/Users/markus/Documents/Projects/nomad-fairdi/nomad/parsing/nexus/nexus.py", line 182, in _populate_data
assert metainfo_def, f'{current.m_def} has no suitable property for {parent_field_name}'
AssertionError: nexus.NXelectronanalyser.NXtransformations:Section has no suitable property for depends_on__field
```
- `nexus.NXelectronanalyser.NXtransformations` has no suitable quantity for `transformation_type__field`
```
WARNING nomad.client 2024-03-14T13:46:18 Error while setting attribute.
- exception: Traceback (most recent call last):
File "/Users/markus/Documents/Projects/nomad-fairdi/nomad/parsing/nexus/nexus.py", line 182, in _populate_data
assert metainfo_def, f'{current.m_def} has no suitable property for {parent_field_name}'
AssertionError: nexus.NXelectronanalyser.NXtransformations:Section has no suitable property for transformation_type__field
```
Then there are a few less frequent things:
- The value "HHG laser" is not an enum value for `nexus.NXsource.type__field`.
- NaN value is not set for field on `size__field[size__field]`
# The third error mentioned in #1934
These errors are caused because we try to interpret enum values as floats. This is all for this `nexus.NXtransformations.AXISNAME__field`, which is defined as a float in the schema. I guess it should not be a float?https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1726Integrity check2024-03-18T15:40:58ZTheodore ChangIntegrity checkTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1719Draft: bump dependencies for pandas and scipy2024-03-11T14:04:41ZAhmed IlyasDraft: bump dependencies for pandas and scipyAhmed IlyasAhmed Ilyashttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1717Move pynxtools and plugins into parsing packages2024-03-15T14:18:47ZFlorian DobenerMove pynxtools and plugins into parsing packagesThis adds pynxtools and respective plugins to the `parsing` section of `pyproject.toml` and regenerates the requirements files with it. It also removes the nexus submodule as this is no longer needed.
This is a _hotfix_ to have the newe...This adds pynxtools and respective plugins to the `parsing` section of `pyproject.toml` and regenerates the requirements files with it. It also removes the nexus submodule as this is no longer needed.
This is a _hotfix_ to have the newest examples ready for DPG and proper pluginisation of pynxtools will be handled in !1629.
As discussed with @lukaspie @sanbrock.
ToDo:
- [x] Update `pynxtools==0.1.0`Florian DobenerFlorian Dobenerhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1706Draft: Resolve "automatic creation and resolution of hdf5 references for larg...2024-03-18T13:40:45ZAlvin Noe LadinesDraft: Resolve "automatic creation and resolution of hdf5 references for large arrays"Closes #1903 #1789Closes #1903 #1789Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1704Draft: Full refactoring of `run` into `data`2024-03-11T12:30:43ZJose PizarroDraft: Full refactoring of `run` into `data`We need to keep track of the changes in files somewhere.
Closes #1728We need to keep track of the changes in files somewhere.
Closes #1728https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1698WIP2024-03-01T15:15:12ZMarkus ScheidgenWIPhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1693Resolve "label of list of subsection"2024-03-01T14:20:14ZMohammad NakhaeeResolve "label of list of subsection"Closes #1906Closes #1906Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1687Switched from the soap plugin project to the atomic descriptors project2024-03-01T11:02:19ZMarkus ScheidgenSwitched from the soap plugin project to the atomic descriptors projecthttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1681Draft: Resolve "derived display unit system"2024-02-21T16:20:01ZMohammad NakhaeeDraft: Resolve "derived display unit system"Closes #1898Closes #1898Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1666Draft: Resolve "Update developing plugin documentation"2024-03-11T14:34:10ZJoseph RudzinskiDraft: Resolve "Update developing plugin documentation"Closes #1887Closes #1887Joseph RudzinskiJoseph Rudzinskihttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1663Draft: Resolve "rework elabftw parser"2024-03-15T10:08:24ZAmir GolparvarDraft: Resolve "rework elabftw parser"Closes #1882Closes #1882Amir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1655Draft: Resolve "Generic xml parser"2024-02-01T04:50:32ZAlvin Noe LadinesDraft: Resolve "Generic xml parser"Closes #1875Closes #1875Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1653New Atomic Descriptors2024-02-05T10:41:34ZJames DarbyNew Atomic DescriptorsHi,
I've updated the atomic descriptors that are computed. The old approach was to store 4 different variations of the SOAP power spectrum, 2 descriptor vectors for each atom in the system and then 2 structure-wise-averaged descriptor ...Hi,
I've updated the atomic descriptors that are computed. The old approach was to store 4 different variations of the SOAP power spectrum, 2 descriptor vectors for each atom in the system and then 2 structure-wise-averaged descriptor vectors. Instead of this, the new approach is to store 2 structure-wise-averaged descriptor vectors only. One of these is an element-agnostic, radially scaled variant of the SOAP power spectrum and the second is the descriptor extracted from the recent MACE-MP-0 foundation model.
Questions/Discussion points:
1. Total storage requirement is now greatly reduced to 296 floats per system. With 13 million structures currently in NOMAD I make this about 16 GB of data. Is this acceptable?
2. Adding the MACE descriptors has introduced new dependencies via [mace-torch](https://github.com/ACEsuit/mace/tree/88d49f9ed6925dec07d1777043a36e1fe4872ff3?tab=readme-ov-file#installation). mace-torch is pip installable so I'm hoping this won't cause any issues. I haven't updated requirement.txt etc. Should I do this?
3. The MACE descriptors are computed using the MACE-MP-0 foundation model. The model file (42 MB) gets automatically downloaded to ~/.cache/mace/ the first time it is called. Think we need to be careful with how this will work with multiple different "workers" running in parallel. Will they try to write to the same file?
4. Loading the MACE model takes much longer than computing the MACE descriptor. Currently the model is loaded once per system, I don't see a great way of avoiding this but worth being aware of.
5. The current schema for SOAP descriptors (shown below) includes entries for the parameters that were used to generate the descriptors e.g. `r_cut`, `n_max` etc. These will be the same for all structures in the database so this seems inefficient? My proposal is to put this information into the quantity definition of the soap_descriptor itself (see image) so the same data isn't duplicated for each structure. Thoughts?
![image](/uploads/c9c58f497f8e66fda43360f74956d74d/image.png)Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1644Draft: Resolve "Add experimental tutorial"2024-01-25T12:49:38ZHampus NaesstroemDraft: Resolve "Add experimental tutorial"Closes #1864Closes #1864Sebastian BruecknerSebastian Brueckner