nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2024-03-01T11:03:51Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1886Packages aliases in metainfo2024-03-01T11:03:51ZMarkus ScheidgenPackages aliases in metainfoWe are currently moving a lot of metainfo packages. It is necessary to have some infrastructure that allows to gracefully deprecate old package names. The goal is to not have to reprocess or otherwise migrate data instantly.
"Package al...We are currently moving a lot of metainfo packages. It is necessary to have some infrastructure that allows to gracefully deprecate old package names. The goal is to not have to reprocess or otherwise migrate data instantly.
"Package aliases" would allow to add old package names to a package definition:
- `m_from_dict` will consider package aliases when looking for definition referenced in `m_def`
- the gui will consider package aliases when looking for definitions referenced in `m_def`
- searches targeting `section_defs` will expand into an "or" on all package aliases
- facilities will be made available to clients to easily work with package names and consider package aliases in the process
Property aliases (espsecially for sub sections) is a related functionality, not covered in the issue.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1845Attribute inheritance2024-02-29T11:46:44ZMarkus ScheidgenAttribute inheritanceThe current metainfo does not support attribute inheritance. Attributes are primarily used in nexus. Here, the lack of attribute inheritance causes an excessive duplication of attribute definitions.
- [x] add inheritance to the metainf...The current metainfo does not support attribute inheritance. Attributes are primarily used in nexus. Here, the lack of attribute inheritance causes an excessive duplication of attribute definitions.
- [x] add inheritance to the metainfo python implementation
- [x] show attribute inheritance in the metainfo browser
- [x] adapt the generated nexus metainfoMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1901possible bug in Msection m_update_from_dict2024-02-26T15:55:35ZAlvin Noe Ladinespossible bug in Msection m_update_from_dict```python
from nomad.datamodel.metainfo.calculation import Calculation
c = Calculation()
c.m_update_from_dict({'temperature': 5, 'dos_electronic': {'n_energies': 1}})
```
this returns an error:
```
return MSection.from_dict(data, cls=cls...```python
from nomad.datamodel.metainfo.calculation import Calculation
c = Calculation()
c.m_update_from_dict({'temperature': 5, 'dos_electronic': {'n_energies': 1}})
```
this returns an error:
```
return MSection.from_dict(data, cls=cls, **kwargs)
File "/home/alvin/work/nomad2/nomad/nomad/metainfo/metainfo.py", line 2762, in from_dict
if 'm_ref_archives' in dct and isinstance(m_context, Context):
TypeError: argument of type 'int' is not iterable
```
@thchang any idea what is going on here?https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1609Type of to extended sections2023-12-21T16:06:52ZNathan DaelmanType of to extended sectionsI ran into this problem for FHI-aims (https://github.com/nomad-coe/electronic-parsers/issues/131), where I wanted to reference `AtomParameters` from a code-specific section (`x_fhi_aims_section_controlIn_basis_set`).
However, directly as...I ran into this problem for FHI-aims (https://github.com/nomad-coe/electronic-parsers/issues/131), where I wanted to reference `AtomParameters` from a code-specific section (`x_fhi_aims_section_controlIn_basis_set`).
However, directly assigning the section (the typical approach) results in a `TypeError`.
It still has `method.AtomParameters` and not `electronicparsers.fhiaims.metainfo.fhi_aims.AtomParameters`.
This might be appropriate in other cases, but not here.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1038Metainfo entries for Machine Learning models, workflows and output2023-12-21T16:06:52ZJose Marquez PrietoMetainfo entries for Machine Learning models, workflows and outputA user asked for advice on how he could have the output of his ML models as NOMAD entries. In particular, he is interested in crystal structures as the output of his ML predictions. But this could be extended to any kind of prediction fo...A user asked for advice on how he could have the output of his ML models as NOMAD entries. In particular, he is interested in crystal structures as the output of his ML predictions. But this could be extended to any kind of prediction for example, `bulk_modulus` values predicted by formulas and so on.
In the same way, as we provide infrastructure for generating entries for output generated by DFT codes, we could also do so for ML for Materials Informatics. Maybe there is something already available for this or a concept that I am not aware of. @lucamghi please, feel free to educate me on this topic. Obviously, with the upcoming variety of ML codes, writing code-specific parsers do not sound to me like a scalable and sustainable way to proceed. But maybe the new NOMAD features coming along like the custom schemas can provide some opportunities to offer a solution in this regard.
I give an example of a particular case: One user is doing ML for generative crystal structures. The output of the prediction is a set of crystal structures that he would like to have as nomad entries and enjoy the features and interoperability that NOMAD offers. One option would be to have in an upload the following:
- A notebook that the user uses to train and run his model, that hopefully could be run directly on NOMAD even after the upload is published.
- The output of the prediction (structure files, i.e. `.cif` files)
- A custom schema file `customML.schema.archive.yaml`. This schema contains sections and quantities that can deal with the output. For the crystal structure case, we could have a base section `BaseSectionWithStructureFile` with a `Quantity` called `structure_file`. This structure file gets parsed into the nomad metainfo populating `run.system.atoms` and then calling the normalizers to populate `results.materials` creating relatively rich NOMAD entries. This schema will necessarily need to have some metadata with a minimal description of the code used, the ML method, and so on. This could be provided as a base class with ELN components or just be written in an `archive.json` file directly that maps a custom schema.
We could then design an exemplary upload for this kind of workflow that users can adapt to their workflow.
With the help of @himanel1 and @ladinesa I have written an `ElnWithStructureFile` base section which kind of does this normalization, but to make it work we needed to do a change in `nomad/normalizing/method.py`. All of this is in [this branch](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/tree/Structure_File_Parser_PoC) as a proof of concept, but I have no idea of what would be the necessary minimal metadata to describe the ML workflow that should go along with these entries.
@mscheidg, @lucamghi, @afekete what do you think about this topic? Please, feel free to tag anyone else that should be interested in the topic.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1630Deprecate or extend metadata.domain2023-12-21T16:06:52ZAlvin Noe LadinesDeprecate or extend metadata.domainEntryArchive.metadata.domain currently is either dft/ems. This obviously need to be extended (see e.g. https://github.com/nomad-coe/nomad/issues/64). However, I am not quite sure if this is still relevant and therefore be deprecated. @hi...EntryArchive.metadata.domain currently is either dft/ems. This obviously need to be extended (see e.g. https://github.com/nomad-coe/nomad/issues/64). However, I am not quite sure if this is still relevant and therefore be deprecated. @himanel1 @mscheidg what do you think?https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1555Add deprecation chip2023-12-21T16:06:51ZNathan DaelmanAdd deprecation chipConsidering our current workflow, where we first have to reparse new quantities before deprecating old ones, we should have a mechanism to communicate the upcoming updates to the user.
I suggest some kind of "_deprecated_" chip (like th...Considering our current workflow, where we first have to reparse new quantities before deprecating old ones, we should have a mechanism to communicate the upcoming updates to the user.
I suggest some kind of "_deprecated_" chip (like the _repeated_ annotation that we have in the Metainfo) that is shown both in Metainfo and the Entry Data.
I think a chip is much more visible than, for example, updating the definition.
Maybe it should also link to the quantity replacing it, in case there is one.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1596Failing substance because of problems with external service2023-12-21T16:06:51ZMarkus ScheidgenFailing substance because of problems with external serviceI had a failed pipline because an HTTP request to an external API was getting a timeout. This had to be either mocked or more cracefully handled. We should not assume that external services are always available.I had a failed pipline because an HTTP request to an external API was getting a timeout. This had to be either mocked or more cracefully handled. We should not assume that external services are always available.Hampus NaesstroemHampus Naesstroemhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1314Clean XC Functional Names2023-12-21T16:06:49ZNathan DaelmanClean XC Functional NamesThe entries under `xc_functional_name` should be cleaned up.
This can be done is some progressive stages:
- [x] Fix simple typos. We continue using the LibXC nomenclature and extend it with `HF_X` for capturing exact exchange.
- [ ] Fi...The entries under `xc_functional_name` should be cleaned up.
This can be done is some progressive stages:
- [x] Fix simple typos. We continue using the LibXC nomenclature and extend it with `HF_X` for capturing exact exchange.
- [ ] Fix the non-trivial mapping issues.
- [ ] Turn `xc_functional_name` into an enumerate quantity.
- @mscheidg How feasible would it be to perform a partial reparsing (`xc_functional_name` only) after this point?
- [ ] Add metainfo for distingusihing internal DFT from those forwarded to LibXC.
- [ ] Improve the mapping logic for getting `HF_X` into the LibXC terminology.
- [ ] Separate out exchange- and correlation-only functionals (not to forget kinetic), while also providing an exchange-correlation field for matched names, e.g. `PBE` as a combination of `GGA_X_PBE` and `GGA_C_PBE` (?)Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1074System links and ASE atoms for results section2023-12-21T16:06:49ZMarkus ScheidgenSystem links and ASE atoms for results sectionThe results section's material does not reference the structure section it is based on.
The result's sections structure property is not referencing the section system/atoms it is based on.
Both the results structure sections and the sys...The results section's material does not reference the structure section it is based on.
The result's sections structure property is not referencing the section system/atoms it is based on.
Both the results structure sections and the system/atoms section could benefit from a function that is producing a respective ase.Atoms object.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1046Unit display in referenced sections2023-12-21T16:04:41ZMichael GötteUnit display in referenced sectionsDear nomad, I have an ELN entry
```
class Solution(Chemical):
'''Base class for a solution'''
chemical = Quantity(
type=Reference(Chemical.m_def),
a_eln=dict(component='ReferenceEditQuantity'))
powder_mas...Dear nomad, I have an ELN entry
```
class Solution(Chemical):
'''Base class for a solution'''
chemical = Quantity(
type=Reference(Chemical.m_def),
a_eln=dict(component='ReferenceEditQuantity'))
powder_mass = Quantity(
type=np.dtype(np.float64),
unit=('mg'),
a_eln=dict(component='NumberEditQuantity',defaultDisplayUnit='mg'))
solvent = Quantity(
type=Reference(Chemical.m_def),
shape=['*'],
a_eln=dict(component='ReferenceEditQuantity'))
solvent_percentage = Quantity(
type=np.dtype(np.float64),
a_eln=dict(component='NumberEditQuantity'))
shaker_temperature = Quantity(
type=np.dtype(np.float64),
unit=('°C'),
a_eln=dict(component='NumberEditQuantity',defaultDisplayUnit='°C'))
shaker_time = Quantity(
type=np.dtype(np.float64),
unit=('s'),
a_eln=dict(component='NumberEditQuantity',defaultDisplayUnit='s'))
shaker_speed = Quantity(
type=np.dtype(np.float64),
unit=('1/second'),
a_eln=dict(component='NumberEditQuantity',defaultDisplayUnit='1/s'))
```
It has defaultDisplayUnits, in the GUI it looks like this:
![grafik](/uploads/47afba7d4948ff21018674f38a270558/grafik.png)
it looks fine :smile:
but now I reference this in another entrydata:
![grafik](/uploads/bacc1f5f706a905447cae06d83f6d3a7/grafik.png)
and the units are the default ones again. So time is in fs, and temperature in K and mass in kg.
I am using a fork of v1.1.3
Best Michahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1375Deprecate browse annotation for explicit JSON and HTML metainfo type.2023-12-21T16:04:41ZMarkus ScheidgenDeprecate browse annotation for explicit JSON and HTML metainfo type.This would remove a relatively unknown and superflous annotation. Explicit types would give us more options to check values (e.g. sanitize) on deserialisation.This would remove a relatively unknown and superflous annotation. Explicit types would give us more options to check values (e.g. sanitize) on deserialisation.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1487Bad geometry convergence extraction2023-12-21T15:58:54ZNathan DaelmanBad geometry convergence extractionI found this [this entry](https://nomad-lab.eu/prod/v1/staging/gui/search/entries/entry/id/oDNOV2gZM8Ts9LCiUcKfmWuKk07l) (entry_id: oDNOV2gZM8Ts9LCiUcKfmWuKk07l, upload_id: CXorGZWOS3Khzk7QCIwtQA), which highlights several areas where ou...I found this [this entry](https://nomad-lab.eu/prod/v1/staging/gui/search/entries/entry/id/oDNOV2gZM8Ts9LCiUcKfmWuKk07l) (entry_id: oDNOV2gZM8Ts9LCiUcKfmWuKk07l, upload_id: CXorGZWOS3Khzk7QCIwtQA), which highlights several areas where our presentation of the convergence criteria in a geometry optimization should be improved.
The figure shows how the convergence criterion were apparently reached at step 7, yet the relaxation still continues.
In reality, the output shows that none of the 4 criteria were met at step 7.
This teaches us that:
- [ ] The extraction of the convergence parameters in `Crystal` have to be double-checked.
- [ ] The GUI could be more explicit in which stopping criterion is shown (namely `convergence_tolerance_energy_difference`).
@himanel1 @ Adrianna Wojas: how do you think that we could better convey the full range of criteria in the GUI?
Some codes stop once 1 is met, other require all to be met...
Should we for example have tabs following multiple properties (see above), each with 1 or 2 convergence criteria lines?
- [ ] Moreover, also wavefunction can be used as a convergence criterion and should be added.Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/928Incosistent use of integrated DOS values2023-12-21T15:58:52ZNathan DaelmanIncosistent use of integrated DOS valuesThe integrated DOS value is shown only in codes which output it (e.g. VASP).
However, given the regular DOS, the integrated counterpart can easily be calculated using a running sum.
For a user unfamiliar with the code output formats, it ...The integrated DOS value is shown only in codes which output it (e.g. VASP).
However, given the regular DOS, the integrated counterpart can easily be calculated using a running sum.
For a user unfamiliar with the code output formats, it is jarring to see integrated DOS "randomly" pop up.
My suggestion:
- Compute the integrated value (in the normalizer?) when one is not given.
- Or remove the Quantity all together.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/289Simulations dynamically handled by ASE2023-12-21T15:58:46ZCuauhtemoc Salazartemok@physik.hu-berlin.deSimulations dynamically handled by ASECc: @mscheidg
This is a comment on the general issue on how to parse calculations "dynamically handled
by ASE". This aroused while solving nomad-lab/nomad-FAIR#273 (which is now fixed, in the
sense that we can parse what the user uploa...Cc: @mscheidg
This is a comment on the general issue on how to parse calculations "dynamically handled
by ASE". This aroused while solving nomad-lab/nomad-FAIR#273 (which is now fixed, in the
sense that we can parse what the user uploaded).
In this post I would like to draw attention to the fact that in a large
number of cases, the main results of calculations "dynamically handled by ASE" are not included in the output files uploaded by the users (see below). Solving this situation
may require consensus within / between NOMAD & ASE.
Background: An inspection of the Q-Espresso input & output files associated with
nomad-lab/nomad-FAIR#273 (now solved) revealed that most of these calculations were
relaxation tasks, as the Espresso input file contained
```
calculation='relax'
...
ion_dynamics='ase3'
```
However, only the starting atom coordinates are reported, i.e., **the intermediate & final coordinates are missing in the
Espresso output file**; this is critical, since the task was a structural relaxation. Yet, such file had a clean end, as usual, with a timing report and a
'JOB DONE' message. See for instance [(repo
link)](https://repository.nomad-coe.eu/app/gui/entry/id/evKd-Lb8S7C5C7s-Ld3MxA/vF8g488PLu2pn0N4omfN0-MBljoj)
```
entry_id: vF8g488PLu2pn0N4omfN0-MBljoj
upload_id: evKd-Lb8S7C5C7s-Ld3MxA
```
ASE's documentation for the Espresso's `ion_dynamics = ase3` reads "_dynamic communication between Quantum Espresso and python, in this case ASE updates coordinates during relaxation_"
[ASE (link)](https://ase-espresso.readthedocs.io/en/latest/api/espresso.html).
My interpretation is that **ASE** calls specific subroutines from Espresso in order to
perform a given task, e.g, a relaxation. But unfortunately, for the cases I inspected, it
seems that ASE didn't call the Espresso subroutines that **report/print** the results to
the Espresso output file (stdout). Instead, it seems that such results were read "on the fly" and
collected in the ASE script/notebook driving the simulation.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/784NOMAD Oasis (GUI)2023-12-21T15:55:39ZMarkus ScheidgenNOMAD Oasis (GUI)The NOMAD Oasis needs to be different:
- [ ] revise the used "terms" in menus, breadcrumbs, tabs, etc. Potentially for all of NOMAD?
- [x] different logo and/or colors to make it distinguishable. Maybe prominently show an installation na...The NOMAD Oasis needs to be different:
- [ ] revise the used "terms" in menus, breadcrumbs, tabs, etc. Potentially for all of NOMAD?
- [x] different logo and/or colors to make it distinguishable. Maybe prominently show an installation name.
- [ ] oasis installations register with central installation
- [ ] show network, allow navigation between installations.
- [ ] "publish": bundle/archive, allow metadata index, allow copy, trigger copy
- [ ] no embargo on Oasishttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1438Issue with adding sections using python syntax2023-12-21T15:40:05ZLauri HimanenIssue with adding sections using python syntaxThere is currently an issue when trying to use the regular python syntax to add subsections in the metainfo.There is currently an issue when trying to use the regular python syntax to add subsections in the metainfo.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/300Mypy and pylint plugins for the metainfo2023-12-21T15:40:02ZMarkus ScheidgenMypy and pylint plugins for the metainfoThe metainfo adds some dynamic properties. It would be great to have hints for mypy and pylint to correctly check those statically. I started on a pylint plugin, before I realized that we would also need a mypy-plugin. For now, the MSect...The metainfo adds some dynamic properties. It would be great to have hints for mypy and pylint to correctly check those statically. I started on a pylint plugin, before I realized that we would also need a mypy-plugin. For now, the MSection sports an unnecessary __getattr__ to make mypy and pylint not check un "unexsisting" members.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/373Explore the use of pydantic or dataclasses for the metainfo2023-12-21T15:40:01ZMarkus ScheidgenExplore the use of pydantic or dataclasses for the metainfoTest, if we can base the metainfo on (pydantic)[https://pydantic-docs.helpmanual.io/] or at least Python (dataclasses)[https://docs.python.org/3/library/dataclasses.html] for more buildin typechecking, code-assist, and re-use of pro libs...Test, if we can base the metainfo on (pydantic)[https://pydantic-docs.helpmanual.io/] or at least Python (dataclasses)[https://docs.python.org/3/library/dataclasses.html] for more buildin typechecking, code-assist, and re-use of pro libs.
Would makes #300 obsoletehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/509Collisions in metainfo2023-12-21T15:40:01ZLauri HimanenCollisions in metainfoThe metainfo sections follow the `collections.abs.Mapping` interface and have functions called `values`, `keys`, `items`, `get`, etc.
What would be the preferred way of dealing with a quantitity or subsection that collides with these fu...The metainfo sections follow the `collections.abs.Mapping` interface and have functions called `values`, `keys`, `items`, `get`, etc.
What would be the preferred way of dealing with a quantitity or subsection that collides with these function names? In my case, I would prefer to assign something to `values`, but can't. I guess it is not an option to use some other name for these functions (e.g. m_values), since that will break the interface. Should I just use another name in this case?