nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2023-10-18T10:16:48Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1507Resolve "Stable graph query result"2023-10-18T10:16:48ZTheodore ChangResolve "Stable graph query result"Closes #1739Closes #1739Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1506Add support for process status2023-10-18T11:38:41ZTheodore ChangAdd support for process statusTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1505Gromacs cleanup2023-10-18T11:06:56ZAlvin Noe LadinesGromacs cleanupAlvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1504Added JOSS paper badge and Citation in README2023-10-18T14:49:39ZJose Marquez PrietoAdded JOSS paper badge and Citation in READMEJose Marquez PrietoJose Marquez Prietohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1503Documentation changes2023-10-18T05:43:00ZAdrianna WojasDocumentation changes- changed label from learn -> explanation
- changed label custom schema -> schema
- corrected misspellings- changed label from learn -> explanation
- changed label custom schema -> schema
- corrected misspellingsAdrianna WojasAdrianna Wojashttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1500Possibility to use custom schemas in apps and search2023-10-24T13:11:13ZLauri HimanenPossibility to use custom schemas in apps and searchThis MR adds the possibility of using scalar quantities from custom schemas (both Python and YAML) in queries, aggregations and in app definitions. A new documentation for Apps is also included.
# Highlights
- That path in the data is n...This MR adds the possibility of using scalar quantities from custom schemas (both Python and YAML) in queries, aggregations and in app definitions. A new documentation for Apps is also included.
# Highlights
- That path in the data is no longer enough to target a quantity. E.g. `data.sample.id` may be defined by several different schemas. We need to add an identifier for the schema in the quantity name.
- Schemas are identified using the `qualified_name` of the root section (can be fetched with `section.qualified_name()`):
- Python schemas: path of the class name, e.g. `nomadschemaexample.schema.MySchema`
- YAML schemas: `entry_id:gTqaJYQ7IH20dl5PeX7ZPzkHudI8.MySchema`
The other option would be to use the reference syntax (can be fetched with `definition_reference()`), but the references to YAML files are very hard to use, as they look like this: `../uploads/Yl6DTVCVS1GqYRwEhmexrA/raw/schema.archive.yaml#/definitions/section_definitions/1`.
- The full quantity identifier is `<path>#<schema_name>` to target a quantity in the schema. E.g. `data.sample.id#nomadschemaexample.schema.MySchema`.
- The GUI will support simplified rendering of schema identifiers to remove clutter from the UI, but in the app config and in our backend the quantities need to be identified by this full name.
- Currently `#` is used as a separator between path and schema name. The choice is complicated by the fact that many separator are reserved for other purposes:
- `.` denotes section hierarchy in paths, and is also used in schema identifiers.
- `:` is used in the YAML schema name and also for query modifiers, e.g. `material.elements:all: ['Si', 'C']`
- `/` is used in inner_section_definitions and in YAML schema paths
- `&` is reserved for url query parameters
- `@` is reserved for indicating a hash digest for a definition. Used to e.g. distinguish between different versions of a schema.
- Any operators commonly used in boolean logic (+, -, &, |) should not be used if we want to later add support for them in the search bar.
- For technical reasons, the API calls targeting YAML quantities will need to include the data type. The current syntax is like this: `<path>#<schema_name>#<dtype>`. This data type is added fully transparently by the GUI, but it is up to the user to include it in manual API calls (you will get a meaningful warning if you omit it).
# Example setups for testing
## Python schema
1. Get the code: `git checkout search`
2. Add the test Python schema included in our source code to your `PYTHONPATH`: `export PYTHONPATH="${PYTHONPATH}:/<root folder>/nomad-FAIR/tests/data/plugins"`
3. Copy and use this `nomad.yaml` file: [nomad.yaml](/uploads/e2f4bfd6cad3918d14acfffb327a0675/nomad.yaml)
4. Boot up docker, appworker, GUI
5. Login, create new upload, upload this file: [dataset.zip](/uploads/1dfa82b621bc107cc212069810a95965/dataset.zip)
6. Go to "Explore/My Python Schema" to try out different things.
## YAML schema
1. Get the code: `git checkout search`
2. Boot up docker, appworker, GUI
3. Login, create new upload, upload this file containing the schema: [schema.archive.yaml](/uploads/f199116fdf9398a504f44d6aa9e32048/schema.archive.yaml). Note down the `upload_id` and `entry_id` for the schema.
4. Modify this `nomad.yaml` file: [nomad.yaml](/uploads/e3e12cbbce7b40b3d8b4cdeb0d9c40aa/nomad.yaml) so that it uses the `entry_id` you got from the previous step.
5. Use the `upload_id` in line 8 of this script: [generator.py](/uploads/ce2d4b12aa9a4dbd6ee211932027d955/generator.py). Run the script, zip the produced dataset folder.
6. Restart appworker with the new nomad.yaml file. Login, upload the zipped dataset.
7. Go to "Explore/My YAML Schema" to try out different things.
# Known limitations compared to "native" quantities
- Search boxes cannot show suggestions for values (due to the hierarchy being flattened within ES, our [suggestion mechanism](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html#completion-suggester) cannot filter out the values to show).
- Only scalar quantities are available (might be possible to get around, not sure)
- [Nested queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html) are not possible (due to the hierarchy being flattened in ES)
- By default, a single document can contain 10 000 nested documents. So for very large archives, this limit may be hit at some point when `search_quantities` is populated.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1497Fixed missing link to download installation files for oasis with keycloak.2023-10-13T16:01:32ZMarkus ScheidgenFixed missing link to download installation files for oasis with keycloak.Changelog: FixedChangelog: Fixedhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1495Ruff2023-10-31T15:08:23ZAhmed IlyasRuffThis merge request introduces Ruff as a new linter, replacing Pycodestyle. The plan is to gradually incorporate additional rules from the [Ruff rules](https://docs.astral.sh/ruff/rules/) and potentially phase out pylint in the future.
...This merge request introduces Ruff as a new linter, replacing Pycodestyle. The plan is to gradually incorporate additional rules from the [Ruff rules](https://docs.astral.sh/ruff/rules/) and potentially phase out pylint in the future.
Some benchmarks:
```
Summary
'ruff check nomad tests' ran
166.68 ± 2.56 times faster than 'pycodestyle --config=pycodestyle.ini nomad tests'
```
```
Summary
'ruff check -e nomad test' ran
2101.03 ± 110.33 times faster than 'pycodestyle --config=pycodestyle.ini nomad tests && pylint --rcfile=.pylintrc nomad tests'
```Ahmed IlyasAhmed Ilyashttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1493Add github commit hash2023-11-23T17:04:10ZAlvin Noe LadinesAdd github commit hashAlvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1488Resolve "Graph model generation, metadata token support"2023-10-10T16:06:31ZTheodore ChangResolve "Graph model generation, metadata token support"Closes #1719Closes #1719Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1487Add support for negative indexing in msection2023-10-10T10:34:02ZAlvin Noe LadinesAdd support for negative indexing in msectionAlvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1486Fixed etags produced by fastapi to work with nginx.2023-10-06T16:51:28ZMarkus ScheidgenFixed etags produced by fastapi to work with nginx.Changelog: FixedChangelog: Fixedhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1479Update file parser and parser documentation2023-10-13T12:56:40ZAlvin Noe LadinesUpdate file parser and parser documentationImprove file parser, text parser documentation and minor refactoring.
Update parser writing documentation #1537 and example parsers.Improve file parser, text parser documentation and minor refactoring.
Update parser writing documentation #1537 and example parsers.Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1473Add parser matching of excel contents2023-09-27T12:30:36ZAlvin Noe LadinesAdd parser matching of excel contentsAdd support for matching contents of excel file to tabular parser.Add support for matching contents of excel file to tabular parser.Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1465Added metadata in results to search for measurements and in particular xrd2023-09-27T13:25:22ZJose Marquez PrietoAdded metadata in results to search for measurements and in particular xrdThis adds a few sections and quantities to the current results to make measurements searchable. It will start being populated from schemas and parsers developed in https://github.com/FAIRmat-NFDI/nomad-measurements.
@himanel1 can you h...This adds a few sections and quantities to the current results to make measurements searchable. It will start being populated from schemas and parsers developed in https://github.com/FAIRmat-NFDI/nomad-measurements.
@himanel1 can you have a look at this? Should be a quick one for you. In the future, we can refactor the EBSD database and remove the top-level measurements section.
@lucamghi, @sbrueck, @sanbrock, @hnaesstroem I tagged you just in case you want to have a second look at it.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1463Resolve "High accuracy porosity modification"2023-09-15T11:18:55ZDinga WonankeResolve "High accuracy porosity modification"The initial code only computed standard porosity. But now included a functionality such that it should perform high accuracy computation of
porosity. The high accuracy computation has now been set as the default value. Some minor change...The initial code only computed standard porosity. But now included a functionality such that it should perform high accuracy computation of
porosity. The high accuracy computation has now been set as the default value. Some minor changes have also been made on the python doc string.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1458Update parser2023-09-07T11:05:16ZAlvin Noe LadinesUpdate parserAlvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1453Updated ref to electronic-parsers with CP2K fix2023-08-29T08:18:05ZJose PizarroUpdated ref to electronic-parsers with CP2K fixhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1452CLI utility to export archive data2023-09-19T13:56:26ZMarkus ScheidgenCLI utility to export archive data... e.g. for use in machine learning.... e.g. for use in machine learning.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1450Added oasis registration link to the docs.2023-08-24T12:03:59ZMarkus ScheidgenAdded oasis registration link to the docs.Changelog: AddedChangelog: Added