nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2023-10-25T19:08:33Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1519Enable recursive listing2023-10-25T19:08:33ZTheodore ChangEnable recursive listingTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1516feat: allow querying owned or shared uploads2023-10-25T10:52:19ZAhmed Ilyasfeat: allow querying owned or shared uploadsAhmed IlyasAhmed Ilyashttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1514Graph small fix2023-10-24T20:25:19ZTheodore ChangGraph small fixTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1500Possibility to use custom schemas in apps and search2023-10-24T13:11:13ZLauri HimanenPossibility to use custom schemas in apps and searchThis MR adds the possibility of using scalar quantities from custom schemas (both Python and YAML) in queries, aggregations and in app definitions. A new documentation for Apps is also included.
# Highlights
- That path in the data is n...This MR adds the possibility of using scalar quantities from custom schemas (both Python and YAML) in queries, aggregations and in app definitions. A new documentation for Apps is also included.
# Highlights
- That path in the data is no longer enough to target a quantity. E.g. `data.sample.id` may be defined by several different schemas. We need to add an identifier for the schema in the quantity name.
- Schemas are identified using the `qualified_name` of the root section (can be fetched with `section.qualified_name()`):
- Python schemas: path of the class name, e.g. `nomadschemaexample.schema.MySchema`
- YAML schemas: `entry_id:gTqaJYQ7IH20dl5PeX7ZPzkHudI8.MySchema`
The other option would be to use the reference syntax (can be fetched with `definition_reference()`), but the references to YAML files are very hard to use, as they look like this: `../uploads/Yl6DTVCVS1GqYRwEhmexrA/raw/schema.archive.yaml#/definitions/section_definitions/1`.
- The full quantity identifier is `<path>#<schema_name>` to target a quantity in the schema. E.g. `data.sample.id#nomadschemaexample.schema.MySchema`.
- The GUI will support simplified rendering of schema identifiers to remove clutter from the UI, but in the app config and in our backend the quantities need to be identified by this full name.
- Currently `#` is used as a separator between path and schema name. The choice is complicated by the fact that many separator are reserved for other purposes:
- `.` denotes section hierarchy in paths, and is also used in schema identifiers.
- `:` is used in the YAML schema name and also for query modifiers, e.g. `material.elements:all: ['Si', 'C']`
- `/` is used in inner_section_definitions and in YAML schema paths
- `&` is reserved for url query parameters
- `@` is reserved for indicating a hash digest for a definition. Used to e.g. distinguish between different versions of a schema.
- Any operators commonly used in boolean logic (+, -, &, |) should not be used if we want to later add support for them in the search bar.
- For technical reasons, the API calls targeting YAML quantities will need to include the data type. The current syntax is like this: `<path>#<schema_name>#<dtype>`. This data type is added fully transparently by the GUI, but it is up to the user to include it in manual API calls (you will get a meaningful warning if you omit it).
# Example setups for testing
## Python schema
1. Get the code: `git checkout search`
2. Add the test Python schema included in our source code to your `PYTHONPATH`: `export PYTHONPATH="${PYTHONPATH}:/<root folder>/nomad-FAIR/tests/data/plugins"`
3. Copy and use this `nomad.yaml` file: [nomad.yaml](/uploads/e2f4bfd6cad3918d14acfffb327a0675/nomad.yaml)
4. Boot up docker, appworker, GUI
5. Login, create new upload, upload this file: [dataset.zip](/uploads/1dfa82b621bc107cc212069810a95965/dataset.zip)
6. Go to "Explore/My Python Schema" to try out different things.
## YAML schema
1. Get the code: `git checkout search`
2. Boot up docker, appworker, GUI
3. Login, create new upload, upload this file containing the schema: [schema.archive.yaml](/uploads/f199116fdf9398a504f44d6aa9e32048/schema.archive.yaml). Note down the `upload_id` and `entry_id` for the schema.
4. Modify this `nomad.yaml` file: [nomad.yaml](/uploads/e3e12cbbce7b40b3d8b4cdeb0d9c40aa/nomad.yaml) so that it uses the `entry_id` you got from the previous step.
5. Use the `upload_id` in line 8 of this script: [generator.py](/uploads/ce2d4b12aa9a4dbd6ee211932027d955/generator.py). Run the script, zip the produced dataset folder.
6. Restart appworker with the new nomad.yaml file. Login, upload the zipped dataset.
7. Go to "Explore/My YAML Schema" to try out different things.
# Known limitations compared to "native" quantities
- Search boxes cannot show suggestions for values (due to the hierarchy being flattened within ES, our [suggestion mechanism](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html#completion-suggester) cannot filter out the values to show).
- Only scalar quantities are available (might be possible to get around, not sure)
- [Nested queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html) are not possible (due to the hierarchy being flattened in ES)
- By default, a single document can contain 10 000 nested documents. So for very large archives, this limit may be hit at some point when `search_quantities` is populated.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1512Resolve "Deprecated plot annotation"2023-10-23T16:59:57ZMohammad NakhaeeResolve "Deprecated plot annotation"Closes #1742Closes #1742Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1502Resolve "Workflow for chemical reaction steps"2023-10-23T16:51:59ZAlvin Noe LadinesResolve "Workflow for chemical reaction steps"Closes #1430Closes #1430Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1455Resolve "Basesections Should Populate Workflow Section"2023-10-20T13:02:58ZHampus NaesstroemResolve "Basesections Should Populate Workflow Section"Closes #1658Closes #1658Hampus NaesstroemHampus Naesstroemhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1509Resolve "Wrong url for dataset on the metadata card"2023-10-19T12:54:15ZMarkus ScheidgenResolve "Wrong url for dataset on the metadata card"Closes #1715Closes #1715https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1508Fixed exessively long entry results columns.2023-10-19T11:36:18ZMarkus ScheidgenFixed exessively long entry results columns.Reduces the maximum width of entry name and entry type columns
to avoid distroted table layouts. The content overflow is
shown with ellipsis.
Changelog: FixedReduces the maximum width of entry name and entry type columns
to avoid distroted table layouts. The content overflow is
shown with ellipsis.
Changelog: Fixedhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1504Added JOSS paper badge and Citation in README2023-10-18T14:49:39ZJose Marquez PrietoAdded JOSS paper badge and Citation in READMEJose Marquez PrietoJose Marquez Prietohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1506Add support for process status2023-10-18T11:38:41ZTheodore ChangAdd support for process statusTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1505Gromacs cleanup2023-10-18T11:06:56ZAlvin Noe LadinesGromacs cleanupAlvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1507Resolve "Stable graph query result"2023-10-18T10:16:48ZTheodore ChangResolve "Stable graph query result"Closes #1739Closes #1739Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1503Documentation changes2023-10-18T05:43:00ZAdrianna WojasDocumentation changes- changed label from learn -> explanation
- changed label custom schema -> schema
- corrected misspellings- changed label from learn -> explanation
- changed label custom schema -> schema
- corrected misspellingsAdrianna WojasAdrianna Wojashttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1378Resolve "Plot support refactor"2023-10-17T14:37:35ZMohammad NakhaeeResolve "Plot support refactor"Closes #1554Closes #1554Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1499Added app customization documentation.2023-10-16T14:59:03ZLauri HimanenAdded app customization documentation.Closes #1733Closes #1733Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1431Resolve "draft for heterogeneous catalysis app"2023-10-16T12:11:31ZJulia SchumannResolve "draft for heterogeneous catalysis app"Closes #1592
Additions to results.py and to gui files to prepare a new search app for Heterogeneous Catalysis.
The schemas to generate the relevant entries are seperate in a plugin published here: https://github.com/schumannj/nomad-s...Closes #1592
Additions to results.py and to gui files to prepare a new search app for Heterogeneous Catalysis.
The schemas to generate the relevant entries are seperate in a plugin published here: https://github.com/schumannj/nomad-schema-plugin-fhi-catalysis
I attach a datafile for the Activity Test entry, from which the reactivity quantities such as products/names are taken.
[MoVOx_C2_performance_cleared_NaN.xlsx](/uploads/9d06b02bfb6df599672638b830a84389/MoVOx_C2_performance_cleared_NaN.xlsx)Julia SchumannJulia Schumannhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1498Fixed issue with column table column descriptions, fixed issue with searchbar...2023-10-14T13:05:33ZLauri HimanenFixed issue with column table column descriptions, fixed issue with searchbar...Fixed issue with column table column descriptions, fixed issue with searchbar data type query, added filterGroups into SearchContext.
Closes #1693Fixed issue with column table column descriptions, fixed issue with searchbar data type query, added filterGroups into SearchContext.
Closes #1693Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1497Fixed missing link to download installation files for oasis with keycloak.2023-10-13T16:01:32ZMarkus ScheidgenFixed missing link to download installation files for oasis with keycloak.Changelog: FixedChangelog: Fixedhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1479Update file parser and parser documentation2023-10-13T12:56:40ZAlvin Noe LadinesUpdate file parser and parser documentationImprove file parser, text parser documentation and minor refactoring.
Update parser writing documentation #1537 and example parsers.Improve file parser, text parser documentation and minor refactoring.
Update parser writing documentation #1537 and example parsers.Alvin Noe LadinesAlvin Noe Ladines