nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2022-03-18T15:25:42Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/601Merge for release v1.0.62022-03-18T15:25:42ZMarkus ScheidgenMerge for release v1.0.6https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/600Merge latest released changes.2022-03-18T14:17:38ZMarkus ScheidgenMerge latest released changes.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/599Allow custom metainfo definitions as part of the data (server-side) #7052022-03-24T16:34:44ZMarkus ScheidgenAllow custom metainfo definitions as part of the data (server-side) #705- refactoring of metainfo context
- implementation of URL and Python based references to definitions
- support for inter entry and inter upload references
- first usage in rudimentary tabular data parser- refactoring of metainfo context
- implementation of URL and Python based references to definitions
- support for inter entry and inter upload references
- first usage in rudimentary tabular data parserMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/598Merge for release v1.0.52022-03-17T09:26:13ZMarkus ScheidgenMerge for release v1.0.5https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/597Resolve "Range picker histograms"2022-04-22T09:59:38ZLauri HimanenResolve "Range picker histograms"Closes #656Closes #656Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/596Misc hot fixes2022-03-17T08:50:03ZMarkus ScheidgenMisc hot fixeshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/595Merge for release of v1.0.42022-03-16T13:22:36ZMarkus ScheidgenMerge for release of v1.0.4https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/594Resolve "ElasticSearch upgrade for v0.x"2022-05-17T14:06:13ZLauri HimanenResolve "ElasticSearch upgrade for v0.x"Closes #767
@mscheidg: I did not update any of the helm stuff, but otherwise this should now include all the necessary changes.Closes #767
@mscheidg: I did not update any of the helm stuff, but otherwise this should now include all the necessary changes.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/593Implemented new full parsers, minor metainfo updates2022-03-24T10:33:25ZAlvin Noe LadinesImplemented new full parsers, minor metainfo updatesdl_poly, cpmd full parsersdl_poly, cpmd full parsersMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/592tabular tree2022-03-15T13:59:55ZFelix Dietrichtabular treeTabular tree metaschema for flexible metaschema definitions during upload.Tabular tree metaschema for flexible metaschema definitions during upload.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/591Add async interface2022-03-15T11:09:31ZTheodore ChangAdd async interfaceCloses #763Closes #763Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/590Added a new util prod deployment.2022-03-15T11:45:15ZMarkus ScheidgenAdded a new util prod deployment.This adds another manual task to the CI/CD "prod deploy util". This allows to deploy the branch to a nomad installation at: https://nomad-lab.eu/prod/v1/util
This installation uses the full nomad data and can be used for a variety of te...This adds another manual task to the CI/CD "prod deploy util". This allows to deploy the branch to a nomad installation at: https://nomad-lab.eu/prod/v1/util
This installation uses the full nomad data and can be used for a variety of tests, e.g. benchmark measurement.
The app is scaled very small to just one replica with a single worker. This hopefully makes it easier to test, as all requests should go to the same process on the server. See `ops/helm/nomad/deployments/prod-util-values.yaml` to change this.Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/589EditMembers accessibility (#764)2022-03-14T15:25:40ZMohammad NakhaeeEditMembers accessibility (#764)Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/588GUI_Test_UploadsPage2022-03-29T06:39:35ZMohammad NakhaeeGUI_Test_UploadsPageAdd GUI tests for uploads page, upload page, and members edit dialog.Add GUI tests for uploads page, upload page, and members edit dialog.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/587Add async interface2022-03-15T09:50:39ZTheodore ChangAdd async interfaceCloses #763
Add async interface that can be directly used in an async environment.Closes #763
Add async interface that can be directly used in an async environment.Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/586Resolve "Missing/wrong constant shifts in unitsData.json"2022-03-14T09:32:07ZLauri HimanenResolve "Missing/wrong constant shifts in unitsData.json"Closes #762Closes #762Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/585Support parsers creating multiple entries from the same file2022-03-21T08:15:16ZDavid SikterSupport parsers creating multiple entries from the same fileCloses #761
We make it possible for an entry (associated with some mainfile) to have _child entries_. This is done by introducing a new field on the entry level: `mainfile_key`. Main entries have `mainfile_key == None`, child entries h...Closes #761
We make it possible for an entry (associated with some mainfile) to have _child entries_. This is done by introducing a new field on the entry level: `mainfile_key`. Main entries have `mainfile_key == None`, child entries have the same value for `mainfile` as the main/parent entry, plus some non-empty string as value for the `mainfile_key`. Note, however, that most parsers will only produce a main entry without any child entries.
Both the main and the child entries are full-fledged entries, i.e. they are distinct objects in mongo and elastic search, they have their own archive files, their own metadata, and so on.
The combination `(upload_id, mainfile, mainfile_key)` uniquely identifies any entry. For every child entry, a main entry must exist (i.e. an entry with the same `upload_id` and `mainfile`, but with `mainfile_key == None`).
Parsers signal that they want to create child entries by returning a set of keys, one for each child, from the `is_mainfile` function (instead of a boolean, like we have done up until now). Parsers that don't want to create any child entries can just return True, as before.
The parse function is called only once, for the main entry, and we pass an additional argument to it: `child_archives`, a dictionary of the format `{mainfile_key: child_archive}`.David SikterDavid Sikterhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/584Resolve "ElasticSearch upgrade for v1.x"2022-03-18T13:22:27ZLauri HimanenResolve "ElasticSearch upgrade for v1.x"Closes #760
@mscheidg: Luckily I did not have to change that much, see the issue for the changes I did. How do we go from here? Will you just merge and then we deploy this to beta? The old indices should work, at least that's what [ES s...Closes #760
@mscheidg: Luckily I did not have to change that much, see the issue for the changes I did. How do we go from here? Will you just merge and then we deploy this to beta? The old indices should work, at least that's what [ES says in their docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/breaking-changes-7.0.html#_indices_created_before_7_0).Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/583Resolve "Improve ArchiveReader performance"2022-03-20T23:10:59ZTheodore ChangResolve "Improve ArchiveReader performance"Closes #747
Use anyio to implement async file I/O.
anyio handles nested event loops well (celery + asyncio).Closes #747
Use anyio to implement async file I/O.
anyio handles nested event loops well (celery + asyncio).Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/582Edit Quantity Components (#740)2022-03-05T08:50:17ZMohammad NakhaeeEdit Quantity Components (#740)Origin: #719Origin: #719Mohammad NakhaeeMohammad Nakhaee