nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2020-07-16T11:01:20Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1340.8.2 release2020-07-16T11:01:20ZMarkus Scheidgen0.8.2 releasehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/16011547-add-support-for-h5md-files2024-01-17T15:11:47ZJoseph Rudzinski1547-add-support-for-h5md-filespartner changes to dev of h5md parser: https://github.com/nomad-coe/atomistic-parsers/pull/71
Closes #1547partner changes to dev of h5md parser: https://github.com/nomad-coe/atomistic-parsers/pull/71
Closes #1547Joseph RudzinskiJoseph Rudzinskihttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/894820 - Oasislink from local to central as TCP server2023-07-05T11:08:40ZDaniel Lehmberg820 - Oasislink from local to central as TCP server@mscheidg
* logtransfer and federation FastAPI backend mostly finished and tested (see open points below)
* rudimentary class for additional statistics (queries to mongodb) that can be transferred
- currently this is located in ``lo...@mscheidg
* logtransfer and federation FastAPI backend mostly finished and tested (see open points below)
* rudimentary class for additional statistics (queries to mongodb) that can be transferred
- currently this is located in ``logtransfer.py`` (it shouldn't be within the main Nomad API as this can run with multiple workers, which turned out to be problematic in the past)
- we agreed that we use reasonable cases from the KPI as well as statistics required for the homepage
* Check documentation for data sharing in oasis.md
* The test case that sometimes used failed for remote CI pipeline is (hopefully) fixed
---
Relates to issues #820 and #886
This merge request follows up the closed merge requests !697, !727 and !784.
Here the logtransfer is implemented in a separate service to the Nomad app - a TCP Server (Python native classes). In particular the service runs on a local Nomad Oasis and receives logs on the same address and port than where logstash logs are sent on the central Oasis.
For Nomad Oasis the server running on logtransfer can therefore be interpreted as a logstash proxy. Unless running the logstash instance itself, all logs are submitted to the central Oasis where the logs are then eventually stored in logstash.
Open questions:
* [x] To identify a local Oasis we read the IP address on the FastAPI receiving end. Do we still need to adapt the field [`deployment_id`](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/blob/develop/nomad/config.py#L357) per local instance (we would make sure that this is unique -- maybe taking a hash from some config parameters)?
To clarify:
* [x] A problem for testing is currently, that when the app is started with `nomad app run app`, then there the logs are different set than to the log config in docker. Should this maybe be fixed? (If starting Nomad with `nomad app run app`, then no API calls are processed in logstash format).
* [x] Use threading TCP server? If a client does not close the connection to the logstash proxy server, it can block the whole server. `ThreadingTCP` server is only supported for UNIX systems.
* [ ] Validate IP address that was received in federation/logs?
* [ ] Protect against malicious (gzipped) content?
* against too large data (load into main memory)
* against gzip bombs (uncompress too much data into main memory)
* too many newlines in the data (long running loop)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/156About2020-08-21T07:16:37ZMarkus ScheidgenAboutImproved the about page with actual encyclopedia and ai toolkit info.Improved the about page with actual encyclopedia and ai toolkit info.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/738About page (#904)2022-06-30T06:32:01ZMohammad NakhaeeAbout page (#904)Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/257Adapted ci for new dev cluster.2021-02-08T21:58:05ZMarkus ScheidgenAdapted ci for new dev cluster.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/252Adapted code for numpy>1.20.0. #4932021-02-02T10:43:51ZMarkus ScheidgenAdapted code for numpy>1.20.0. #493https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/389Adapted lobster and aflow parser2021-09-17T07:13:32ZAlvin Noe LadinesAdapted lobster and aflow parserAdapted parsers to new metainfo, extended aflow to include phonon module.Adapted parsers to new metainfo, extended aflow to include phonon module.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1547Add additional ruff check to print linting errors to console2023-11-09T14:30:07ZAhmed IlyasAdd additional ruff check to print linting errors to consoleThis MR adds an additional ruff check to the CI. Ruff is very fast so another 50ms doesn't harm us.
This way we will maintain the old behavior of having the linting errors in the console output in the linting pipeline in addition to ha...This MR adds an additional ruff check to the CI. Ruff is very fast so another 50ms doesn't harm us.
This way we will maintain the old behavior of having the linting errors in the console output in the linting pipeline in addition to having a Code Quality report in merge requests.Ahmed IlyasAhmed Ilyashttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/627Add AI CoursePage to analyze section2022-04-06T06:48:21ZMarkus ScheidgenAdd AI CoursePage to analyze sectionhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/875Add all inter-archive reference into archive metadata2022-10-12T06:10:37ZTheodore ChangAdd all inter-archive reference into archive metadata```
archive_references:
target_reference
target_entry_id
target_name
target_path
source_name
source_path
```
The detection of references is consistent with resolving in context.
Closes #999```
archive_references:
target_reference
target_entry_id
target_name
target_path
source_name
source_path
```
The detection of references is consistent with resolving in context.
Closes #999Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1472Add AppTokenForm to APIs page2023-10-02T15:28:18ZSascha KlawohnAdd AppTokenForm to APIs pageCloses #1681
TODO: Fix typo when squashing (AppFormToken -> AppTokenForm)Closes #1681
TODO: Fix typo when squashing (AppFormToken -> AppTokenForm)Sascha KlawohnSascha Klawohnhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/591Add async interface2022-03-15T11:09:31ZTheodore ChangAdd async interfaceCloses #763Closes #763Theodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/994Add birch_murnaghan to results.eos fit functions2022-12-16T15:34:58ZAlvin Noe LadinesAdd birch_murnaghan to results.eos fit functionsFixes issue with TypeError: The value birch_murnaghan is not an enum value for quantity nomad.datamodel.results.EnergyVolumeCurve.type (nomad github issue 37)Fixes issue with TypeError: The value birch_murnaghan is not an enum value for quantity nomad.datamodel.results.EnergyVolumeCurve.type (nomad github issue 37)Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/969Add bopfox parser (removed)2022-12-12T17:22:42ZAlvin Noe LadinesAdd bopfox parser (removed)Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1459Add check for `normalization_factors` for the `Normalize intensities` optino...2023-09-11T08:35:42ZNathan DaelmanAdd check for `normalization_factors` for the `Normalize intensities` optino...Add check for `normalization_factors` for the `Normalize intensities` option under the DOS kebab menu
Closes #1663Add check for `normalization_factors` for the `Normalize intensities` option under the DOS kebab menu
Closes #1663Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/412Add cli command to quarantine certain raw files. #5902021-10-13T06:21:54ZMarkus ScheidgenAdd cli command to quarantine certain raw files. #590https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1649Add coauthor, reviewer groups2024-01-29T13:47:24ZSascha KlawohnAdd coauthor, reviewer groups- Add user groups, API `/groups`
- Add coauthor, reviewer groups
- Add writer, viewer groups
- Add groups to ES owner query
- Add tests for groups, uploads, search
Closes #1760- Add user groups, API `/groups`
- Add coauthor, reviewer groups
- Add writer, viewer groups
- Add groups to ES owner query
- Add tests for groups, uploads, search
Closes #1760Sascha KlawohnSascha Klawohnhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1553Add computational core-hole metadata2023-12-20T06:12:10ZNathan DaelmanAdd computational core-hole metadataAdd `CoreHole` under `AtomParameters`, but with an interface to `AtomsGroup` and by extension topology.Add `CoreHole` under `AtomParameters`, but with an interface to `AtomsGroup` and by extension topology.Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1429Add config key to mount external volumes into north tool2023-08-03T07:52:21ZFlorian DobenerAdd config key to mount external volumes into north toolThis basically readds a feature to mount external folders into a north tools container. It was removed during the refactoring of the north config (see https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/commit/b0e28fa7fd3f47fcd538f33bd84b...This basically readds a feature to mount external folders into a north tools container. It was removed during the refactoring of the north config (see https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/commit/b0e28fa7fd3f47fcd538f33bd84bad659ee6549a#0c8c279f7dd1dae6be962aa1e764f2e07e029710_272_142).
Example config (under north):
```
north:
external_mounts:
- host_path: /mnt/pcshare
bind: pcshare
mode: ro
- host_path: /scratch/localDataStore:
bind: localDataStore
mode: rw
```