nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2024-03-15T00:49:13Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1733Resolve "add reference to elabftw parser"2024-03-15T00:49:13ZAmir GolparvarResolve "add reference to elabftw parser"Closes #1937Closes #1937Amir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1732Increased helm chart version.2024-03-15T11:19:44ZMarkus ScheidgenIncreased helm chart version.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1730Add pytest options for faster local testing2024-03-18T10:05:01ZSascha KlawohnAdd pytest options for faster local testingThe `celery.inspect.().active()` broadcast used in the cleanup of fixture `worker` waits until timeout (default 1 s) is reached, which accumulates for each test with this fixture, e.g. about 14 min instead of 7 min on a laptop. Removal m...The `celery.inspect.().active()` broadcast used in the cleanup of fixture `worker` waits until timeout (default 1 s) is reached, which accumulates for each test with this fixture, e.g. about 14 min instead of 7 min on a laptop. Removal may lead to tasks still running during the next test.
This MR adds the pytest option `--celery-inspect-timeout`, which can be used to decrease the timeout for faster local testing.
It further adds the option `--fixture-filters` to run only tests that depend (directly or indirectly) on a list of fixtures and do not depend on fixtures prefixed with '!'.Sascha KlawohnSascha Klawohnhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1729Resolve "Config refactor"2024-03-15T09:20:46ZLauri HimanenResolve "Config refactor"Closes #1917Closes #1917Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1728Revert "Merge branch '1917-delay-config-validation' into 'develop'"2024-03-14T10:23:55ZLauri HimanenRevert "Merge branch '1917-delay-config-validation' into 'develop'"This reverts merge request !1716This reverts merge request !1716https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1727Fixed broken error handling on datacite errors.2024-03-13T21:02:00ZMarkus ScheidgenFixed broken error handling on datacite errors.Changelog: FixedChangelog: FixedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1725Added entry create time as an available search filter.2024-03-13T08:15:28ZLauri HimanenAdded entry create time as an available search filter.Closes #1931Closes #1931Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1724Archive converter refinement2024-03-15T10:06:20ZTheodore ChangArchive converter refinementTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1722Added new troubleshooting section to OASIS installation.2024-03-12T12:21:58ZLauri HimanenAdded new troubleshooting section to OASIS installation.Closes #1927Closes #1927Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1721Remove cache2024-03-21T15:05:25ZTheodore ChangRemove cacheTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1720Resolve "data structure of the .eln file format has changed"2024-03-12T12:37:34ZAmir GolparvarResolve "data structure of the .eln file format has changed"Closes #1928Closes #1928Amir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1718Documentation: Tiny improvement and typo fix2024-03-11T13:14:18ZFabian SchöppachDocumentation: Tiny improvement and typo fixhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1717Move pynxtools and plugins into parsing packages2024-03-27T17:30:38ZFlorian DobenerMove pynxtools and plugins into parsing packagesThis adds pynxtools and respective plugins to the `parsing` section of `pyproject.toml` and regenerates the requirements files with it. It also removes the nexus submodule as this is no longer needed.
This is a _hotfix_ to have the newe...This adds pynxtools and respective plugins to the `parsing` section of `pyproject.toml` and regenerates the requirements files with it. It also removes the nexus submodule as this is no longer needed.
This is a _hotfix_ to have the newest examples ready for DPG and proper pluginisation of pynxtools will be handled in !1629.
As discussed with @lukaspie @sanbrock.
ToDo:
- [x] Update `pynxtools==0.1.0`Florian DobenerFlorian Dobenerhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1716Resolve "Config refactor"2024-03-14T10:19:46ZLauri HimanenResolve "Config refactor"Closes #1917Closes #1917Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1715Resolve "Include custom quantities from the Perovskite Database schema"2024-03-13T09:09:02ZJose Marquez PrietoResolve "Include custom quantities from the Perovskite Database schema"Closes #1926Closes #1926Jose Marquez PrietoJose Marquez Prietohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1714Fixed gui data etag.2024-03-11T20:10:22ZMarkus ScheidgenFixed gui data etag.Changelog: FixedChangelog: Fixedhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1712copy local path to temp directory2024-03-07T12:54:20ZAhmed Ilyascopy local path to temp directoryIf a local path is provided, copy it to a temp dir before importing. Otherwise when the upload is deleted, the original local path file ends up getting deleted too.If a local path is provided, copy it to a temp dir before importing. Otherwise when the upload is deleted, the original local path file ends up getting deleted too.Ahmed IlyasAhmed Ilyashttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1711Add raw_path_exists method to ClientContext2024-03-06T14:23:57ZSarthak KapoorAdd raw_path_exists method to ClientContext`nomad.datamodel.Context` has a `raw_path_exists` method which needs to implemented in it's child classes. But this implementation is missing in one of the child classes `ClientContext`. The current MR fixes it.
_Why is this needed?_ Wh...`nomad.datamodel.Context` has a `raw_path_exists` method which needs to implemented in it's child classes. But this implementation is missing in one of the child classes `ClientContext`. The current MR fixes it.
_Why is this needed?_ When testing a schema class locally using pytest, the context associated with the archive is of class `ClientContext`. If the schema uses `archive.m_context.raw_path_exists(filepath)` in normalize, the test fails as the method is not implemented.Sarthak KapoorSarthak Kapoorhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1710Improved app and helm chart with respect to timeouts and rate limiting.2024-03-11T20:49:57ZMarkus ScheidgenImproved app and helm chart with respect to timeouts and rate limiting.This partially helps with: #1914, it includes the changes of !1701
Changes:
- Timeouts are now consistently applied to ingress and proxy rules based on shared values
- Separate ingress for api and others (gui, docs) for tighter rate lim...This partially helps with: #1914, it includes the changes of !1701
Changes:
- Timeouts are now consistently applied to ingress and proxy rules based on shared values
- Separate ingress for api and others (gui, docs) for tighter rate limits at the api
- Concurrent connections limit in addition to connection per second limit
- ArchiveQuery defaults fit the timeout and rate limiting settings
- Increased the HPC cloud loadbalancer timeouts to be slightly longer than the nomad timeouts (not this MR)
- Removed the joblib based threading for multi entry archive apis. This was a noop due to GIL.
- Added an await call into the multi entry archive loop, allowing requests (e.g. probes) during a running multi entry archive call.
- Multi entry archive apis stop computing the requested archive list after a client disconnect.
- refactored the main app, because HTTP middlewares are prohibiting recognising client disconnects (https://github.com/encode/starlette/discussions/2094). Now the api does not use any HTTP middleware
- more consistent use of parameter free events in api logging
Solutions:
- The app now does stop when a request is canceled (e.g. via timeout).
- Timeouts are a consistent 60s and the rate limit is set to 10 concurrent api requests and 32 requests per second.
- The long running multi entry archive api calls allow concurrent requests. This already worked for all downloads via the used StreamingResponses.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1709updated submodules and gui artifacts2024-03-05T14:30:33ZJoseph Rudzinskiupdated submodules and gui artifactsCloses #1921Closes #1921Joseph RudzinskiJoseph Rudzinski