nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2024-03-27T17:30:38Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1717Move pynxtools and plugins into parsing packages2024-03-27T17:30:38ZFlorian DobenerMove pynxtools and plugins into parsing packagesThis adds pynxtools and respective plugins to the `parsing` section of `pyproject.toml` and regenerates the requirements files with it. It also removes the nexus submodule as this is no longer needed.
This is a _hotfix_ to have the newe...This adds pynxtools and respective plugins to the `parsing` section of `pyproject.toml` and regenerates the requirements files with it. It also removes the nexus submodule as this is no longer needed.
This is a _hotfix_ to have the newest examples ready for DPG and proper pluginisation of pynxtools will be handled in !1629.
As discussed with @lukaspie @sanbrock.
ToDo:
- [x] Update `pynxtools==0.1.0`Florian DobenerFlorian Dobenerhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1715Resolve "Include custom quantities from the Perovskite Database schema"2024-03-13T09:09:02ZJose Marquez PrietoResolve "Include custom quantities from the Perovskite Database schema"Closes #1926Closes #1926Jose Marquez PrietoJose Marquez Prietohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1714Fixed gui data etag.2024-03-11T20:10:22ZMarkus ScheidgenFixed gui data etag.Changelog: FixedChangelog: Fixedhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1712copy local path to temp directory2024-03-07T12:54:20ZAhmed Ilyascopy local path to temp directoryIf a local path is provided, copy it to a temp dir before importing. Otherwise when the upload is deleted, the original local path file ends up getting deleted too.If a local path is provided, copy it to a temp dir before importing. Otherwise when the upload is deleted, the original local path file ends up getting deleted too.Ahmed IlyasAhmed Ilyashttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1711Add raw_path_exists method to ClientContext2024-03-06T14:23:57ZSarthak KapoorAdd raw_path_exists method to ClientContext`nomad.datamodel.Context` has a `raw_path_exists` method which needs to implemented in it's child classes. But this implementation is missing in one of the child classes `ClientContext`. The current MR fixes it.
_Why is this needed?_ Wh...`nomad.datamodel.Context` has a `raw_path_exists` method which needs to implemented in it's child classes. But this implementation is missing in one of the child classes `ClientContext`. The current MR fixes it.
_Why is this needed?_ When testing a schema class locally using pytest, the context associated with the archive is of class `ClientContext`. If the schema uses `archive.m_context.raw_path_exists(filepath)` in normalize, the test fails as the method is not implemented.Sarthak KapoorSarthak Kapoorhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1710Improved app and helm chart with respect to timeouts and rate limiting.2024-03-11T20:49:57ZMarkus ScheidgenImproved app and helm chart with respect to timeouts and rate limiting.This partially helps with: #1914, it includes the changes of !1701
Changes:
- Timeouts are now consistently applied to ingress and proxy rules based on shared values
- Separate ingress for api and others (gui, docs) for tighter rate lim...This partially helps with: #1914, it includes the changes of !1701
Changes:
- Timeouts are now consistently applied to ingress and proxy rules based on shared values
- Separate ingress for api and others (gui, docs) for tighter rate limits at the api
- Concurrent connections limit in addition to connection per second limit
- ArchiveQuery defaults fit the timeout and rate limiting settings
- Increased the HPC cloud loadbalancer timeouts to be slightly longer than the nomad timeouts (not this MR)
- Removed the joblib based threading for multi entry archive apis. This was a noop due to GIL.
- Added an await call into the multi entry archive loop, allowing requests (e.g. probes) during a running multi entry archive call.
- Multi entry archive apis stop computing the requested archive list after a client disconnect.
- refactored the main app, because HTTP middlewares are prohibiting recognising client disconnects (https://github.com/encode/starlette/discussions/2094). Now the api does not use any HTTP middleware
- more consistent use of parameter free events in api logging
Solutions:
- The app now does stop when a request is canceled (e.g. via timeout).
- Timeouts are a consistent 60s and the rate limit is set to 10 concurrent api requests and 32 requests per second.
- The long running multi entry archive api calls allow concurrent requests. This already worked for all downloads via the used StreamingResponses.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1709updated submodules and gui artifacts2024-03-05T14:30:33ZJoseph Rudzinskiupdated submodules and gui artifactsCloses #1921Closes #1921Joseph RudzinskiJoseph Rudzinskihttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1703Reverted automated re-lableing of sub sections in archive browser and ELN.2024-03-04T16:33:41ZMarkus ScheidgenReverted automated re-lableing of sub sections in archive browser and ELN.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1702Resolve "errors in the gui whenquantitities are of type=np.float64 and no uni...2024-03-04T15:04:11ZMarkus ScheidgenResolve "errors in the gui whenquantitities are of type=np.float64 and no unit given"Closes #1913Closes #1913https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1699Add functionality to convert old archives to new ones2024-03-12T22:03:03ZTheodore ChangAdd functionality to convert old archives to new onesMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1697Changed loader type.2024-03-01T11:53:24ZLauri HimanenChanged loader type.Closes #1911Closes #1911Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1696Final revision to 1887 update developing plugin documentation2024-03-11T14:32:46ZJoseph RudzinskiFinal revision to 1887 update developing plugin documentationThis will be the last revision to #1887 before merging. There are still a number of changes being made based on the discussions in the MR [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1666/diffs#fea849f6bbd6379...This will be the last revision to #1887 before merging. There are still a number of changes being made based on the discussions in the MR [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1666/diffs#fea849f6bbd6379c645bc004194ddc05c634399f).
@pizarroj While I am waiting for feedback from Alvin, could you take a look at the computational section and provide a review? (No rush, I am not expecting to hear back from Alvin till at least next week).Joseph RudzinskiJoseph Rudzinskihttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1692Changed config to use the new archive writer by default.2024-03-09T03:03:52ZMarkus ScheidgenChanged config to use the new archive writer by default.Changelog: AddedChangelog: AddedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1690Loop elastic check2024-02-27T18:39:13ZTheodore ChangLoop elastic checkTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1689saving and loading nexus schema instead of building it from NXDL to save abou...2024-03-01T17:42:48ZSandor Brockhausersaving and loading nexus schema instead of building it from NXDL to save about 50%https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1685Update group tests2024-02-27T11:07:18ZSascha KlawohnUpdate group tests- Streamline group tests, creation and update tests.
- Add tests for group name restrictions.
- Test dropping duplicate members.
- Change members field from set to list for compatibility.- Streamline group tests, creation and update tests.
- Add tests for group name restrictions.
- Test dropping duplicate members.
- Change members field from set to list for compatibility.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1683Add *.json/*.yaml config files for pynxtools readers2024-02-27T09:39:41ZFlorian DobenerAdd *.json/*.yaml config files for pynxtools readersSome readers have *.json files for configuration. This adds them to nomads `MANIFEST.in` so they are copied into the package.Some readers have *.json files for configuration. This adds them to nomads `MANIFEST.in` so they are copied into the package.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1680Ensure RFC3161 server is reachable2024-02-21T18:53:59ZTheodore ChangEnsure RFC3161 server is reachableTheodore ChangTheodore Changhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1679Added a publish command and util deployment.2024-02-23T10:24:47ZMarkus ScheidgenAdded a publish command and util deployment.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1675Added a simple locust load test script.2024-02-23T10:52:54ZMarkus ScheidgenAdded a simple locust load test script.