nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2021-03-10T07:47:36Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/268Implemented new gaussian parser2021-03-10T07:47:36ZAlvin Noe LadinesImplemented new gaussian parserMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/273Implemented new gpaw parser2021-03-11T07:27:11ZAlvin Noe LadinesImplemented new gpaw parserImplemented unified parser for gpw and gpw2Implemented unified parser for gpw and gpw2Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/173Implemented new lammps parser2020-09-16T06:33:53ZAlvin Noe LadinesImplemented new lammps parserNew lammps parser using metainfo. Added metainfo quantities to accommodate lammps outputsNew lammps parser using metainfo. Added metainfo quantities to accommodate lammps outputsMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/275Implemented new octopus parser2021-03-12T07:53:30ZAlvin Noe LadinesImplemented new octopus parserMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1319Implement eos fit normalization #14392023-06-08T19:20:53ZAlvin Noe LadinesImplement eos fit normalization #1439Closes #1439Closes #1439Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1023Implement extended matching of h5 file #12682023-01-10T12:48:44ZAlvin Noe LadinesImplement extended matching of h5 file #1268Closes #1268Closes #1268Jose PizarroJose Pizarrohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1040Implement handling of large data in archive #12162023-03-03T08:06:31ZAlvin Noe LadinesImplement handling of large data in archive #1216Implement serialization of large datasets into hdf5 file. On the parser side, implemented parsing of CHGCAR file.Implement serialization of large datasets into hdf5 file. On the parser side, implemented parsing of CHGCAR file.Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1076Implement parsing of xml output in fleur2023-02-03T09:19:49ZAlvin Noe LadinesImplement parsing of xml output in fleurAddress user request for parsing of xml outputs in fleur.Address user request for parsing of xml outputs in fleur.Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/681Implement sampling of large traj in fhiaims2022-05-19T12:53:08ZAlvin Noe LadinesImplement sampling of large traj in fhiaimsMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1211Implement static nodes for workflow visualizer2023-05-03T12:47:53ZAlvin Noe LadinesImplement static nodes for workflow visualizerCloses #1420Closes #1420Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1710Improved app and helm chart with respect to timeouts and rate limiting.2024-03-11T20:49:57ZMarkus ScheidgenImproved app and helm chart with respect to timeouts and rate limiting.This partially helps with: #1914, it includes the changes of !1701
Changes:
- Timeouts are now consistently applied to ingress and proxy rules based on shared values
- Separate ingress for api and others (gui, docs) for tighter rate lim...This partially helps with: #1914, it includes the changes of !1701
Changes:
- Timeouts are now consistently applied to ingress and proxy rules based on shared values
- Separate ingress for api and others (gui, docs) for tighter rate limits at the api
- Concurrent connections limit in addition to connection per second limit
- ArchiveQuery defaults fit the timeout and rate limiting settings
- Increased the HPC cloud loadbalancer timeouts to be slightly longer than the nomad timeouts (not this MR)
- Removed the joblib based threading for multi entry archive apis. This was a noop due to GIL.
- Added an await call into the multi entry archive loop, allowing requests (e.g. probes) during a running multi entry archive call.
- Multi entry archive apis stop computing the requested archive list after a client disconnect.
- refactored the main app, because HTTP middlewares are prohibiting recognising client disconnects (https://github.com/encode/starlette/discussions/2094). Now the api does not use any HTTP middleware
- more consistent use of parameter free events in api logging
Solutions:
- The app now does stop when a request is canceled (e.g. via timeout).
- Timeouts are a consistent 60s and the rate limit is set to 10 concurrent api requests and 32 requests per second.
- The long running multi entry archive api calls allow concurrent requests. This already worked for all downloads via the used StreamingResponses.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/762Improved/corrected url parsing utils and documentation2022-08-03T05:58:26ZDavid SikterImproved/corrected url parsing utils and documentationImprovement of documentation and utilities for handling a harmonized "nomad url" concept. Part of the work with #828.Improvement of documentation and utilities for handling a harmonized "nomad url" concept. Part of the work with #828.David SikterDavid Sikterhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1009Improved documentation and handling of config classes2022-12-22T16:59:57ZDavid SikterImproved documentation and handling of config classesDid some minor improvements to the doc strings.
I also noted that if type annotation isn't added to the `NomadConfig` classes, the attributes might be listed in a strange order in the generated documentation, different from the order in...Did some minor improvements to the doc strings.
I also noted that if type annotation isn't added to the `NomadConfig` classes, the attributes might be listed in a strange order in the generated documentation, different from the order in which the attributes are defined, so I added some type annotations to make sure the logical order is preserved.
I also took myself the liberty to change the behaviour of the `customize` method so that it can take either a `NomadConfig` object or a dictionary as its sole argument (this meant I could simplify the code in some places, and I think it is cleaner. The important thing is that the dict is converted to an instance of the right `NomadConfig` class before we start accessing various settings, and I think this should be the responsibility of the `customize` method. This method also does all the checks in one place.) There was also a bug in the CLI commands relating to this (fixed now).David SikterDavid Sikterhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/136Improved DOS normalization2020-07-20T13:05:01ZLauri HimanenImproved DOS normalizationFixed issues with DOS normalization and added an extended description to the metainfo description about the normalization procedure.Fixed issues with DOS normalization and added an extended description to the metainfo description about the normalization procedure.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/495Improved incremental processing and upload API2021-12-21T15:16:19ZDavid SikterImproved incremental processing and upload APICloses #587. Introduces an additional `path_filter` argument to the `process_upload` that enables limiting the processing to a specific path. This is used to optimize the API calls to put and delete individual files and folders at the sp...Closes #587. Introduces an additional `path_filter` argument to the `process_upload` that enables limiting the processing to a specific path. This is used to optimize the API calls to put and delete individual files and folders at the specified raw path.David SikterDavid Sikterhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/752Improved metainfo browser GUI tests2022-07-12T11:23:54ZDavid SikterImproved metainfo browser GUI tests- Makes the gui tests in the CI pipeline run with the `--silent` flag, to reduce the noise.
- Changed the metainfo browser test to a pseudo-random walk. Rather than changing the timeout, in the future, we should adjust the probability de...- Makes the gui tests in the CI pipeline run with the `--silent` flag, to reduce the noise.
- Changed the metainfo browser test to a pseudo-random walk. Rather than changing the timeout, in the future, we should adjust the probability decay factor, which determines how many nodes will be visited.
- The test also computes a hash for the metainfo tree (the parts visited), to give the user a notice if the tree changes.
- If tree changes are expected, the user will have to update the test case with the new hash.
- Maybe requiring the user to update the hash in this way is a bit cumbersome, but let's test to see how it works.David SikterDavid Sikterhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/460Improved migrate cli2021-11-22T15:20:23ZDavid SikterImproved migrate cliA couple of small improvements of the migrate CLI command. Resolves #652 and #650.A couple of small improvements of the migrate CLI command. Resolves #652 and #650.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1075Improved overview UX2023-02-09T06:28:52ZLauri HimanenImproved overview UXImproved overview UX.
Closes #1312Improved overview UX.
Closes #1312Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/567Improved preview2022-02-22T07:49:02ZDavid SikterImproved previewSome minor fixes and improvements of the file browser and preview functionality:
- the `FilePreview` component is moved to a separate file
- code is refactored by introducing viewer objects, one for each file type we support. The viewer ...Some minor fixes and improvements of the file browser and preview functionality:
- the `FilePreview` component is moved to a separate file
- code is refactored by introducing viewer objects, one for each file type we support. The viewer object stores constant information, like matching file extensions, maxSizeAutoPreview etc, plus a `render` function.
- a viewer is introduced for json files (the same viewer used elsewhere).
- txt, yaml and yml files are automatically previewed with the text viewer
- error handling is added. If the viewer fails, we show a message and offers the user to view the file using the text viewer instead.
- optimizing the vertical file browser api calls, using the `include_entry_info` option instead of making a costly second api call.
- Highlighting the current entry mainfileMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/563Improved preview, including images and automatically previewing small files (...2022-02-16T07:43:48ZDavid SikterImproved preview, including images and automatically previewing small files (#738)Adds support for simple image preview, and paves the way for adding viewers for additional file types. Images are previewed automatically if they are small enough (<10 MB), otherwise the user has to request it by clicking on a preview bu...Adds support for simple image preview, and paves the way for adding viewers for additional file types. Images are previewed automatically if they are small enough (<10 MB), otherwise the user has to request it by clicking on a preview button. For text files (the fallback viewer) the user always has to press the button to open the viewer.
Also solves some issues with the text preview scrolling.Markus ScheidgenMarkus Scheidgen