nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2021-07-28T14:35:31Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/361Implemented fixes for phonopy and fhi-aims2021-07-28T14:35:31ZAlvin Noe LadinesImplemented fixes for phonopy and fhi-aimsImplemented fixes for phonopy and fhi aimsImplemented fixes for phonopy and fhi aimshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/362New sync api2021-09-04T15:51:45ZDavid SikterNew sync apiIntroduces new functionality for downloading and uploading "upload bundles" from/to NOMAD deployments. Intended to replace existing CLI and publish-from-oasis functionality.
An "upload bundle" is a file bundle (a zipfile, normally) cont...Introduces new functionality for downloading and uploading "upload bundles" from/to NOMAD deployments. Intended to replace existing CLI and publish-from-oasis functionality.
An "upload bundle" is a file bundle (a zipfile, normally) containing all information (raw files, archive files, metadata) needed for transferring an upload from one NOMAD deployment to another. They are also intended for internal operations and for future CLI commands for doing things like "pulling" all new uploads from the central NOMAD to an oasis.
The main methods for exporting and importing are defined in `nomad.processing.Upload`, named `export_bundle` and `import_bundle`, respectively. They are quite parametrized (maybe too parametrized?) to allow for flexibility. The import method does more extensive checking than the existing functionality for importing uploads from OASIS. In the future, we might also want to include code for conversion between different nomad versions.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/363Fixed orjson and uvicorn dependencies.2021-08-06T06:57:09ZLauri HimanenFixed orjson and uvicorn dependencies.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/364V0.10.52021-08-06T11:53:06ZMarkus ScheidgenV0.10.5https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/365New Search Interface2021-08-09T14:04:34ZLauri HimanenNew Search InterfaceCompletely rewritten search interface and search context for entries. Major changes include:
- New components for advanced querying:
- FilterText.js: freeform text search with suggestion
- FilterSelect.js: Dropdown for search options...Completely rewritten search interface and search context for entries. Major changes include:
- New components for advanced querying:
- FilterText.js: freeform text search with suggestion
- FilterSelect.js: Dropdown for search options with large number of available values
- FilterCheckboxes.js: Checkboxes for search options with small number of available values
- FilterSlider.js: Two-handled slider for selecting a numeric range
- FilterDate.js: For selecting a date-range.
- FilterContext.js: provides several custom hooks for interacting with the current set of search filters. Optimized usage of Recoil.js Atoms and Selectors that minimize the number of renders when filters are updated. Also has a built-in debounce mechanism.
- Improved unit conversion tools (needed especially for recalculating Filter values from the current unit system to SI and back).
- Completely rewritten search bar: Now provides meaningful suggestions, is synced with the filters shown on the left panel, supports simple range queries, has a primitive validation mechanism.
Also fixes the following API issues:
- Fixed issues with range validation when using gte, lte queries for numeric values and timestamps.
- Fixed issue with invalid 'next_page_after_value' reported by the API.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/366V1.0.0 bugfixes2021-08-11T11:46:13ZMarkus ScheidgenV1.0.0 bugfixes@himanel1 Can you review the GUI changes, especially on the unit conversion.
- omit `section_` from section names
- limit the width of lanes, e.g. to limit long formulas or other strings
- fix exceptions on unit conversions
- minor sim...@himanel1 Can you review the GUI changes, especially on the unit conversion.
- omit `section_` from section names
- limit the width of lanes, e.g. to limit long formulas or other strings
- fix exceptions on unit conversions
- minor simplifications for dev setupLauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/367Added support for serializing/deserializing query to/from URL query string2021-08-16T05:56:38ZLauri HimanenAdded support for serializing/deserializing query to/from URL query stringNow the filter state is instantly update to the query string as a query is being built. The query string format is based on the API 'shortcut' syntax (e.g. `n_elements:gte=2`).
The exclusive/inclusive element queries are a bit tricky to...Now the filter state is instantly update to the query string as a query is being built. The query string format is based on the API 'shortcut' syntax (e.g. `n_elements:gte=2`).
The exclusive/inclusive element queries are a bit tricky to handle ATM, since they use entirely different fields. Would be great if the API could support an ':only'-prefix for elements. This would make it easy to switch between inclusive/exclusive mode (`elements:only=Na` vs. `elements:all=Na`).Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/368Added metadata to entries and added number of entries to upload in upload API...2021-08-16T06:29:16ZMarkus ScheidgenAdded metadata to entries and added number of entries to upload in upload API....Added metadata to entries and added number of entries to upload in upload API. Fixed issue with empty uploads processing.Added metadata to entries and added number of entries to upload in upload API. Fixed issue with empty uploads processing.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/369Refactored search interface code2021-08-19T09:52:52ZLauri HimanenRefactored search interface codeRefactored search interface code:
- New folder layout, improved filenames. Created new subfolders: `input`, `menus` and `results`. Renamed files so that the functionality can better be guessed by simply looking at the file name.
- Create...Refactored search interface code:
- New folder layout, improved filenames. Created new subfolders: `input`, `menus` and `results`. Renamed files so that the functionality can better be guessed by simply looking at the file name.
- Created custom components for `FilterMenu` and items in it (see `search/menus/FilterMenu.js`). This greatly simplified the layout of `search/menus/FilterMainMenu.js` making the addition of new menu items a breeze. These new components are structured very similarly to how MaterialUI deals with [Tabs](https://material-ui.com/components/tabs/#experimental-api).
- Added new "Access" menu and `InputRadio` component. Here one can control the visibility of the calculations (controls the `owner` parameter in API calls.)
- Refactored the `Actions` -component.
- Several small performance tweaks.
With this refactoring done, I will continue integrating the new search interface to Uploads/Datasets and removing the old search code.
@mscheidg: can you check that the new folder layout and filenames make sense?Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/370Refactored AppBar, MenuBar and FilterMenu.2021-08-24T07:50:54ZLauri HimanenRefactored AppBar, MenuBar and FilterMenu.- Restyled AppBar: inverted menu/title order, more compact the layout
- Small refactor of MenuBar, separated Consent and Breadcrumbs into their own files
- Renamed "Settings" to "Units"
- Improved the structure of FilterMenu
- Updated a ...- Restyled AppBar: inverted menu/title order, more compact the layout
- Small refactor of MenuBar, separated Consent and Breadcrumbs into their own files
- Renamed "Settings" to "Units"
- Improved the structure of FilterMenu
- Updated a few help texts
- Renamed some menu entries and their titles to be more consistent
- Improved developer documentation
- Improved filter panel sliding behaviour, decreased its width a bitMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/371Fixes to the sync api for oasis uploads2021-09-04T15:51:45ZDavid SikterFixes to the sync api for oasis uploadsA number of fixes and improvements for the new sync api so that it should now be fully able to handle uploads published from oasises.
- The entry id's provided in the bundle must agree with the automatically generated values, or the bun...A number of fixes and improvements for the new sync api so that it should now be fully able to handle uploads published from oasises.
- The entry id's provided in the bundle must agree with the automatically generated values, or the bundle will be rejected from import (previously, it was possible to provide any kind of string value as entry id in the oasis metadata file)
- Added the option to keep the bundle_info.json file when importing (true by default). The file will be stored in the root folder of the upload (not the raw folder, as before).
- Removed the old oasis functionality from the post upload endpoint in the fastapi (should not be used when we switch to the bundle-based functionality)
- Corrected handling of published_to (have kept it as a list of strings)
- Fixed an issue with the dataset import, and some minor polishingMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/372Routing, breadcrumbs, menu, experiment data, some upload ui, domain-less entry.2021-08-28T07:13:08ZMarkus ScheidgenRouting, breadcrumbs, menu, experiment data, some upload ui, domain-less entry.#574
#563#574
#563v1.0.0-betahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/373Metainfo refactor2021-09-03T11:02:25ZMarkus ScheidgenMetainfo refactor#419, #558, #557, #551, #540
This effects more of the source base due to heavy changes in the metainfo. Those include:
- removing `section_` prefixes
- renaming convoluted name like `section_single_configuration_calculation` -> `calcul...#419, #558, #557, #551, #540
This effects more of the source base due to heavy changes in the metainfo. Those include:
- removing `section_` prefixes
- renaming convoluted name like `section_single_configuration_calculation` -> `calculation`, renaming most refs, etc.
- changing the structure of major sections (`calculation`, `system`, `method`)
- creating sub packages for dft definitions
- lots of small changes to BS, DOS, k-points, energies
- lots of new workflows and stuff
Somethings should be changed again or should at least be discussed:
- [x] remove `...Reference` and just use plain references
- [x] method to method refs become subsection with plain references called core, main, ...
- [x] package run -> simulation
- [ ] `repeats=True` for sections that mostly not repeat, but for that one tiny case
- [x] some sections sub-sections feel over-engineered `system.chemical_formula_hill|reduced|anonymous`.
- [x] rename to `channel_info:ChannelInfo`
The following still does not work or is not tested:
- [ ] `test_api_encyclopedia.py` fails, i disabled it for now
- [x] all the property viewers are not working (except structure)
- [x] there are some assertions when loading `metainfo.js`
- [x] the viewers in the archive browser also not work, because the system, dos, ... layout has changed
- [x] the `OverviewView` test fails, i disabled it for nowhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/374Merge v0.10.6 for release2021-08-31T12:41:36ZMarkus ScheidgenMerge v0.10.6 for releasehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/375V1.0.0 metainfo refactor alvin2021-09-01T06:07:57ZAlvin Noe LadinesV1.0.0 metainfo refactor alvinImplemented suggested metainfo changes.Implemented suggested metainfo changes.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/376Moving the embargo setting to the upload level, tied to the embargo_length2021-09-04T16:29:20ZDavid SikterMoving the embargo setting to the upload level, tied to the embargo_lengthMoving the embargo setting to the upload level, controlled using the embargo_length attribute, 0 meaning no embargo.
The idea is that henceforth one should never fiddle with the `with_embargo` flags on the entry level, and the `embargo_...Moving the embargo setting to the upload level, controlled using the embargo_length attribute, 0 meaning no embargo.
The idea is that henceforth one should never fiddle with the `with_embargo` flags on the entry level, and the `embargo_length` attribute on the Upload level should be set/changed only using the following methods:
1) at creation of a new upload
2) Using Upload.set_upload_metadata (@process or local method) or the corresponding fastapi endpoint
4) when publishing (@process + REST endpoints)
5) when publishing to central nomad (@process + REST endpoints)
6) when importing a bundle (@process + REST endpoints)
7) through certain CLI commands, all internally invoking Upload.set_upload_metadata. Commands affected so far: chown, lift-embargo (command for automatically lifting embargo when the embargo period is over)
The attribute `embargo_length` is from now on **required** on Upload (i.e. can't be None) and is 0 by default.
Removed:
- The possibility to lift the embargo through the flask api (nomad.app.flask.api.repo.EditRepoCalcsResource)
- the possibility to set the embargo flag through user metadata files
- the cli command nomad.cli.admin.uploads.edit, as it does not do anything very usefulMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/377GUI/Encyclopedia/Normalizer adapted to new metainfo2021-09-03T18:08:56ZLauri HimanenGUI/Encyclopedia/Normalizer adapted to new metainfo- Overview page now uses new metainfo layout: DOS, BS, Heat capacities, free energy, geometry optimization.
- ArchiveBrowser overview refactored: separated overview logic into indidividual components, adapted to new metainfo layout.
- En...- Overview page now uses new metainfo layout: DOS, BS, Heat capacities, free energy, geometry optimization.
- ArchiveBrowser overview refactored: separated overview logic into indidividual components, adapted to new metainfo layout.
- Encyclopedia API/normalization/GUI now adapted to new metainfo: some problems with normalization, API now uses new archive format, removed unnecessary metainfos (`section_band_gap`, `section_brillouin_zone`) and adapted ENcyclopedia GUI to to these changes.
- Fixed unit issue with thermodynamical properties (#582)Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/378Metainfo refactor2021-09-03T18:40:23ZMarkus ScheidgenMetainfo refactorRe-enabled channel information for both spin channels, fixed issues in missing index information for channel info, fixed problem with segment labels.Re-enabled channel information for both spin channels, fixed issues in missing index information for channel info, fixed problem with segment labels.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/379Workflow parsers2021-09-06T12:19:26ZAlvin Noe LadinesWorkflow parsersIncluded aflow parserIncluded aflow parserMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/380Hotfix for v0.10.6: broken provider specific optimade properties2021-09-06T19:20:58ZMarkus ScheidgenHotfix for v0.10.6: broken provider specific optimade propertiesRemoved non primitive provider specific properties form optimade, because they cant be encoded without exception.Removed non primitive provider specific properties form optimade, because they cant be encoded without exception.