nomad-FAIR merge requestshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests2021-10-22T15:29:51Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/430Use the real mongo for the tests2021-10-22T15:29:51ZMohammad NakhaeeUse the real mongo for the testsConcerning issue #579Concerning issue #579Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/431Fixes to nginx-config (and other deployment related issues) #6382021-10-22T16:27:49ZMarkus ScheidgenFixes to nginx-config (and other deployment related issues) #638v1.0.0-betahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/432Fixed parser metadata generation2021-10-25T11:22:26ZMarkus ScheidgenFixed parser metadata generationhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/429remove preview functionality2021-10-26T10:22:20ZMohammad Nakhaeeremove preview functionalityThe preview button has been removed in this branch.The preview button has been removed in this branch.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/434Introduce CLI for upgrading and migrating mongo data2021-10-26T10:22:29ZDavid SikterIntroduce CLI for upgrading and migrating mongo dataFirst version of CLI command for migrating data from one mongodb to another. The data transformations needed are kept track of by #578 (all requirements concerning mongo specified in this issue until now are handled by the CLI).
The com...First version of CLI command for migrating data from one mongodb to another. The data transformations needed are kept track of by #578 (all requirements concerning mongo specified in this issue until now are handled by the CLI).
The command takes options for specifying a source and a destination mongo DB, and it offers a dry run mode. It migrates selected uploads and all records related to them (entries and datasets). It currently just overwrites any existing records in the destination DB, but it may be a good idea to add some customizable smartness (for example a "don't overwrite" or "overwrite only if newer" mode, etc).
Uploads can be selected by a query or by a file listing upload IDs. Uploads that have a status saying that they are currently processing are ignored. If an entry is marked as processing though the upload is not (or at least was not at the moment when we read it), that particular upload will be reported as failed (neither the upload nor its entries will be updated). The script also performs various validation checks.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/433Added nexus parser2021-10-27T08:26:23ZSandor BrockhauserAdded nexus parserhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/439Freeze pymongo version.2021-10-27T12:54:26ZMarkus ScheidgenFreeze pymongo version.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/440Froze most of the requirements.txt packages.2021-10-28T06:26:15ZMarkus ScheidgenFroze most of the requirements.txt packages.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/438Support for mechanical properties, better nested query support + GUI refactoring2021-10-28T07:19:17ZLauri HimanenSupport for mechanical properties, better nested query support + GUI refactoringThe main goal of this merge was to add support for mechanical properties. This includes:
- Adding metadata and normalizer functionality for storing mechanical properties under `results.properties.mechanical`
- New GUI filters for `bulk_...The main goal of this merge was to add support for mechanical properties. This includes:
- Adding metadata and normalizer functionality for storing mechanical properties under `results.properties.mechanical`
- New GUI filters for `bulk_modulus`, `shear_modulus` and available `mechanical_properties`.
- GUI support for defining nested queries. The current mechanism works so that if you wrap input filter components within `InputSection`, they automatically become nested. Currently, only one nested query per section is supported, might change in the future. Also, there is not yet any syntax for defining nested properties through the search bar.
- GUI support for showing active nested properties. Added a new component for grouping `FilterChips` so that they differ from the "flat" filters.
- Support for the serialization of nested filter in the URL and in the API queries.
- Added GUI support for pressure units (needed for bulk/shear modulus).
- Normalizer + GUI tests
As a side effect, the following changes were made:
- The layout of property cards was simplified and unified. One no longer has to manually specify an aspect ratio and the layout is fully grid-based.
- Small restyling of the filter menus: Added vertical separators in order to better distinguish between different filters, especially important for the new nested filter sections.
- The new nested filters were enabled for band gap search (one task in #569). This introduced a small rename for the metainfo related to band gaps.
- Restyling of the `SectionTable` component. Now looks like the default table from MUI.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/441Migration fix cli imports2021-10-29T07:59:41ZMarkus ScheidgenMigration fix cli imports@dsikter This is what I tried to explain with my last comments in !437. Let me know, if this is acceptable.
This MR contains the commits of !437@dsikter This is what I tried to explain with my last comments in !437. Let me know, if this is acceptable.
This MR contains the commits of !437David SikterDavid Sikterhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/421New GUI for AI toolkit2021-10-29T08:55:46ZLuigi SbailoNew GUI for AI toolkitThe GUI of the AI toolkit comprises three different pages. A landing page and two tutorials lists.The GUI of the AI toolkit comprises three different pages. A landing page and two tutorials lists.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/443Release v0.10.82021-10-29T12:47:37ZMarkus ScheidgenRelease v0.10.8Add explanation to increase the virtual memory for Elasticsearch. (At least for Linux users)Add explanation to increase the virtual memory for Elasticsearch. (At least for Linux users)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/435Fix Downloading2021-11-01T14:26:58ZMohammad NakhaeeFix DownloadingRelated to issue #494 and #514 and a fix for #640
I am creating this Merge request to start a discussion around the concept of filterAbbreviations[key]
This is the function that changes datasets.dataset_id to dataset_id and many keys as...Related to issue #494 and #514 and a fix for #640
I am creating this Merge request to start a discussion around the concept of filterAbbreviations[key]
This is the function that changes datasets.dataset_id to dataset_id and many keys as well.
The downloads work correctly when I skipped this function.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/444Improved the suggestion feature2021-11-04T08:08:57ZLauri HimanenImproved the suggestion featureIn the GUI we offer suggestions for certain index fields. These are implemented using the [ES Completion suggester feature](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-suggesters-completion.html) which is a special...In the GUI we offer suggestions for certain index fields. These are implemented using the [ES Completion suggester feature](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/search-suggesters-completion.html) which is a special feature built exactly for this kind of task.
Previously we were defining a new [ES field](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) with the `completion` type in order to create these suggestions. This way the indexing of suggestion values comes pretty much for free and the index size is not affected too much. This means, however, that only one suggestion value was supported per quantity, which makes it impossible to make autocompletion suggestions if the user types e.g. a word in the middle of a keyword or string: e.g. typing `salt` would not create a suggestion for `rock salt`. Technically this is because the Completion suggester is based on a special graph data structure that can only start suggesting values from the beginning of a string.
To overcome this limitation, the default approach is to augment the suggestion values by adding several suggestion values at index time (examples of this discussion [here](https://discuss.elastic.co/t/completion-suggester-and-analyzer/13867), [here](https://www.elastic.co/blog/you-complete-me) and [here](https://taranjeet.medium.com/elasticsearch-using-completion-suggester-to-build-autocomplete-e9c120cf6d87)). **This merge request implements this mechanism by storing suggestions under `<quantity name>__suggestion` when the values need to be tokenized.**
The suggestion mechanism is selected by specifying a predefined string option (`suggestion="simple"`: the old behaviour, `suggestion="default"`: tokenization using whitespaces and underscores, `suggestion="formula"`: tokenization using formula fragments, `suggestion=<function>`: custom tokenization function). The old mechanism of storing a single suggestion string under a field is used when `suggestion="simple"`.
The downside of using a new index attribute instead of a field for each suggestion is that the index size becomes bigger and that the source documents look quite busy with all this new suggestion data. By default, the suggestion values are completely excluded from the source documents in all metadata searches. Depending on our experience with the index size, we can think about limiting suggestions, using a separate index for suggestions (really nasty), or experimenting with other suggestion mechanisms ([here is a really good article about all reasonable options](https://medium.com/@mourjo_sen/a-detailed-comparison-between-autocompletion-strategies-in-elasticsearch-66cb9e9c62c4)).Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/446MaterialCard improvements2021-11-05T15:44:16ZLauri HimanenMaterialCard improvements- Neẃ improved `MaterialCard` shown in the overview page, broken down into four sections:
- Composition: formula, elements, material name, etc. Always shown.
- Symmetry: all symmetry related properties. Shown only if symmetry data is...- Neẃ improved `MaterialCard` shown in the overview page, broken down into four sections:
- Composition: formula, elements, material name, etc. Always shown.
- Symmetry: all symmetry related properties. Shown only if symmetry data is available (bulk materials), otherwise showing `NoData`.
- Lattice parameters: Lattice parameters for the currently selected structure. Shown only if data is available (materials with atomic structure and unit cell present), otherwise showing `NoData`.
- Structure: Shows the 3D viewer for the currently selected structure. Shown only if data is available (materials with atomic structure), otherwise showing `NoData`.
This layout should accommodate all types of data: simulations with/without symmetry or lattices and experimental data which typically has only composition. If there are any structures present, there is a dropdown selector in the top right corner for selecting which one is shown (affects Structure and Lattice parameters).
- Improved and extended tests: I created proper tests for MaterialCard and greatly simplified the GUI test setup and usage in general.
- Added angles as a new unit in the GUI (needed by angles used in lattice parameters).
- Homogenized the look of tables containing data from metainfo (e.g. Symmmetry/Composition/LatticeConstants are now shown within a table, bulk/shear modulus and band gaps are shown within a table. The new look adds thin borders around the tables to make it easier to recognize where the contents start/end. This was especially a problem in the new MaterialCard where without the borders the different sections are harder to distinguish.
@mscheidg: I'm open to any suggestions for modifying the new look. I can make the changes before merging.
![materialcard](/uploads/765c95c9a594d1262ca5c3eb15127972/materialcard.png)Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/448Added exclude from search option to aggregations. #573, #5752021-11-05T18:21:56ZMarkus ScheidgenAdded exclude from search option to aggregations. #573, #575https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/447Hotfixes v0.10.9. Fixes #644, #6492021-11-08T14:58:49ZMarkus ScheidgenHotfixes v0.10.9. Fixes #644, #649https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/445Some fixes in GUI2021-11-09T13:29:08ZMohammad NakhaeeSome fixes in GUIRelated to issue #647Related to issue #647Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/449Merge v0-hotfixes into v0.10.92021-11-09T14:03:13ZMarkus ScheidgenMerge v0-hotfixes into v0.10.9https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/451Merge for release of v0.10.92021-11-10T12:09:30ZMarkus ScheidgenMerge for release of v0.10.9