nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2023-12-21T16:04:41Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1Run processing tests with the internal celery test worker.2023-12-21T16:04:41ZMarkus ScheidgenRun processing tests with the internal celery test worker.Currently, you have to run processing tests with a running and manually started celery worker.Currently, you have to run processing tests with a running and manually started celery worker.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/17GITLap QU/CI/CD pipeline2018-12-13T09:01:38ZMarkus ScheidgenGITLap QU/CI/CD pipelinegood software engineering practices are important.good software engineering practices are important.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/42Offline processing2018-12-13T09:01:26ZMarkus ScheidgenOffline processingMake nomad@FAIR aware of the original offline processing procedure. Nomad@FAIR should be able to reproduce the old offline processing for testing and migration purposes.Make nomad@FAIR aware of the original offline processing procedure. Nomad@FAIR should be able to reproduce the old offline processing for testing and migration purposes.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/30Logs for parsing, normalizing (calc processing) should be made available.2018-12-13T09:01:26ZMarkus ScheidgenLogs for parsing, normalizing (calc processing) should be made available.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/38Some guidelines and fixes for logging in parsers, normalizers, processing2018-12-13T09:01:25ZMarkus ScheidgenSome guidelines and fixes for logging in parsers, normalizers, processingThe existing parsers are seem to change the root logger on their own. We need to establish a common framework for logging. Also, all projects should use the same 'guidelines' to decide on what to log, under what level, and how (e.g. stru...The existing parsers are seem to change the root logger on their own. We need to establish a common framework for logging. Also, all projects should use the same 'guidelines' to decide on what to log, under what level, and how (e.g. structural logging).https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/7A pagenated list of all repository entries.2018-08-25T09:22:09ZMarkus ScheidgenA pagenated list of all repository entries.... as a starter for *browse* functionality, before we implement an actual search.... as a starter for *browse* functionality, before we implement an actual search.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/6Handle double uploads2018-09-06T12:17:19ZMarkus ScheidgenHandle double uploadsUse the upload hash to fail after extracting.Use the upload hash to fail after extracting.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/13Remove unwanted and failed uploads.2018-09-06T12:18:43ZMarkus ScheidgenRemove unwanted and failed uploads.This also includes a general delete function that removes uploads and calcs from:
- uploads
- repo
- archive
- filesThis also includes a general delete function that removes uploads and calcs from:
- uploads
- repo
- archive
- fileshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/23Decent error handling in GUI2018-09-06T12:19:00ZMarkus ScheidgenDecent error handling in GUIthere is none at the momentthere is none at the momenthttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/16A suite of normal, large, broken, mixed, test calcs2018-09-06T12:18:26ZMarkus ScheidgenA suite of normal, large, broken, mixed, test calcs- to have more realistic uploads
- put some more toy data into the system- to have more realistic uploads
- put some more toy data into the systemhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/5Handle large uploads with more than 100 calcs2018-09-18T09:41:28ZMarkus ScheidgenHandle large uploads with more than 100 calcsCurrently calcs are embedded in uploads. This is the case for the processing state in UploadProc, the user data Upload, and upload data given by the API. We should make this their own first cititizen objects in dbs and API. Further, the ...Currently calcs are embedded in uploads. This is the case for the processing state in UploadProc, the user data Upload, and upload data given by the API. We should make this their own first cititizen objects in dbs and API. Further, the GUI should pagenate calcs per upload.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/12Integrated documentation2018-08-25T22:59:57ZMarkus ScheidgenIntegrated documentationThe documentation as part of the nomad GUI, also add more help/info to the individual pages.The documentation as part of the nomad GUI, also add more help/info to the individual pages.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/15Scale with calculations2018-12-13T08:55:21ZMarkus ScheidgenScale with calculationsCurrently we will probably put calculations into HDF5 files, but eventually we need:
- a tighter interface for metainfo based data
- implementations based on some form of database (custom, column, ?)Currently we will probably put calculations into HDF5 files, but eventually we need:
- a tighter interface for metainfo based data
- implementations based on some form of database (custom, column, ?)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/11Fully intergated docker compose2018-08-28T07:42:38ZMarkus ScheidgenFully intergated docker composeA docker compose that also has
- a build of the gui
- the api
- configured minio (access for read-only buckets)A docker compose that also has
- a build of the gui
- the api
- configured minio (access for read-only buckets)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/27Show the nomad-xt version/ref/commit in the GUI.2018-08-30T11:16:18ZMarkus ScheidgenShow the nomad-xt version/ref/commit in the GUI.This had already helped me once, and will be needed again.This had already helped me once, and will be needed again.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/28Progress on uploads2018-09-06T12:19:39ZMarkus ScheidgenProgress on uploadsThe upload itself must not block the UI and should be shown in the Upload component.The upload itself must not block the UI and should be shown in the Upload component.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/33Uploads can be accepted into the public nomad2018-09-18T09:37:55ZMarkus ScheidgenUploads can be accepted into the public nomadhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/31Sorting and filtering for upload calculations2018-09-18T09:41:51ZMarkus ScheidgenSorting and filtering for upload calculationshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/209File preview (for textfiles) in GUI2019-11-14T08:39:43ZMarkus ScheidgenFile preview (for textfiles) in GUIIt would be nice, to actually see output and other files in the GUIIt would be nice, to actually see output and other files in the GUIMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1692Group-Based Access Control for Uploads2023-10-30T14:40:46ZAdam FeketeGroup-Based Access Control for UploadsIt would be beneficial to implement a feature allowing users to define groups, where members of these groups can either view or edit specific uploads. This would provide a more flexible way for users to share and manage access to their u...It would be beneficial to implement a feature allowing users to define groups, where members of these groups can either view or edit specific uploads. This would provide a more flexible way for users to share and manage access to their uploads.
## Possible Implementation:
1. **Group Management in Keyclock**
- While this is possible, this will mean that the group management will reside outside NOMAD.
2. **Group Management within NOMAD:**
- Instead of using external systems like Keycloak for group management, integrate group creation and modification directly into the NOMAD application.
- Each group should have designated roles like 'group_admin' and 'members'.
- **Admins**: Can create, edit, and delete the group, add/remove members.
- **Members**: Have read/write permissions based on the group settings.
- Similar to platforms like GitLab, members should be able to leave a group on their own.
2. **Upload Visibility and Access:**
- By default, an upload should only be visible to the uploader at creation time.
- While uploading or editing an upload, there should be a dropdown menu allowing users to select which groups or individuals should have access to the specific upload. An upload can be shared with multiple groups.
- On the main upload page, there should be a clear indication that the upload is visible to a user/group (i.e., through which group membership). If an upload is visible due to multiple group memberships, all relevant groups should be displayed.
3. **Ideas for the User Interface:**
- On the top right of the Nomad interface, incorporate a button for users to manage their group memberships.
- The design and user experience should be intuitive and align with familiar platforms like GitLab, ensuring ease of use and understanding.
## Use-Case:
- If multiple groups or projects are using the same Nomad Oasis deployment, each could have their dedicated shared uploads.
- Before making an AI toolkit publicly available, it could first be shared with a select group of contributors.
## Additional Notes:
- It's mentioned that there's another feature request regarding upload visibility ([Issue #1691](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1691)). It might be advantageous to address these features together, ensuring consistent and coherent implementation.
---
Note: This feature suggestion was inspired by input from @josma and further enhanced by suggestions from @g-michaelgoette based on his use cases.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1287Tabular Parser: sub_sub_sections in row mode2023-04-17T16:31:16ZAndrea AlbinoTabular Parser: sub_sub_sections in row modeWe are getting deeper with this tool and this leads us relentlessly to some new feature request.
When using the row mode, some users have a bundle of columns that repeats along the row because they use a single repeated class multiple t...We are getting deeper with this tool and this leads us relentlessly to some new feature request.
When using the row mode, some users have a bundle of columns that repeats along the row because they use a single repeated class multiple times in that row.
This is an example of table we may face (note that process step, date, user, location, are not repeated in the row):
|Process step|date|user|location| Material | Sputtering| Density | Voltage |Material | Sputtering| Density | Voltage |
|-|-|-|-|-|-|-|-|-|-|-|-|
|1|01-01-22|john|Berlin|MgO|Y|2.3|4|TiO|Y|4.5|45|
|2|02-01-22|micha|Berlin|ZnO|Y|2.3|4|CS|Y|4.5|45|
and this is how it should be parsed in the archive:
```
data:
m_def: my_experiment
process_steps: # this is the repeated section parsed currently by our table parser
- date: 01-01-22
user: john
location: Berlin
sputtered_materials:
- name: MgO
sputtering: Y
density: 2.3
voltage: 4
- name: TiO
sputtering: Y
density: 4.5
voltage: 45
- date: 02-01-22
user: micha
location: Berlin
sputtered_materials:
- name: ZnO
sputtering: Y
density: 2.3
voltage: 4
- name: CS
sputtering: Y
density: 4.5
voltage: 45
```
So each row gives rise to an instance of a repeated subsection, and each bundle of columns in the same row gives rise to an instance of a repeated sub_sub_section.
I can already implement a schema where I parse every column manually into a different instance but this carries a couple of drawbacks:
- I have to rename each column with a different name so I can use the tabular annotation as it is now
- I have to redundantly define a schema for each bundle that is essentially the same class
- I have to know a priori how many times that class is used in that row, but this is not always know and can even vary
I explained already this issue to @amgoAmir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1181Bundle format and functionality2023-12-21T15:39:10ZMarkus ScheidgenBundle format and functionalityAn upload "bundle" is a data artifact that contains everything necessary to interpret the data of a single upload.
I needs to contain:
- raw files (optionally)
- archive (optionally based on a required)
- all upload and entry metadata
...An upload "bundle" is a data artifact that contains everything necessary to interpret the data of a single upload.
I needs to contain:
- raw files (optionally)
- archive (optionally based on a required)
- all upload and entry metadata
- all related DOI records
- all necessary schema
- a directory of all external references
- hashes/checksums, etc. ideally following some long-term archive standard
We should use these "bundles":
- to internally store immutable/published uploads
- transfer uploads from installation to installation
- import/export functionalityhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1595No type checking in ArchiveParser2023-12-21T15:38:50ZAhmed IlyasNo type checking in ArchiveParserCurrently the `parse_file` function in `ArchiveParser` doesn't fully type check user uploaded data files to see if the types match the user uploaded schema.Currently the `parse_file` function in `ArchiveParser` doesn't fully type check user uploaded data files to see if the types match the user uploaded schema.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1546Allow showing top N values from a set of fixed options in InputField2023-12-21T15:55:21ZLauri HimanenAllow showing top N values from a set of fixed options in InputFieldhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1663Dos normalization menu even when normalization is not possible2023-09-11T08:35:42ZJose PizarroDos normalization menu even when normalization is not possible@himanel1 @ndaelman I realized the kebab menu in the DOS always appears and let user click on "Normalize intensities", independently of whether the normalization factor was extracted by the DosNormalizer or not.
Could this be rendered o...@himanel1 @ndaelman I realized the kebab menu in the DOS always appears and let user click on "Normalize intensities", independently of whether the normalization factor was extracted by the DosNormalizer or not.
Could this be rendered or not depending on that condition? Otherwise is confusing that the units on the DOS axis change from "states / eV" to "1 / eV" but nothing happens.Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1493Small ELN improvements2023-12-21T15:55:19ZMarkus ScheidgenSmall ELN improvements### Bugs that need to be fixed
- [x] When a numeric field filled and the value is removed afterwards, an error occurs (#1362). There seems to be different problems with both int and floats. The error is happening in the processing, not t...### Bugs that need to be fixed
- [x] When a numeric field filled and the value is removed afterwards, an error occurs (#1362). There seems to be different problems with both int and floats. The error is happening in the processing, not the UI.
- [unable to reproduce] You cannot remove the value from a ReferenceEditQuantity field.
- [x] Only the `main_author` should be allowed to remove an upload. Currently all co-authors can. This needs to be fixed in the API and in the UI.
- [x] When the `Datatable` component shows table rows that are wider than the table, you have to scroll to reach the table actions (e.g. the "->" button). We need to make sure that the table actions are always visible.
### Simple things and low hanging fruits
- [x] Duplicate functionality for entries. Entries can already be duplicated in the file browser. For convinience, we need to add a "duplicate" button (there is a well-known icon in the MUI icon set). This button should be added to the entry overview page and as an additional table action to the entry table on the upload overview page. (#1099)
- [x] For convenience we should add a "search" button to the upload page. Clicking the button should open a menu similar to the "explore" menu. Clicking on a menu item should navigate to the respective search app and preselect the respective upload id (#1100, #1143)
- [x] Currently, plots generated by the plot annotation use the units from the global units system. If there is a display unit on the respective quantity, this should be used instead. (#1059)
- [x] FileEditQuantity on a list-shaped quantity does not allow to navigate to the file preview. This only works for scalar-shaped file quantities (#1033)
- [x] When uploading a file with the FileEditQuantity, no upload progress is shown. We need to show the same progress model (and disable the ui) as we used for uploading files to an upload
- [x] It should be possible to upload multiple files to a list-shaped FileEditQuantity.
- [x] The checkbox options in the ReferenceEditQuantity dialog should all be unchecked by default. We need other options: "From this upload", "From my uploads", "From my (co)-authored uploads", "From published uploads"
- [x] The RichTextEditor annotation needs an option to define the default hight of the text editor
- [x] ReferenceEditQuantities that target an entry with processing error, do not allow to navigate to this entry (even if an archive exists).
- [x] We need to visually tell that an entry has a (processing) error. It should be visible on all entry pages, in the upload page entry list, in the file browser, when opening a reference to an entry with error.
- [x] It is really hard to see list-shape edit quantities. They should be foldable and unfolded edit field should be indented. Folding like with repeating sub-sections.
- [x] Clicking the (+) button on folded repeated sub-sections, should unfold the sub-section and highlight the new sub-section.
- [x] Indicate that there are co-authors in the uploads list (and on the upload page). The "publish" icon could show this.
- [x] We might also offer co-authors and reviewers as table columns (similar to authors on entry tables).
- [x] If upload and entry names are available, use them in the breadcrumbs. Maybe fall back to file name for entries. (#1558)
### Maybe not too hard
- [ ] A button EditQuantity for boolean quantities that is automatically set to False after processing. (#1242)
- [ ] Show (incoming) references as a table that shows, allows to sort, and/or filter based on the reference quantity definition
- [ ] Allow to parameterize error and warning logs with a path to the quantity or section that causes the error. The logs can be taken from the processing logs to show the error on the quantity or section.
- [ ] API+UI to create+delete groups. Co-authors and reviewers can be added or removed from groups. Groups can be added/removed to uploads. Needs more discussion. (#1016)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1619Default entry for reference edit quantity not possible.2023-12-21T16:07:26ZJohannes LehmeyerDefault entry for reference edit quantity not possible.We would like to refer to specific json files that are available to every user, eg. our evaporation machines for some processes.We would like to refer to specific json files that are available to every user, eg. our evaporation machines for some processes.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1618Change fields in „Create new entry“ Window2023-12-21T16:04:42ZJohannes LehmeyerChange fields in „Create new entry“ WindowDefault option should be Custom schema (or customiseable by user). Custom schemas should be visible without knowing what to type in the search box.Default option should be Custom schema (or customiseable by user). Custom schemas should be visible without knowing what to type in the search box.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1610Buttons in ELNs with callback functions2023-12-21T16:04:42ZMichael GötteButtons in ELNs with callback functionsThere is a need for buttons in ELNs to trigger special functionality. Right now the workaround is to have a BoolEditQuantity and set it to true, click save, and then check in the normalizer and then execute the code. It would be better t...There is a need for buttons in ELNs to trigger special functionality. Right now the workaround is to have a BoolEditQuantity and set it to true, click save, and then check in the normalizer and then execute the code. It would be better to directly link buttons with functions in the backend.
Usecases:
1. Creating Samples in a Batch
Each sample is an archive entry. Usually when new samples are created they all have the same properties and will be grouped in a batch. One could specify the number of samples and then click a button which creates all archives.
2. Creating Samples and Processes
A button could be used to create processes for which no data exists based on templates and link them to the correct samples.
3. Perform complex normalization
I have a 500mb csv file with time resolved data. To have interactive graphs it make sense to group the data in 15 minutes intervals to reduce the data before moving it to the archive. I can do this in the normalizer, but then it would be triggered every time i click save. An extra button could only trigger this step when needed.
4. Performing mass renamings of quantities within multiple entries. Such actions could be shown in the 'upload' page when several entries have been chosen for performing the action on.
Buttons are helpful when some stuff should not be exceuted everytime something is saved. Or if stuff is created only once. They are a flexible tool to speed up the lab workflows for scientists.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1435Skip parser matching when parser is already specified2023-08-22T13:08:54ZNathan DaelmanSkip parser matching when parser is already specifiedFrom a discussion with @ladinesa we came with the following proposal:
atm, running `nomad parse --parser <parser name> --show-archive` will still perform the parser matching.
We know this, since some auxiliary files (e.g. `OUTCAR`) are n...From a discussion with @ladinesa we came with the following proposal:
atm, running `nomad parse --parser <parser name> --show-archive` will still perform the parser matching.
We know this, since some auxiliary files (e.g. `OUTCAR`) are not being picked up on in the presence of the main file (e.g. `vasprun.xml`).
`nomad` then just returns `assert parser is not None, 'there is no parser matching %s' % mainfile`.
I'd argue that this behavior is counterintuitive in light of the `--parser <parser name>` option.
If we change this, I'd also request to add further specification in `nomad parse --help` on how to formulate `<parser name>`.
Trial-and-error doesn't work in guessing the naming format, and I eventually only figured it out thx to @ladinesa .
If the naming can be customized via a settings file, just pointing to its location is also fine.
@mscheidg I'd really like your feedback, given that this touches on the command line behavior.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1590scatterplotts of vectors or dataobjects in Explore app2023-12-21T16:07:12ZJulia Schumannscatterplotts of vectors or dataobjects in Explore appCan we have a feature in the Explore app (for the use case catalysis), that allows to plot selectivity - conversion plots to compare several entries. Similar to what is possible with the plots in a section, e.g.
`'x': ['conversion/0:1/...Can we have a feature in the Explore app (for the use case catalysis), that allows to plot selectivity - conversion plots to compare several entries. Similar to what is possible with the plots in a section, e.g.
`'x': ['conversion/0:1/conversion_product_based'],
'y': ['products/:/selectivity'],`
![image](/uploads/690004c6eb68582eddd3018eaa464881/image.png)
but for comparing different data entries which have some common propoerties (same reactant and product). That means it should be possible to have arrays or vectors in a scatterplot.
@josma: Does this capture what I need?https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1568Create new entries named after one quantity, not after their class name2023-12-21T16:04:41ZAndrea AlbinoCreate new entries named after one quantity, not after their class nameI'm testing the tabular parser after last MR by @amgo, it is super. So I can go on in the refinements of the tool:
when we create a new entry, such in the schema at the bottom, the entry will be named "Substrate_1.entry_data.archive.yam...I'm testing the tabular parser after last MR by @amgo, it is super. So I can go on in the refinements of the tool:
when we create a new entry, such in the schema at the bottom, the entry will be named "Substrate_1.entry_data.archive.yaml" "Substrate_2.entry_data.archive.yaml" etc, depending on how many substrates I have in my excel file.
What I ask here is to have the entry named with one of the quantities contained inside the m_def, for example the `name` of the Substrate in this case. It would be a feature very similar to `label_quantity` for nested sections in one single entry, showed in the code snippet here.
```yaml
sub_sections:
procedure:
repeats: true
section:
base_section: '#/Procedure_step'
more:
label_quantity: name
quantities:
name:
type: str
description: "The step name"
m_annotations:
eln:
component: StringEditQuantity
tabular:
name: Substrate/Material
```
```yaml
definitions:
name: 'MOVPE Experiment (IKZ Ga2O3)'
sections:
MovpeExperiment:
base_sections:
- nomad.parsing.tabular.TableData
- nomad.datamodel.metainfo.eln.Activity
- nomad.datamodel.data.EntryData
m_annotations:
eln:
hide: ['end_time', 'lab_id', 'location']
quantities:
substrate_data_file:
type: str
description: |
A reference to an uploaded .xlsx
m_annotations:
tabular_parser:
sep: ','
comment: '#'
browser:
adaptor: RawFileAdaptor
eln:
component: FileEditQuantity
sub_sections: substrates:
repeats: true
section:
quantities:
substrate:
type: '#/Substrate'
m_annotations:
eln:
component: ReferenceEditQuantity
Substrate:
base_sections:
- nomad.datamodel.metainfo.eln.Ensemble
- nomad.datamodel.data.EntryData
quantities:
name:
type: str
description: FILL
m_annotations:
eln:
component: StringEditQuantity
tabular:
name: Substrate/Material
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1622Automatic File-naming2023-12-21T16:04:41ZJohannes LehmeyerAutomatic File-namingAutomatic File-naming should be possible with option to override exceptions. The intention of us is to hide the true filenames and work with the names the users choose as quantities in their entriesAutomatic File-naming should be possible with option to override exceptions. The intention of us is to hide the true filenames and work with the names the users choose as quantities in their entrieshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1355Visually consolidate related entries and respective files2023-12-21T16:05:38ZMarkus ScheidgenVisually consolidate related entries and respective filesThere are many cases, where multiple mainfiles (and aux-files) and respectively multiple entries form a logical whole. A few examples:
- parsed files (e.g. nexus) that are augmented with user-entered information from a "sister" ELN entry...There are many cases, where multiple mainfiles (and aux-files) and respectively multiple entries form a logical whole. A few examples:
- parsed files (e.g. nexus) that are augmented with user-entered information from a "sister" ELN entry
- workflow entries and the tasks, input, outputs, etc. they reference
- entries that are coming from an .xls (or other tabular file)
It is important to show this connected (or even hide the parts) from the user, because in their minds these things belong to each other and form a semantic unit.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1274Strukturbericht rendering improvement2023-12-21T16:06:49ZLauri HimanenStrukturbericht rendering improvement The strukturbericht names should be rendered in the UI with the correct subscripts etc. In editable fields, they should still be entered and displayed in plain text. The strukturbericht names should be rendered in the UI with the correct subscripts etc. In editable fields, they should still be entered and displayed in plain text.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1255Adding reduce and descriptive formula to the GUI search2023-02-09T10:37:36ZJose Marquez PrietoAdding reduce and descriptive formula to the GUI searchJose Marquez PrietoJose Marquez Prietohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1249A "custom quantity" search widget for the dashboard2023-12-21T16:04:41ZJose Marquez PrietoA "custom quantity" search widget for the dashboardAt the moment, the customs quantity search under the "ELN search page" cannot be dragged into the dashboard. In the future, it will be useful to have it permanently there for the ELN app.At the moment, the customs quantity search under the "ELN search page" cannot be dragged into the dashboard. In the future, it will be useful to have it permanently there for the ELN app.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1189Cannot delete uploads if .volumes lies on an nfs share2023-12-21T15:40:10ZFlorian DobenerCannot delete uploads if .volumes lies on an nfs shareI created a nomad deployment at fhi via an adapted docker-compose file. For backup and storage issues we moved the .volumes folder to an nfs share. However, this makes it impossible to delete any uploads via the nomad gui. The uploads pe...I created a nomad deployment at fhi via an adapted docker-compose file. For backup and storage issues we moved the .volumes folder to an nfs share. However, this makes it impossible to delete any uploads via the nomad gui. The uploads persists and the status changes to `Process delete_upload failed: OSError: [Errno 39] Directory not empty: 'archive'`. A second try changes the status again to `Process delete_upload failed: OSError: [Errno 16] Device or resource busy: '.nfs000000000c01329400000001'`.
I tried it via `nomad admin uploads rm`, too. It shows a similar error message:
```
1 uploads selected, deleting ...
ERROR nomad.cli 2022-11-22T07:29:36 could not delete files
- exception: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/nomad/cli/admin/uploads.py", line 424, in delete_upload
upload_files.delete()
File "/usr/local/lib/python3.7/site-packages/nomad/files.py", line 671, in delete
shutil.rmtree(self.os_path)
File "/usr/local/lib/python3.7/shutil.py", line 494, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/local/lib/python3.7/shutil.py", line 432, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/usr/local/lib/python3.7/shutil.py", line 452, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File "/usr/local/lib/python3.7/shutil.py", line 450, in _rmtree_safe_fd
os.unlink(entry.name, dir_fd=topfd)
OSError: [Errno 16] Device or resource busy: '.nfs000000000c01329400000001'
- exception_hash: JM34EDxYaKn6rnUZaYL8erVJcKaF
- nomad.commit: 88fba0386
- nomad.deployment: oasis
- nomad.service: nomad_oasis_app
- nomad.version: 1.1.5
```
This is the error message after a first try of deletion with `nomad admin uploads rm`:
```
ERROR nomad.cli 2022-11-22T07:32:02 could not delete files
- exception: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/nomad/cli/admin/uploads.py", line 424, in delete_upload
upload_files.delete()
File "/usr/local/lib/python3.7/site-packages/nomad/files.py", line 671, in delete
shutil.rmtree(self.os_path)
File "/usr/local/lib/python3.7/shutil.py", line 494, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/usr/local/lib/python3.7/shutil.py", line 436, in _rmtree_safe_fd
onerror(os.rmdir, fullname, sys.exc_info())
File "/usr/local/lib/python3.7/shutil.py", line 434, in _rmtree_safe_fd
os.rmdir(entry.name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'archive'
- exception_hash: i4sNv8DQkHjsy-qHI4I0XBUjcmeu
- nomad.commit: 88fba0386
- nomad.deployment: oasis
- nomad.service: nomad_oasis_app
- nomad.version: 1.1.5
```
Running cli removes the uploads from the nomad gui but they still persist in `.volumes/fs`.
This seems to be related to nfs creating hidden files, while files are still open (see https://bugzilla.redhat.com/show_bug.cgi?id=1362667). Therefore `rmdir` is not able to remove the folder. So probably the access to the upload should be closed before deleting (if possible?). Alternatively, `os.remove(...)` could be used recursively to delete this hidden nfs files before calling `os.rmdir(...)`.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/845Updating the legacy Keycloak deployment2023-12-21T15:39:09ZAdam FeketeUpdating the legacy Keycloak deploymentAlthough the 16.1.1 of Keycloak was released about 5 months ago since there is a major update starting with version 17.0.0. The "backend" has been changed and some migration is needed for using the latest versions.
Useful documentations...Although the 16.1.1 of Keycloak was released about 5 months ago since there is a major update starting with version 17.0.0. The "backend" has been changed and some migration is needed for using the latest versions.
Useful documentations:
- [Main page about the migration](https://www.keycloak.org/migration/migrating-to-quarkus)
- [New base image is used for building a container](https://www.keycloak.org/server/containers)
- [Development (using localhost) versus production mode](https://www.keycloak.org/server/configuration)
- https://www.keycloak.org/guides#serverhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1017Customize NOMAD installations2023-12-21T15:38:47ZMarkus ScheidgenCustomize NOMAD installationsOasis admins might want to disable some features. This can be archived with filters on
- available search filters
- available schemas (e.g. for eln, but also in the metainfo)
- available parsers
- available examples
- available north too...Oasis admins might want to disable some features. This can be archived with filters on
- available search filters
- available schemas (e.g. for eln, but also in the metainfo)
- available parsers
- available examples
- available north tools
This is pure on an installation or run level and should not require a build or separate images.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/947Support for displaying values as percent '%'2023-12-21T15:55:18ZJose Marquez PrietoSupport for displaying values as percent '%'From several sources, we have heard multiple times the need to support in some kind of way to display values in percentage. One possibility could be to introduce it as a unit of ratio. @himanel1 what is your opinion on this?
@aalbino, @...From several sources, we have heard multiple times the need to support in some kind of way to display values in percentage. One possibility could be to introduce it as a unit of ratio. @himanel1 what is your opinion on this?
@aalbino, @dflor Any thoughts about this?Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/846Improved dataset search and functionality2023-12-21T15:55:41ZLauri HimanenImproved dataset search and functionality- [ ] Add functionality to download an entire dataset. Would probably be best to implement as an API endpoint `datasets/{dataset_id}/download` in order to be similar to other dowload endpoints. Probably a button in the dataset page?
- [ ...- [ ] Add functionality to download an entire dataset. Would probably be best to implement as an API endpoint `datasets/{dataset_id}/download` in order to be similar to other dowload endpoints. Probably a button in the dataset page?
- [ ] Similar to `entries` and `materials`, we also need a search page + search context for datasets. Should be implementable as a simple aggregation over datasets as was available in v0.9.0. This would allow users to use the existing filters to look for a particular dataset. Currently, users have to first find an entry with a dataset, open the overview and click on the dataset link there.
- [ ] Possibility to add more detailed documentation for datasets? Instead of a simple comment, maybe we could support a rich text field that could be shown for each dataset?Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/970Add RDF/MSD plots to overview GUI page for molecular dynamics trajectories2022-12-16T11:14:56ZJoseph RudzinskiAdd RDF/MSD plots to overview GUI page for molecular dynamics trajectoriesMolecular radial distribution function plots (g(r) vs r) and mean square displacements (msd vs. t) should be added to the overview page for molecular dynamics trajectories. Similar to the "thermodynamic properties" section, these should ...Molecular radial distribution function plots (g(r) vs r) and mean square displacements (msd vs. t) should be added to the overview page for molecular dynamics trajectories. Similar to the "thermodynamic properties" section, these should be placed under "structural properties" (rdfs) and "dynamic properties" (msds)Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/9612D statistic plot2023-02-27T16:14:27ZLauri Himanen2D statistic plotWe already allow users to enable statistics "widgets" per search quantity. This should be extended to include combinations of numeric values in order to show 2D scatter plots. This would be great for the solar cell data where you could p...We already allow users to enable statistics "widgets" per search quantity. This should be extended to include combinations of numeric values in order to show 2D scatter plots. This would be great for the solar cell data where you could plot e.g. the solar cell efficiency vs some other property.
### 1. General widget component
We need a generic widget component that wraps all the widgets including the scatter plot.
- [x] Needs to be resizable.
- [x] Needs to be closeable.
- [x] Needs to be draggable.
- [x] Needs customizable actions bar at the top. E.g. the plot actions will be shown there.
- [x] Needs to have a title + possible description.
### 2. Scatter plot component
The first task would be to test out the plot implementation without thinking too much about how we get the data. The rough specifications would include:
- [x] Button to show/hide the edit dialog (see below).
- [x] Show the quantity names as x/y axis labels with a tooltip.
- [x] Additionally, the ability to choose quantities associated with the marker color. These quantities do not necessarily need to be numerical, but also text-type for the color.
If a string type quantity is picked as a color, a marker label could be added showing the value of that quantity. Filtering the data by clicking and removing labels would be added value. If a numerical field is chosen, a color scale bar should be shown instead of the labels.
- [x] Should resize to fit parent.
- [x] Reacts to changes in the "global" units.
- [x] Shows `Placeholder` when loading
- [ ] ~~Shows `NoData` when no results are available.~~ This breaks e.g. the region focus selection, simply show an empty plot instead.
- [x] The plot should be able to react to changes. E.g. when a user enables additional filters, the plot data will change quite often.
- [x] Pan, zoom, view reset.
- [ ] ~~Ability to click individual calculations, leading the user to that entry.~~ _Will be done later_
- [ ] ~~Tooltip when hovering on individual calculations to show the plotted values and entry id~~ _Will be done later_
- [x] Should be able to handle large amounts of data, something in the 10K range.
- [x] Visuals should be customized to match the look and feel of our UI. Mostly this can be achieved by using the theme font and colors. MUI components should be used whenever possible, e.g. `Tooltip`, `Select` etc.
- [x] Ability to select the maximum number of data points to retrieve and plot. A dropdown with a few presets should do. E.g. `100`, `1K`, `10K`, `100K`, `all`. Should maybe default to `10K`?
- [ ] ~~The component should show some notifications about the data: The number of retrieved samples, and some indication of whether all available data was loaded or not.~~ _Will be done later_
We should try two parallel implementations: one using `Plotly` (@mnakh) and one using ~~`Vega`~~ `d3.js` (@himanel1). This will give us an idea about the pros/cons of these two approaches and we can then decide if we keep one of them or both of them.
### 3. Scatter plot edit component.
We need a separate edit dialog for the scatter plot. It will be shown when first adding the plot and whenever the user wants to edit the widget.
- [x] The ability to freely choose the x and y axes. This could e.g. be e.g. two dropdowns - one for the x-axis and one for the y-axis - with some text search possibility + autocompletion. All numeric quantities would be shown as options.
- [x] The ability to choose a third quantity for coloring the points. This could e.g. be a dropdown. All quantities with a scalar value (can be numeric or string) should be available.
- [ ] ~~Ability to choose the unit for the x, y, and coloring axes.~~ _Will be done later_
- [x] Ability to propagate changes between the edit dialog and the plot itself.
### 4. Data fetching
Initially, we will simply use some dummy data ([dummy_data.json](/uploads/df70cad50ef7c4a4759cf7dc34e7426e/dummy_data.json)) to do the component implementation, but in the final version, all of the data will be served by ES. We need to see if using simple ES hits is performant enough or if ES provides some special tooling for these graphs. The way Kibana does this should be used as inspiration. The requirements for the data fetching are:
- [x] Should be able to handle large amounts of data, something in the 10K range. The delay to get this data can be slightly bigger, as these queries will be executed in parallel to the queries that populate other parts of the UI.
- [x] The fetched data should be pseudo-random. This is the only way we get a representative sample.
- [x] `SearchContext` needs the ability to display different sets of results with different pagination.
### 5. Other changes
This new feature will also affect other parts of the UI:
- [x] The dashboard needs to be collapsible.
- [ ] ~~We need a URL serialization schema for these kinds of widgets.~~ Url serialization probably not enough. Needs to be serialized in JSON.
- [x] We need a place in the search UI where these more general widgets can be added. The initial idea is to have buttons for different widgets above the dashboard.
- [x] We need to start using the term "widget" in our codebase and in the UI. Currently, we say that we are "anchoring a statistic", but the new terminology should revolve around "displaying a widget".
- [x] The initial dashboard for each app is loaded using configs from nomad.yaml.
- [x] The dashboard config needs JSON export/import functionality.
- [ ] ~~Download data from the chart as CSV with a button. Maybe this [dash component](https://dash.plotly.com/dash-core-components/download?_gl=1*5ehz72*_ga*NzA4MTI5MTg1LjE2NDQ5MzQ2MTE.*_ga_6G7EE0JNSC*MTY2NTc0NTUzNy43LjAuMTY2NTc0NTUzNy4wLjAuMA..) helps.~~_Will be done later_Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/960Better unit options2023-12-01T06:01:19ZLauri HimanenBetter unit optionsOur ELNs already have a fairly sophisticated way of defining custom units, but this same flexibility is not currently offerred for the globa units in the main NOMAD app. To rectify this, we need:
- [ ] Add new basic units in the dropdow...Our ELNs already have a fairly sophisticated way of defining custom units, but this same flexibility is not currently offerred for the globa units in the main NOMAD app. To rectify this, we need:
- [ ] Add new basic units in the dropdowns (cm, nm, ns)
- [ ] Add possibility for defining a custom unit by typing it
- [ ] Add possibility to persist unit choices per device (localStorage) (later these settings might get tied to your profile).Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/922Refactor NORTH and Jupyterhub deployment2023-02-15T14:54:30ZAdam FeketeRefactor NORTH and Jupyterhub deployment- [x] Custom jupyterhub_config.py or hub image for supporting different applications (named servers)
- [x] setup upload folders (dynamic) storage. Mount all the uploads.
like: `uploads/uid/`
- [x] shared folders: shared/user (read only...- [x] Custom jupyterhub_config.py or hub image for supporting different applications (named servers)
- [x] setup upload folders (dynamic) storage. Mount all the uploads.
like: `uploads/uid/`
- [x] shared folders: shared/user (read only) VS sandboxAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/753Material UI2023-12-21T15:55:41ZLauri HimanenMaterial UIWe need to start prototyping UI functionality that could possibly replace the current NOMAD Encyclopedia.We need to start prototyping UI functionality that could possibly replace the current NOMAD Encyclopedia.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/933Visualization of multiple columns parsed with TableData2023-12-21T15:39:13ZAndrea AlbinoVisualization of multiple columns parsed with TableDataIn the following example, I parse several columns belonging to a recipe description.
The visualization as separate arrays makes it difficult to retrieve info from the same step.
Let's say that I want to see all the parameters at the s...In the following example, I parse several columns belonging to a recipe description.
The visualization as separate arrays makes it difficult to retrieve info from the same step.
Let's say that I want to see all the parameters at the step no. 7, I have to navigate to each of them and fish the 7th element.
How can we deal with this kind of "row visualization" of multiple columns?
```
sub_sections:
process_steps:
# repeats: true
m_annotations:
eln:
section:
base_section: nomad.parsing.tabular.TableData
quantities:
data_file:
type: str
description: |
A reference to an uploaded .csv
m_annotations:
tabular_parser:
sep: ','
comment: '#'
browser:
adaptor: RawFileAdaptor
eln:
component: FileEditQuantity
step_duration:
type: np.float64
unit: minute
shape: ['*']
description: Past time since process start
m_annotations:
tabular:
name: Duration
elapsed_time:
type: np.float64
unit: minute
shape: ['*']
description: Duration of each step
m_annotations:
tabular:
name: Timestamp
step_number:
type: int
shape: ['*']
description: sequential number of the step on going
m_annotations:
tabular:
name: Step
step_title:
type: str
shape: ['*']
description: what this step consists of
m_annotations:
tabular:
name: Reason
step_comment:
type: str
shape: ['*']
description: more verbose description of the step
m_annotations:
tabular:
name: Description
step_annotation:
type: str
shape: ['*']
description: final remarks on the step
m_annotations:
tabular:
name: annotations
argon_flow:
type: np.float64
shape: ['*']
unit: centimeter ** 3 / minute
description: Argon process gas flow in to the reaction chamber
m_annotations:
tabular:
name: Argon flow
oxygen_flow:
type: np.float64
shape: ['*']
unit: centimeter ** 3 / minute
description: Oxygen process gas flow in to the reaction chamber
m_annotations:
tabular:
name: Oxygen flow
argon_push_titan:
type: np.float64
shape: ['*']
unit: centimeter ** 3 / minute
description: Argon gas in the flash evaporator used to push Ti-Precursor into the reaction chamber
m_annotations:
tabular:
name: Argon push Titan
argon_purge_titan:
type: np.float64
shape: ['*']
unit: centimeter ** 3 / minute
description: Argon gas in the flash evaporator used to purge Ti-Precursor
m_annotations:
tabular:
name: Argon purge Titan
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/962Solar cell ELN base classes2022-09-08T15:43:51ZJose Marquez PrietoSolar cell ELN base classesA set of ELN base classes for describing solar cells and making them searchable by populating quantities in the results section.
- [x] Reference: A class to include a reference to a publication with a DOI.
- [x] SolarCell: Description o...A set of ELN base classes for describing solar cells and making them searchable by populating quantities in the results section.
- [x] Reference: A class to include a reference to a publication with a DOI.
- [x] SolarCell: Description of the solar cell, contains info about layer stack, cell area, architecture...
- [x] SolarCellLayer: Describes a layer within the solar cell stack. Contains `name`, `chemical_formula`, `layer_type`, `thickness`.
- [x] SolarCellLayerWithBandgap: Extends SolarCellLayer adding a `bandgap` and a `bandgap` estimation basis.
- [x] ExternalQuantumEfficiency: EQE data parser and extraction of additional data like `bandgap`, `short_circuit_current_density`...
- [x] CurrentDensityVoltage: Solar cell parameters class. Has a checker to make sure that the data entered is correct (Calculation of efficiency, FF between 0 and 1...).
- [x] A yaml exemplary schema building on these classes.
- [ ] Create a .csv with common cell layers that can be passed to provide suggestions in the eln componentsJose Marquez PrietoJose Marquez Prietohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/989Improvement to docker-compose to support dev work2023-12-21T15:38:43ZSherjeel ShabihImprovement to docker-compose to support dev workWe would like to have a docker-compose file that spins up needed docker containers and mounts our Nomad dev folder from the host machine into these containers. This means we run the local version of our Nomad source code rather than the ...We would like to have a docker-compose file that spins up needed docker containers and mounts our Nomad dev folder from the host machine into these containers. This means we run the local version of our Nomad source code rather than the one compiled into the container image.
This allows us to have almost the same dev environment as a production Nomad Oasis install.
Here are some recommendations or ideas we have already discussed in Area B:
- Make use of the docker-compose from ops/docker-compose/nomad-oasis
- Mount the dev folder into all containers that are not just infrastructure i.e. the app, hub, gui, etc.
- Run yarn in hot reload on the gui folder instead of using the production build for the gui container
- Maybe it is a good idea to check if we need to change/add some helper scripts to reload things like generate_gui_artifacts.sh. We would guess for scripts that just create new files on disk this wouldn't be an issue.
@dflor @sanbrock @kuehbachm @ecarohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/994User defined molecule groups in topology section2023-12-21T15:55:20ZJoseph RudzinskiUser defined molecule groups in topology sectionCurrently we have adopted a standard in the topology (atoms_group) with the following hierarchy: atoms-->monomers (if exist)-->monomer groups (if exist)--> molecules --> molecule groups, with molecule groups for each type of distinct mol...Currently we have adopted a standard in the topology (atoms_group) with the following hierarchy: atoms-->monomers (if exist)-->monomer groups (if exist)--> molecules --> molecule groups, with molecule groups for each type of distinct molecule. We should also have support for user-defined molecule groups. A simple implementation would be that the user provides a yaml/json file with the desired group labels and molecule information (I guess atom indices in general) and group index (I guess this would also somehow need to be linked to a particular main file as well)...And then these custom molecule groups should added on top of the "parsed" molecule groups within the Atoms_group section.
@himanel1 , @ladinesa : Does something like this already exist, e.g., from the Area A/B work? Or do you see a relatively simple way to do this?Joseph RudzinskiJoseph Rudzinskihttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1002Fitting GAP from NOMAD (Oasis)2023-12-21T15:58:53ZSascha KlawohnFitting GAP from NOMAD (Oasis)I would like to add our MPI parallel `gap_fit` (Gaussian Approximation Potential) to the analysis options of NOMAD. I will probably start in our local Oasis at Warwick.
What do I have to do for that?
### ToDo
- [ ] Notebook
- [x] Que...I would like to add our MPI parallel `gap_fit` (Gaussian Approximation Potential) to the analysis options of NOMAD. I will probably start in our local Oasis at Warwick.
What do I have to do for that?
### ToDo
- [ ] Notebook
- [x] Query
- [x] Process
- [x] Run on Cluster
- [ ] Determine job config
- [ ] Uploadhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/958Improve test coverage for perovskite solar cell database integration and opto...2023-12-21T16:06:50ZJose Marquez PrietoImprove test coverage for perovskite solar cell database integration and optoelectronic properties.After the merge of the perovskite solar cell database branch, there are a couple of tests that need to be implemented. These are the following as mentioned by @himanel1:
- GUI tests for the Optoelectronic data.
- Python units tests for ...After the merge of the perovskite solar cell database branch, there are a couple of tests that need to be implemented. These are the following as mentioned by @himanel1:
- GUI tests for the Optoelectronic data.
- Python units tests for parsing the solar cell data.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/964SampleID as SampleidEditQuantity2023-12-21T15:38:46ZAndrea AlbinoSampleID as SampleidEditQuantityLooking at User and Author edit quantities by @mnakh (#918) and at SampleID class by @josma (#945), I would like to propose to make the SampleID in the same shape of AuthorEditQuantity as it is more conveniently shown in the elnLooking at User and Author edit quantities by @mnakh (#918) and at SampleID class by @josma (#945), I would like to propose to make the SampleID in the same shape of AuthorEditQuantity as it is more conveniently shown in the elnhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/948Conditional display of ELN fields2023-12-21T15:55:16ZFlorian DobenerConditional display of ELN fieldsI'm developing on different methods which require to change the input fields depending on the value of an Enum.
This can be as easy as showing or hiding one numeric input if a specific entry is selected up to showing a complete group of ...I'm developing on different methods which require to change the input fields depending on the value of an Enum.
This can be as easy as showing or hiding one numeric input if a specific entry is selected up to showing a complete group of different inputs for each of the Enum selections.
It would be great if this would be possible in the yaml schemes.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/969Testing data compatibility with metainfo schema2022-09-28T07:33:47ZLauri HimanenTesting data compatibility with metainfo schemaWe would like to add some form of testing how the production data complies with the current metainfo schema. This will give some important insights into how well the real data complies with our schema and lets us examine any compatibili...We would like to add some form of testing how the production data complies with the current metainfo schema. This will give some important insights into how well the real data complies with our schema and lets us examine any compatibility problems.
I would envision this as a manually triggered CI task. Would be written in python since we can easily load the data with python libs, and also perform the schema tests easily by loading the metainfo.Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1020Improve selections for ELN base sections and references2022-12-05T07:34:56ZMarkus ScheidgenImprove selections for ELN base sections and references- We only show pulldowns with plain lists of available schemas when creating ELN entries in CreateEntry
- We only show plain lists when showing a list of potential reference targets in RefernceEditQuantity- We only show pulldowns with plain lists of available schemas when creating ELN entries in CreateEntry
- We only show plain lists when showing a list of potential reference targets in RefernceEditQuantityMohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1113New config settings an related GUI features2022-12-16T11:15:29ZJose Marquez PrietoNew config settings an related GUI featuresThis is a collection of features that would make the experience nice in the app's concept in search entries in general.
- [x] Allow passing the text editable components of the search filter components in the left panel to the central p...This is a collection of features that would make the experience nice in the app's concept in search entries in general.
- [x] Allow passing the text editable components of the search filter components in the left panel to the central panel when clicking the "+" button to add them to the central widgets.
- [x] The search results table does not support units at the moment.
- [x] An option to limit the number of significant digits in `config.py` displayed for a quantity would be good. (See the attached screenshot as an example)
- [x] Pre-defined a search filter for looking only at a pre-filtered set of entries. i.e. results.material.functional_type=solar cell for looking at only Solar Cells in the solar cell app.
- [x] Columns from text quantities of type arrays get displayed by joining the items of the array without any separator.
![image](/uploads/8c890ada225f472ab8c5c17289aaa5cd/image.png)Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1202Nomad config formalization2022-12-20T12:25:29ZLauri HimanenNomad config formalizationWe need to formalize our noamd.yaml file structure with a proper datamodel and versioning. The best approach might be to define it as a pydantic model wit h proper descriptions, validation, (de)serialization etc.
This should be started ...We need to formalize our noamd.yaml file structure with a proper datamodel and versioning. The best approach might be to define it as a pydantic model wit h proper descriptions, validation, (de)serialization etc.
This should be started with a first version that:
- [x] Define a basic pydantic model with descriptions and data types, no complex validation yet
- [x] Add the possibility to create a markdown document from the model that can be put into our docs.
- [x] Recreate the existing functionality: defaults->config file->env variables->derived values
- [ ] Add a separate version number to the config model. This will allows us to perform migrations and checks when people use different config versions.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1681Token GUI2023-10-02T15:28:17ZSascha KlawohnToken GUITo make the app token from #1654 easier to use, we should add some GUI elements.
### External Comments
- https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1654#note_210301
- https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/is...To make the app token from #1654 easier to use, we should add some GUI elements.
### External Comments
- https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1654#note_210301
- https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1654#note_210304
### Prototypes
#### New Prototype
![image](/uploads/8d254b27a6e116f3eb1a305eb74bdaea/image.png)
#### Old Prototype
![image](/uploads/4a80ce2bf3acadf2419496f1a9e61b24/image.png)Sascha KlawohnSascha Klawohnhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1849Converter for json to and from pd dataframe2024-02-15T09:12:26ZAmir GolparvarConverter for json to and from pd dataframeflatten json into a pandas dataframe
reconstruct dataframe to json
options: depth of the json to flatten
options: order resultant dataframe alphabetically or by a list of stringsflatten json into a pandas dataframe
reconstruct dataframe to json
options: depth of the json to flatten
options: order resultant dataframe alphabetically or by a list of stringsAmir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1586renaming widget titles2023-10-04T10:20:10ZJulia Schumannrenaming widget titlesCan we have an easy way to rename the widget titles in the explore app? At the moment the title is class and quantity name. There are situations, when I dont want to have the class name, but just the quantity that is presented in the wid...Can we have an easy way to rename the widget titles in the explore app? At the moment the title is class and quantity name. There are situations, when I dont want to have the class name, but just the quantity that is presented in the widget.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1567eln: using label_quantity for complex datastructures2023-12-21T15:55:21ZAdam Feketeeln: using label_quantity for complex datastructuresThe goal: having a list of authors as a subsection where the name is represented as full name.
There are two different issue that might need to be resolved:
- [ ] label_quantity should be able to access property inside a json type quanti...The goal: having a list of authors as a subsection where the name is represented as full name.
There are two different issue that might need to be resolved:
- [ ] label_quantity should be able to access property inside a json type quantity
- [ ] label_quantity should be able to handle of mixing of two properties (anonymous function?) (eg name => first_name + last_name )https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1554Plot support refactor2023-10-17T14:37:33ZLauri HimanenPlot support refactorOur current plotting functionality is quite limited. In customs schemas one can define `PlotAnnotation`s for sections or quantities, but they don't offer enough functionality in many cases.
# Problems
- Only static configuration is all...Our current plotting functionality is quite limited. In customs schemas one can define `PlotAnnotation`s for sections or quantities, but they don't offer enough functionality in many cases.
# Problems
- Only static configuration is allowed. I guess the origin of this is the use in yaml schemas which can't really contain code. What if the layout or plotted traces should change based on data that is being plotted? What if we need to perform some postprocessing to the data before plotting? There was already a request coming from @jschumann about being able to determine the visualization of different lines when it is not known how many lines will be plotted.
- The attributes in our `PlotAnnotation` is not completely aligned with the way Plotly plots are created.
- Plot needs to be associated with a section or quantity. What if I want to define it in a completely custom location?
# Possible solution
Plotly plots can be configured with JSON: Each plot takes in `data`, `layout` and `config`, more info [here](https://plotly.com/javascript/plotlyjs-function-reference/#plotlynewplot). If we are able to produce a single JSON dictionary:
```json
{
data: {...},
layout: {...},
config: {...}
}
```
This could be loaded from our API and visualized directly. It would only be a matter of how this JSON data is created and rendered:
- It could similar to a `derived` value: The data would be created on the fly when requesting it from the API.
- It could be a quantity with JSON dtype that is filled by a normalize-function of a custom schema. This would mean it is persisted always and there is data duplication. There could be a special annotation that tells that Plotly should be used to view this JSON file within an ELN.
- Maybe we could create a special type of `Quantity` for plots? This might be easier for users to grasp. This special quantity could then maybe be configured to either store the plot, or always recreate it on the fly.
This solution would only be available in python schemas, since it requires specifying python code. But I think this should be an acceptable limitation.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1251Add Tight Binding Studio parser2023-11-20T11:31:51ZJose PizarroAdd Tight Binding Studio parserAdding the parser for [Tight Binding Studio](https://tight-binding.com/) package for `Projection` calculations.
- [ ] Add parser and code-specific metainfo for TBStudio.
- [x] Change name from `Projection` to `TightBinding` -> move `Met...Adding the parser for [Tight Binding Studio](https://tight-binding.com/) package for `Projection` calculations.
- [ ] Add parser and code-specific metainfo for TBStudio.
- [x] Change name from `Projection` to `TightBinding` -> move `Method.Projection.Wannier` to `Method.TightBinding.Wannier` -> **NO**, keep `Projection` for now to differentiate from DFTB+ and xTB.
- [ ] Create metainfo for `Method.Projection.SlaterKoster` (input parameters defining the calculation).
- [ ] Define common metainfo for Projection model fittings (n_orbitals? n_bands?).
- [ ] Populate `Calculation.HoppingMatrix` following Wannier90 style.
- [ ] Define `Projection` workflow, and connect TBStudio with the original DFT calculation using upload_id/entry_id.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/896ELN for integration (elabFTW, labfolder, chemotion)2023-12-21T15:38:50ZMarkus ScheidgenELN for integration (elabFTW, labfolder, chemotion)Labs are using ELNs to acquire lab data. ELN is a broad term with many slightly different interpretations and there is a wide variety of ELNs with some functionally shared and some functionality being unique. The overall FAIRmat approach...Labs are using ELNs to acquire lab data. ELN is a broad term with many slightly different interpretations and there is a wide variety of ELNs with some functionally shared and some functionality being unique. The overall FAIRmat approach calls for an integration of ELNs into NOMAD. Currently, it is fully open how such an integration can look like. Ideally, we would survey some common information from all potential ELNs.
This might help as an introduction: https://www.youtube.com/watch?v=6mXJJH_sTVw&list=PLrRaxjvn6FDVZvROmQW_zmIyhDMMJfwZL&index=2
We can collect all information in this thread for now.
These these the ELN that come up most often in FAIRmat context:
- [x] elabFTW
- [x] labfolder
- [x] chemmotion
This is the information we need:
- license information, business/development model, i.e. can we use it, can we contribute to it, etc.
- technology stack: front end, back end, how is the data internally stored
- data model: which entities are supported, key properties for each entity, can the entities be extended, is there a schema, can the scheme be extened, support for standardized formats.
- API: does it exist, link to API reference documentation, what can be exported with API (data, metadata, files, which entities, etc.), API standards used, a few example requests/responses
- how does the user management work, how does one authenticate in the API, can the user management be "linked" with other IDPs
We should have a table with very brief information on all ELNs and a section on each ELN with details.
Why is this information important? Ideally, we could model the entities of each ELN as a metainfo package and use ELN APIs to create uploads with all data from an ELN (or some query on an ELN) that populate the ELN's metainfo definitions.Amir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1654Application Token2023-09-19T14:01:53ZSascha KlawohnApplication TokenTo enable easier API access for external applications we should add another authentication method using application tokens (app token). A user can request a new app token via API/GUI including an expiration duration, authenticating initi...To enable easier API access for external applications we should add another authentication method using application tokens (app token). A user can request a new app token via API/GUI including an expiration duration, authenticating initially through another method. The returned token can then be used to perform actions with the API.
- Use case
- Long-living token to access NOMAD API
- Presented to the user only once at creation (GUI/API)
- Enforcing expiration, user-defined but admin-limited
- Endpoint `auth/app_token?expires_in=3600`
- New auth method (others reuse `Authorization: Bearer <token>`)
- ~~Query `?app_token=<token>`~~ Rather not for long-living tokens.
- Header `Authorization: Application <token>`?
- Usage equal to authentication scheme `Bearer`, should use that instead.
- May distinguish via prefix, e.g. `app_<token>`.
- JWT token, cf. with signature token
- Less verbose than keycloak's combined id/access token
- Alternative: Opaque string
- Configuration
- `APP_TOKEN_MAX_EXPIRE_IN`, default ~~`400 d`~~ `10d`?
- (`APP_TOKEN_MAX_ENTRIES`?, default `10`?)
- Extensions
- More granularity (fine-grained rights, or scope)
- Revocable token storage
#### Thoughts
Currently, we appear to be looking rather for a Personal Access Token to impersonate a user than a dedicated **App** Token.
There are already several auth dependencies:
- `basic_auth`: username and password
- `bearer_token`: more precisely Keycloak's OAuth JWT
- `upload_token`: persistent via user id
- `signature_token`: simple, self-made JWT; only contains user id and expiration time
So we already use three different auth sources, keycloak and two self-made solutions.
#### Links
- [Auth Tokens at GitHub](https://docs.github.com/en/rest/overview/authenticating-to-the-rest-api?apiVersion=2022-11-28#authenticating-with-a-personal-access-token)
- https://github.blog/changelog/2021-03-31-authentication-token-format-updates-are-generally-available/
- https://github.blog/2021-04-05-behind-githubs-new-authentication-token-formats/
- https://github.blog/2022-10-18-introducing-fine-grained-personal-access-tokens-for-github/
- [Token overview at GitLab](https://docs.gitlab.com/ee/security/token_overview.html)
- [Best practices for API access](https://dzone.com/articles/security-best-practices-for-managing-api-access-to)
- [MDN Web Docs: HTTP Authentication](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication)
- [Opinion: Short-lived JWT, long-lived opaque refresh token](https://supertokens.com/blog/are-you-using-jwts-for-user-sessions-in-the-correct-way)
- The refresh token is allowed to be long-lived since you have to get the new short-lived access token from the server, where access rights and revocations can be checked.Sascha KlawohnSascha Klawohnhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1244Scatter plot update2023-12-21T15:55:42ZLauri HimanenScatter plot update- [ ] Improved tooltips when hovering over items. Should show at least x, y, color, entry_id, if possible also a link to the entry.
- [ ] Fix issue when updating one of the axis and doing a region selection after the plot has been create...- [ ] Improved tooltips when hovering over items. Should show at least x, y, color, entry_id, if possible also a link to the entry.
- [ ] Fix issue when updating one of the axis and doing a region selection after the plot has been created.
- [ ] The component should show some notifications about the data: The number of retrieved samples, and some indication of whether all available data was loaded or not.
- [ ] Ability to choose the unit for the x, y, and color axes.
- [ ] Support for timestampsLauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1805Separating the plugin's options from the nomad.yaml file2024-03-20T10:08:09ZAdam FeketeSeparating the plugin's options from the nomad.yaml fileIt would be nice to separate ui/plugin options in the plugin and nomad.yaml. It would be nice to define `ui > apps > options ...` in nomad_plugin.yaml.
- options' attribute of ui/plugin could be part of nomad_plugin.yaml file
- it would ...It would be nice to separate ui/plugin options in the plugin and nomad.yaml. It would be nice to define `ui > apps > options ...` in nomad_plugin.yaml.
- options' attribute of ui/plugin could be part of nomad_plugin.yaml file
- it would behave as defaults so you can still override with nomad.yaml
- nomad.yaml would mostly control exclude/include attribute
Use cases:
- @sbrueck 's use case: installing multiple plugins using in a nomad oasis installation. some of the plugins depends on python packages not available in the main image so a custom image needs to built which would contain all plugins and them configuration.
- @afekete 's use case: the support for jupyter notebooks will be implemented as an external parser but should be still part of the core nomad package. It would be nice to use the plugin's yaml file to store its configs instead of putting them as default values of the main configuration.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1790Improved searchbar2024-01-15T09:07:50ZLauri HimanenImproved searchbarAs brought up by Oasis users (e.g. @alfu) in the project meeting, we should add support for a fully free-text search. This means that you can type in any text in the search bar, and get the best matches without having to have exact match...As brought up by Oasis users (e.g. @alfu) in the project meeting, we should add support for a fully free-text search. This means that you can type in any text in the search bar, and get the best matches without having to have exact matches or having to target a specific field in the metainfo.
My suggestion would be to have a generic text mapping in our metadata section that would be reserved for this purpose. This would generalize across all entries, and filling it could be even done through the ES annotation, with some intelligent default (all string fields in a schema...?). This field could also be used to test some better suggestion mechanisms that could work with filters.
This addition has a big impact on the search bar logic: we need to fully refactor how suggestions are prioritized and shown, and make sure that the UX does not suffer. Also, we need a mechanism for deciding which types of queries are available in the apps: free-text search should be initially disabled in the most basic apps as it will create confusion due to not returning any results when dealing with computational and not-reprocessed data.
Tasks:
- [x] Add new metainfo that stores the free-text data(e.g. `metadata.keywords`)
- [x] Setup the ES config with an annotation. Should be annotated with `text`, queried with `match`. Default analyzer should be used for now.
- [x] Add logic for populating the free-text field: all string quantities in custom schemas should be considered. Every _unique_ string (whitespace in beginning and end removed) field should be added as a new entry into the free-text search. By using unique values we can avoid populating terms that are used multiple times e.g. in repeating sections.
- [x] The apps should be able to define whether they want to enable free-text queries. It is impossible to automate this decision reliably, because an app may work with any kind of data which may or may not have the `keywords` field. This functionality can be added in a generic way, which can later be used to control/add new kinds of query syntaxes for the search bar.
- [x] Sorting relevance logic should be changed:
- [x] By default sorting happens by what is set in the app
- [x] When a free-text query is executed, sorting should be changed to relevance automatically
- [x] If a user explicitly sets sorting, this is used until a new free-text query is created
- [x] Improve search bar UX:
- [x] The suggestions type should be shown, and the priority of the suggestions should be logical
- [x] Executed queries should be stored into `localStorage` so that we can show suggestions from history
- [x] Items in search history can be deleted
- [x] Autoselect text when navigating suggestions (helps with e.g. modifying a query in history)
- [x] Add a better explanation of the available search queries
- [x] Add tests that check that correct string quantities are added to the new quantity
- [x] Add search tests that check that the API does the correct queries on this field
- [x] Extend search bar tests
In the future, this feature could also be extended using some LLM model: the `metadata.keywords` field would be translated into a latent space vector of an LLM and queries would be performed against it.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1732Scatter plots with multiple values per entry2024-02-27T15:46:38ZLauri HimanenScatter plots with multiple values per entryThe scatter plot widget is too limited at the moment: it can only be used to display scalar values from non-repeated structures.
To improve this, the following should be done:
- [x] Extend the widget pydantic model to include
- [x] I...The scatter plot widget is too limited at the moment: it can only be used to display scalar values from non-repeated structures.
To improve this, the following should be done:
- [x] Extend the widget pydantic model to include
- [x] Instead of providing a string for `x`, `y`, and `color`, we should use more complex configuration objects that can include units, labels etc.
- [x] Document the possibility of using JMESPath syntax on `quantity`
- [x] Add `root_validator` that ensures backward compatibility with old models.
- [x] Create a new function utility function `parseJMESPath` for parsing JMESPath syntax
- [x] Create tests for `parseJMESPath`:
- [x] Raise JMESPath errors correctly
- [x] Handle functions
- [x] Handle selectors
- [x] Handle quantity names with schema + dtype
- [x] Create new input component `InputJMESPath` for JMESPath queries
- [x] Create tests for `InputJMESPath`:
- [x] Raise JMESPath errors correctly
- [x] Only allow values from current `SearchContext`
- [x] Create tests for `WidgetScatterPlot`:
- [x] Test that using quantity names that target scalar values works as normal
- [x] Test that custom axis labels are shown correctly
- [x] Test that custom units are handled correctly.
- [x] Test that using (subset) of JMESPath syntax works:
- [x] selection: `results.material.topology[0].cell.a`
- [x] functions: `max(results.material.topology[*].cell.a)`
- [x] Create tests for `WidgetScatterPlotEdit`:
- [x] Test that using invalid JMESPath syntax gives a warning
- [x] Test that targeting unavailable quantities gives warning
- [x] Test that invalid units gives warning
- [x] Test that incompatible units gives warning
- [x] Test that missing required values units gives warning
- [x] Test that targeting repeatable data with no JMESPath syntax gives a warning.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1740Search interface improvements2023-11-01T06:55:19ZLauri HimanenSearch interface improvements- Change label for Structure to be Structure / Symmetry
- Add a default dashboard for the "Calculations" app:
- Periodic table
- Structure type
- Crystal system
- Space group name- Change label for Structure to be Structure / Symmetry
- Add a default dashboard for the "Calculations" app:
- Periodic table
- Structure type
- Crystal system
- Space group nameLauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1689Apps for python schemas2023-10-14T11:11:48ZLauri HimanenApps for python schemasWe need to ensure that app definitions targeting quantities in Python schemas can be created.
# Tasks
- [x] Widgets can be configured for scalar quantities within a schema
- [x] Result columns can target scalar quantities
- [x] Result c...We need to ensure that app definitions targeting quantities in Python schemas can be created.
# Tasks
- [x] Widgets can be configured for scalar quantities within a schema
- [x] Result columns can target scalar quantities
- [x] Result columns can be sorted by scalar quantity
- [x] The available filters can be limited with the `filters`-field.
# Decisions
We need to make a decision about the syntax for targeting custom quantities:
- That path in the data is no longer enough to target a quantity. E.g. `data.sample.id` may be defined by several different schemas. We need to somehow add an identifier for the schema in the quantity name.
- Schemas are identified using the `qualified_name` of the root section:
- Python schemas: path of the class name, e.g. `nomadschemaexample.schema.MySchema`
- YAML schemas: `entry_id:gTqaJYQ7IH20dl5PeX7ZPzkHudI8.MySchema`
- Current proposal: use `path` + `separator` + `schema` to target a quantity in the schema. E.g. `data.sample.id#nomadschemaexample.schema.MySchema`.
- The GUI will support simplified rendering of schema identifiers to keep things simple, but in the app config and in our backend the quantities need to be identified by this full name.
- We need to make a decision on the separator. This is complicated by the fact that many separator are reserved for other purposes:
- `.` denotes section hierarchy in paths, and is also used in schema identifiers.
- `:` is used in the YAML schema name and also for query modifiers, e.g. `material.elements:all: ['Si', 'C']`
- `/` is used in inner_section_definitions and in YAML schema paths
- `&` is reserved for url query parameters
- `@` is reserved for indicating a hash digest for a definition. Used to e.g. distinguish between different versions of a schema.
- Any operators commonly used in boolean logic (+, -, &, |) should not be used if we want to later add support for them in the search bar.
# Known limitations compared to "native" quantities
- Search boxes cannot show suggestions for values (due to the hierarchy being flattened within ES, our [suggestion mechanism](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html#completion-suggester) cannot filter out the values to show).
- Only scalar quantities are available (might be possible to get around, not sure)
- [Nested queries](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-nested-query.html) are not possible (due to the hierarchy being flattened in ES)
# Screenshot
![image](/uploads/7eba0ac824529457d53caab7089230d1/image.png)Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1430Workflow for chemical reaction steps2023-10-23T16:51:58ZNathan DaelmanWorkflow for chemical reaction stepsA workflow ready to contain a general (computational) **elementary reaction step** and extract the energy, e.g. calculating adsorption energies: Fe_50 + CO_gas -> Fe_50CO, E[Fe_50CO] - E[Fe_50] - E[CO_gas].
In general, there can be m rea...A workflow ready to contain a general (computational) **elementary reaction step** and extract the energy, e.g. calculating adsorption energies: Fe_50 + CO_gas -> Fe_50CO, E[Fe_50CO] - E[Fe_50] - E[CO_gas].
In general, there can be m reactants and n products.
(We could also consider a reaction with a transition state, which has a 3rd intermediate state?)
- [ ] Add example data.
- [ ] Define a workflow schema with input section and yaml (reactants vs products).
- [ ] Build workflow normalizer to compute (adsorption) energies.
- [ ] Add a tester (based on the example data).
Set up jointly with @jschumann and @ladinesa .Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1464MOF normalizer2023-06-19T07:55:06ZLauri HimanenMOF normalizerWe should develop a normalizer that automatically extracts information about porous materials such as MOFs and zeolites. @diwon has already developed the MOFStructures parser together with Alvin, but we should generalize and extend it's ...We should develop a normalizer that automatically extracts information about porous materials such as MOFs and zeolites. @diwon has already developed the MOFStructures parser together with Alvin, but we should generalize and extend it's functionality.
The plan is to create a new normalizer that will automatically detect porous materials and calculate their properties. This would run on top of all data which contains periodic, atomistic structures.Dinga WonankeDinga Wonankehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/959Make nomad/parsing/tabular/TableRow accepting files with different names and ...2022-10-21T14:08:05ZAndrea AlbinoMake nomad/parsing/tabular/TableRow accepting files with different names and passing pandas annotationsI needed to parse an excel table line by line as in the periodic table example.
It would be nice to extend also in this class the possibility to load a file with a name different from the yaml schema one and also to pass the annotations...I needed to parse an excel table line by line as in the periodic table example.
It would be nice to extend also in this class the possibility to load a file with a name different from the yaml schema one and also to pass the annotations to the read_table_data function, to benefit, for example, of comment lines.Amir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/547OpenKIM database2023-12-21T15:58:52ZAlvin Noe LadinesOpenKIM databaseI recently implemented a new openkim parser that parse a downloaded json file from the openkim database. However, I am not quite sure if we should adapt what was done for the eels parser which only provides a reference to the eels data. ...I recently implemented a new openkim parser that parse a downloaded json file from the openkim database. However, I am not quite sure if we should adapt what was done for the eels parser which only provides a reference to the eels data. Also, do we proceed writing a crawler for openkim? I am not sure if there exists an agreement with openkim. In addition, I have no idea how big their repository is. @mscheidg what do you think?https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/928Incosistent use of integrated DOS values2023-12-21T15:58:52ZNathan DaelmanIncosistent use of integrated DOS valuesThe integrated DOS value is shown only in codes which output it (e.g. VASP).
However, given the regular DOS, the integrated counterpart can easily be calculated using a running sum.
For a user unfamiliar with the code output formats, it ...The integrated DOS value is shown only in codes which output it (e.g. VASP).
However, given the regular DOS, the integrated counterpart can easily be calculated using a running sum.
For a user unfamiliar with the code output formats, it is jarring to see integrated DOS "randomly" pop up.
My suggestion:
- Compute the integrated value (in the normalizer?) when one is not given.
- Or remove the Quantity all together.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/887Quality definitions for datasets2023-12-21T15:55:48ZFelix DietrichQuality definitions for datasetsDatasets should have one or more indicators on their "quality", e.g. a star system.
# How does the quality indicator get assigned?
* Check: Does the data satisfy all requirements
* Compute: number of AI toolkit notebooks that use it
...Datasets should have one or more indicators on their "quality", e.g. a star system.
# How does the quality indicator get assigned?
* Check: Does the data satisfy all requirements
* Compute: number of AI toolkit notebooks that use it
* Manual: assign star by users
* ...
# Critical datasets
We need a definition of "critical dataset", to secure these and assure long-term central backup. This relates to the TIER system, where "the most important" data is stored centrally. "Critical dataset" may not mean "good dataset", but critical to be included in the central repository database (e.g. for FAIRmat-political reasons). This means we need (at least) two quality definitions (well managed / complete vs. critical for NOMAD/FAIRmat).
# Todo
- [ ] Define quality indicator for "well managed dataset"
- [ ] Define quality indicator for "critical dataset"
- [ ] Automatic way to compute and assign quality indicator?Felix DietrichFelix Dietrichhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/884Parsing Hubbard parameters2023-12-21T16:06:49ZNathan DaelmanParsing Hubbard parametersThe Hubbard parameters (as listed in the [metainfo](https://nomad-lab.eu/prod/v1/staging/gui/analyze/metainfo/EntryArchive/run/method/dft/dft_plus_u)) should be added to the parser
**Note on the schema:** `dft_plus_u.functional` (as def...The Hubbard parameters (as listed in the [metainfo](https://nomad-lab.eu/prod/v1/staging/gui/analyze/metainfo/EntryArchive/run/method/dft/dft_plus_u)) should be added to the parser
**Note on the schema:** `dft_plus_u.functional` (as defined), is superfluous.
What would be more sensible is renaming it to `dft_plus_u.method`/`dft_plus_u.approach` for catching the physics model, i.e. Lichtenstein, Dudarev, etc.
**Some VASP examples:**
- [zRE1NkTdewiRS3vahFQQ5og0RTbz](https://nomad-lab.eu/prod/v1/staging/gui/search/entries/entry/id/zRE1NkTdewiRS3vahFQQ5og0RTbz) (and all cerium oxide calculations from [10.17172/NOMAD/2020.02.18-3](https://dx.doi.org/10.17172/NOMAD/2020.02.18-3))
- [ZaS0VzyvW0tAGXN4HEwZ9w_0Vo3H](https://nomad-lab.eu/prod/v1/staging/gui/search/entries/entry/id/ZaS0VzyvW0tAGXN4HEwZ9w_0Vo3H) (and most oxides from AFLOW computed at LDA, GGA, or mGGA level)Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/876Incomplete capture of SCAN functional2023-12-21T15:58:51ZNathan DaelmanIncomplete capture of SCAN functionalThe parser only extracts MGGA_X_SCAN, but not its correlational counterpart **MGGA_C_SCAN**, e.g. zqY1ANv-nZa86sVSA3TIlqwzBjZ3. As such, the xc functionals with scan are now incomplete.
Note: libxc contains various SCAN implementations....The parser only extracts MGGA_X_SCAN, but not its correlational counterpart **MGGA_C_SCAN**, e.g. zqY1ANv-nZa86sVSA3TIlqwzBjZ3. As such, the xc functionals with scan are now incomplete.
Note: libxc contains various SCAN implementations. While the one from Sun, Ruzsinszky, and Perdew is the most popular, we should verify with each code's documentation .https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/645Response status code for data deletion2023-12-21T15:40:03ZMohammad NakhaeeResponse status code for data deletionThe delete routes that take time should return response status 202 instead of 200.The delete routes that take time should return response status 202 instead of 200.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/957Standard output / Log file "duplication" for Molecular Dynamics data from Gro...2022-10-12T10:14:09ZJoseph RudzinskiStandard output / Log file "duplication" for Molecular Dynamics data from Gromacs### Example:
upload-id: DBB5jVKpTOmIUOAw9rPLbw in beta version
### NOTE:
In this example, EQ1.log and minimEQ1.log are the proper log files from the equilibration and energy minimization, respectively. They are currently not parsed co...### Example:
upload-id: DBB5jVKpTOmIUOAw9rPLbw in beta version
### NOTE:
In this example, EQ1.log and minimEQ1.log are the proper log files from the equilibration and energy minimization, respectively. They are currently not parsed correctly in the beta version due to a version control problem which is fixed in the dlpoly-fix branch of the atomistic parser.
2678628_2.log is not a proper log file, but is the standard output during the simulation.
### Description of problem:
Typically, when one runs an MD simulation on a cluster, the standard output is saved to a log file. For Gromacs simulations, this log file is recognized as a proper simulation log file by the parser, since it prints the program and version name, e.g., ":-) GROMACS - gmx grompp, 2016.1 (-:" identical to the log file. Thus, in practice, there will be a duplication of entries.
### Possible solutions:
unknown.Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/854Parsing error relativistic data2023-12-21T16:06:51ZNathan DaelmanParsing error relativistic dataUnder Method/Simulation/DFT/Relativity method, _scalar_relativistic_ captures two different methods from CASTEP (Koelling-Harmon) and FHI-aims (zora scalar) under the same name. **The latter should correspond to ZORA**, not be confused w...Under Method/Simulation/DFT/Relativity method, _scalar_relativistic_ captures two different methods from CASTEP (Koelling-Harmon) and FHI-aims (zora scalar) under the same name. **The latter should correspond to ZORA**, not be confused with atomic ZORA, which is also implemented in FHI-aims [1].
All of these methods are scalar relativistic, so **I suggest introducing Koelling-Harmon as its own category**.
[1] [https://doi.org/10.1016/j.cpc.2009.06.022](https://doi.org/10.1016/j.cpc.2009.06.022)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/851Parsing hybrid functional parameters2023-12-21T16:06:51ZNathan DaelmanParsing hybrid functional parameters**General explanation**
Hybrid functionals have, by definition, a mixing parameter (typically abbreviated as alpha) that determines the fraction of Hartree-Fock/Exact Exchange mixed in.
All libxc names come with defaults (most often alp...**General explanation**
Hybrid functionals have, by definition, a mixing parameter (typically abbreviated as alpha) that determines the fraction of Hartree-Fock/Exact Exchange mixed in.
All libxc names come with defaults (most often alpha = 0.25), but many codes also allow the user to tweak these at will.
Actually, it is quite common to recalibrate the alpha to a specific use case/observable.
Hence, the parsers should be able to at least capture this value.
Similarly, the special subclass of short-range hybrid functionals introduce a screening parameter (abbreviated as omega) that cuts off the Exact Exchange after a certain range.
This parameter too is common enough that it has to be captured.
**Specific requests**
1. Parse and store the alpha parameter as a fraction (i.e. ]0,1[) in all electronic structure code that supports hybrid functionals, when present.
2. Likewise, parse and store the omega parameter in Ångstrom.
3. Default values do not have to be exposed (yet).
[xc_hybrid_keywords](/uploads/da311896eedf3d86d8bf8dde0d6dd19e/xc_hybrid_keywords.md.yaml) contains the code-specific keywords (to be updated in the future).https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/837Align nexus schema definition language with metainfo2022-09-06T11:11:39ZMarkus ScheidgenAlign nexus schema definition language with metainfoAdd stuff to `metainfo.py`:
- [ ] names (variable names, names that are not valid identifier)
- [x] groups and fields with attributes
- [x] units on data
- X "dimension" instead of unit –– this is purely informative on the schema level f...Add stuff to `metainfo.py`:
- [ ] names (variable names, names that are not valid identifier)
- [x] groups and fields with attributes
- [x] units on data
- X "dimension" instead of unit –– this is purely informative on the schema level for the metainfo and can be stored with "more"
- X required property –– this is purely informative on the schema level for the metainfo and can be stored with "more"
- [x] descriptions for enum values (#655)
Fix the rest:
- [ ] apply changes to gui
- [ ] apply changes to nxdl conversion
- [ ] apply changes to nexus parserMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/626Fix the similarity CLI commands to support the new materials index2023-12-21T15:40:03ZLauri HimanenFix the similarity CLI commands to support the new materials indexWe have a CLI command for populating the DOS similarity data in MongoDB using an external file. If we want to continue supporting this feature, the command should be updated to use the new data layout.We have a CLI command for populating the DOS similarity data in MongoDB using an external file. If we want to continue supporting this feature, the command should be updated to use the new data layout.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/911Capturing simple molecule + surface systems in the topology2023-12-21T15:58:51ZLauri HimanenCapturing simple molecule + surface systems in the topologyAfter issue #855, we are able to identify surfaces within the data with fairly good accuracy. Another interesting extension to this would be to try to capture surface + molecule combinations = adsorption systems (there are probably +10k ...After issue #855, we are able to identify surfaces within the data with fairly good accuracy. Another interesting extension to this would be to try to capture surface + molecule combinations = adsorption systems (there are probably +10k of these in NOMAD). To do this, we need to
- [ ] Modify the `topology` function in nomad/normalizing/material.py:
- After the clustering by MatID, you can run the network analysis for the remaining outlier atoms, and if they are connected + are organic (C, H, O, N, S, F?) the outliers are classified as molecule.
- [ ] Add unit tests:
- One can find nice examples in the search with structural_type=surface + some organic chemical element. Maybe 5-6 examples will be a good start. Including a system that is "split" by the periodic boundary would be good to make sure that our PBC handling is correct.
- Test that both the surface and the molecule are correctly identified (structural_type=surface/molecule). The molecule should have 100% correct indices.
- Test that in systems with several unconnected outliers, only the surface is detected. In such systems, there is a higher chance that what we think is a molecule is actually something else. We can think about this more when we get some experience with handling single molecules.Thea DenellThea Denellhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/980GW visualization2022-11-22T10:50:35ZAlvin Noe LadinesGW visualizationSuggestions:
1. reprocess existing GW data (still classified as geometry optimization, should be single point)
2. band structure and dos for both dft and gw should be shown together for comparison
3. should be possible to create separat...Suggestions:
1. reprocess existing GW data (still classified as geometry optimization, should be single point)
2. band structure and dos for both dft and gw should be shown together for comparison
3. should be possible to create separate entries from one mainfile that performs workflow calculation (at the moment one archive containing multiple calculations)Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/358Non local factors in derived properties2023-12-21T15:39:06ZLauri HimanenNon local factors in derived propertiesSome derived properties, take large data (e.g. a matrix) and normalize it based on small factors (e.g. for scaling or shifting). While the normalized data is usually part of the same section, the scaling factors might come from different...Some derived properties, take large data (e.g. a matrix) and normalize it based on small factors (e.g. for scaling or shifting). While the normalized data is usually part of the same section, the scaling factors might come from different sections. To evaluate such a derived property, not just the local data, but also the distant sections containing factors need to be available (e.g. loaded from the archive). This breaks the otherwise local nature of quantities, i.e. you cannot necessarily access the quantity on its section alone.
There are several thinkable solutions:
- the quantity stores the factors as a tuple
- the factors are assigned to the quantity (e.g. by parser/normalizer) and passed to the derive function as parameters
- the factors are defined in the quantity definitions and automatically stored when the archive is written
- we shoehorn this into quantity references. The serialized reference would be something like `/run/scc/20/dos/values#normalized?cell_volume=0.19282e-12`.
- the parameters have to be provided when the reference is set
- the parameters are somehow automatically set on serialization
- the actual computation is added to the values quantity definition
- the actual computation is added to the reference quantity definition
### the original post
I'm facing a small dilemma with the derived metainfo quantities.
The metainfo `specific_heat_capacity` is simply a re-weighted version of `thermodynamical_property_heat_capacity_C_v`. Due to this simple relation, the metainfo is implemented as a derived quantity: it is calculated upon access. These derived properties are not stored in the archive and are not available until a proper MSection is de-serialized from the underlying data.
In the API I would want to access this information. Because of the derived nature, I tried to de-serialize only the parent section `section_thermodynamical_properties`. This can be done easily. But when I access the derived property, the calculation needs to resolve several references to obtain the normalization constant (in this case it needs to resolve the `section_system` that was used for calculating this property). To solve these references I thus basically have to deserialize the entire `section_run`, and only after that, all the references can be resolved properly.
Question: Although this mechanism works in practice, it means that upon accessing any derived property, there is a possibility that the whole Archive needs to be de-serialized into an MSection object in RAM to resolve the needed properties. This may become problematic and I'm wondering if there are better alternatives? This issue will affect all properties where we have have both the normalized and non-normalized version of the same data available (DOS, band structures, etc.). I'm wondering if it would make sense to add a new type of quantity (e.g. QuantityNormalized) that would explicitly store the normalization constants in the archive? This would be an intermediate solution between storing the full normalized data and using the derived properties.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/506Incomplete SCC in VASP geometry optimization2023-12-21T15:58:50ZLauri HimanenIncomplete SCC in VASP geometry optimizationThe last step of a geometry optimization workflow in entry [-4oK7xt-I9sbkL7684cpd4v5TrS_](https://nomad-lab.eu/prod/rae/beta/gui/entry/id/10k9kKRNTiWFsaK56Ft3og/-4oK7xt-I9sbkL7684cpd4v5TrS_/archive/section_run/section_single_configuratio...The last step of a geometry optimization workflow in entry [-4oK7xt-I9sbkL7684cpd4v5TrS_](https://nomad-lab.eu/prod/rae/beta/gui/entry/id/10k9kKRNTiWFsaK56Ft3og/-4oK7xt-I9sbkL7684cpd4v5TrS_/archive/section_run/section_single_configuration_calculation:43) is mostly empty (also many others from this particular upload have the same problem). I can also reproduce this issue with a newer processing version (0.10.0). Does not happen in all VASP geometry optimizations.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/886Link between NOMAD OASIS and NOMAD Central2023-12-21T16:05:24ZFelix DietrichLink between NOMAD OASIS and NOMAD Central## General
Currently every OASIS is just a clone of the central repository.
They are not linked together, not even through simple pings.
In addition to simple links, it is necessary to define how OASES can differ from the central instal...## General
Currently every OASIS is just a clone of the central repository.
They are not linked together, not even through simple pings.
In addition to simple links, it is necessary to define how OASES can differ from the central installation (in their version, their data, their user rights, ...).
This may be communicated to central, too. In this issue, we outline these challenges and provide a road map for implementation and discussion.
## Current GIT Issues related to OASIS (2022-05-24)
* Improve oasis synchronisation functionality #585
* Integrating NORTH into OASIS #825
* Oasis installation and registration #820
* NOMAD Oasis (GUI) #784
* NOMAD Oasis UX Test: user roles #757
## Requirements: LINK
* Use case 1: Send telemetry data to central in regular intervals. This is discussed in {## Requirements: Telemetry}
* Use-case 2: send data buckets + metadata. This is discussed in section {## Requirements: Data LINK}
## Requirements: Telemetry
* Which data should be sent?
* Users registered
* Data uploaded, stored
* Number of searches
* OASIS location
* Do we need fine-grained info about site-access? This may be a security/leak risk that some may not like.
* See #820
## Requirements: Soft constraints / Installation
* Central: be careful with DDOS attacks from "too many / malicious OASIS links"
* LINK is opt-out, enabled by default
* OASIS owners can disable LINK in a config
* See roles, #757
* The fact that each OASIS sends telemetry data by default MUST be communicated clearly, also how to disable it
* We must be careful with "user data processing" requirements (GDPR). Aggregated data should be fine, but maybe some telemetry data can be used to identify users (link clicks, etc.).
## Requirements: Data LINK
* Also called "Synchronization"
* Send data buckets
* Send metaschema information, when they are stored in files
* How to send new metaschemas defined in python? Not send at all?
* Versioning of data: what if metaschema in the OASIS is newer/older than the one in central?
* See #784
* See #585Daniel LehmbergDaniel Lehmberghttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/628Better user-magement2023-12-21T15:55:38ZMarkus ScheidgenBetter user-magement- upgrade keycloak (to version 8?), keeping our custom interface
- connect via OAuth (e.g. with github, google)
- connect via SAML (e.g. with DFN AAI)
- allow local keycloaks for Oasis (why?)
- common user id (e.g. ORCID)
- consolidate u...- upgrade keycloak (to version 8?), keeping our custom interface
- connect via OAuth (e.g. with github, google)
- connect via SAML (e.g. with DFN AAI)
- allow local keycloaks for Oasis (why?)
- common user id (e.g. ORCID)
- consolidate users (e.g. automatically based on ORCID, CLI functions to migrate users)
- properly use (or don't use at all) affiliations
- allow authors that are not usershttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/402Enforcing HTTPS across all services2023-12-21T15:40:02ZLauri HimanenEnforcing HTTPS across all servicesCurrently many of the resources that our production machine serves can be retrieved both with http or https protocols.
As far as I understand, the standard these days is to always use https, no matter what the resource is. This ensures ...Currently many of the resources that our production machine serves can be retrieved both with http or https protocols.
As far as I understand, the standard these days is to always use https, no matter what the resource is. This ensures that all outgoing data is always properly secured (we don't have to selectively enable https, as that is prone to mistakes), and that there will be no issue with resources interacting with different protocols.
Doing this should be relative easy: we will still accept incoming requests through http, but these requests will always be redirected to https. This can be done through our nginx server, [with something like this.](https://serversforhackers.com/c/redirect-http-to-https-nginx) You can see that most sites do something similar, e.g. if you try to load this issue page with http, Gitlab will automatically switch to https.
What do you think?https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/551Metainfo update: Energy, Stress, Forces2023-12-21T15:38:42ZLauri HimanenMetainfo update: Energy, Stress, ForcesHere are a few comments:
1. Having a list of repeatable sections (`energies`, `forces`, `stress`) where the contribution type is identified by a metainfo _value_ has some drawbacks: In order to find a specific value, you have to loop ov...Here are a few comments:
1. Having a list of repeatable sections (`energies`, `forces`, `stress`) where the contribution type is identified by a metainfo _value_ has some drawbacks: In order to find a specific value, you have to loop over the entire list. I think a better solution from the perspective of finding information would be to have something like this.
```python
class Energy(MSection):
m_def = Section(
validate=False,
description="""
Section describing a type of energy or a contribution to the total energy.
"""
)
reference = Quantity(
type=np.dtype(np.float64),
shape=[],
unit='joule',
description='''
Value of the reference energy to be subtracted from value to obtain a
code-independent value of the energy.
''')
value = Quantity(
type=np.dtype(np.float64),
shape=[],
unit='joule',
description='''
Value of the energy identified by energy_kind.
''')
value_per_atom = Quantity(
type=np.dtype(np.float64),
shape=[],
unit='joule',
description='''
Value of the energy normalized by the total number of atoms in the simulation cell.
''')
values_per_atom = Quantity(
type=np.dtype(np.float64),
shape=['number_of_atoms'],
unit='joule',
description='''
Value of the atom-resolved energies identified by energy_kind.
''')
class EnergyTotal(Energy):
m_def = Section(
validate=False,
description="""
Contains values for the total energy.
"""
)
class EnergyKinetic(Energy):
m_def = Section(
validate=False,
description="""
Contains values for the kinetic energy.
"""
)
energy_total = SubSection(sub_section=EnergyTotal.m_def)
energy_kinetic = SubSection(sub_section=EnergyKinetic.m_def)
...
```
Comparing the process of finding information (in Python, but the same logic applied to Javascript used in GUI):
1. `energy_kinetic = next((x.energy_value for x in energies if x.energy_kind == "kinetic"),
None)`
2. `energy_kinetic = energy_kinetic.value`
This does not, however, allow easily adding new energy types, which was maybe your original purpose. But I think we should try to optimize the process of information lookups, even if we have to make storing the information a bit harder (=you just need to explicitly create metainfo sections).
2. I'm not sure if we need the `atom_`-prefix used at least in `atom_forces` and `atom_charges`. It would be nice if all force-related values would start with `forces_`-, the same applies for charges, stresses, and energies. But this is a minor thing.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/838Draggable lane separators2023-12-21T15:55:15ZMarkus ScheidgenDraggable lane separatorsWe could allow users to rearrange the size of lanes.We could allow users to rearrange the size of lanes.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/836Timezones2023-12-21T15:38:42ZMarkus ScheidgenTimezonesWe were never looking into timezones before. This becomes a problem now that people use ELNs and care about times and not just dates. In prinziple most date formats and libraries support to add a timezone to the date. We just have to ens...We were never looking into timezones before. This becomes a problem now that people use ELNs and care about times and not just dates. In prinziple most date formats and libraries support to add a timezone to the date. We just have to ensure that this information is added in the right places. This issue goes through all aspects of the platform.
- [x] ensure timezone when adding dates to the metainfo. Either use information in the given date or use a "configured" time zone?
- [x] add the local browser timezone to datestrings in the GUI. If those are forwarded to the metainfo they are kept.
- [x] when displaying datetimes in the gui, make sure that the datetime is converted to local browser timezone, based on the timezone information in the datetime
- [ ] ensure timezone information in dates in mongo and elastic searchhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/458UI/UX design ideas2023-12-21T15:55:40ZMarkus ScheidgenUI/UX design ideasThis issue is more a meta issue. Use it to discuss the over all NOMAD UI/UX design. Also use it to discuss the short-term, mid-term, and long-term steps towards a better NOMAD and new FAIRmat.
@lcalvo @himanel1 @avaid @ghuhs @ikerh @dclauThis issue is more a meta issue. Use it to discuss the over all NOMAD UI/UX design. Also use it to discuss the short-term, mid-term, and long-term steps towards a better NOMAD and new FAIRmat.
@lcalvo @himanel1 @avaid @ghuhs @ikerh @dclauhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/300Mypy and pylint plugins for the metainfo2023-12-21T15:40:02ZMarkus ScheidgenMypy and pylint plugins for the metainfoThe metainfo adds some dynamic properties. It would be great to have hints for mypy and pylint to correctly check those statically. I started on a pylint plugin, before I realized that we would also need a mypy-plugin. For now, the MSect...The metainfo adds some dynamic properties. It would be great to have hints for mypy and pylint to correctly check those statically. I started on a pylint plugin, before I realized that we would also need a mypy-plugin. For now, the MSection sports an unnecessary __getattr__ to make mypy and pylint not check un "unexsisting" members.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/509Collisions in metainfo2023-12-21T15:40:01ZLauri HimanenCollisions in metainfoThe metainfo sections follow the `collections.abs.Mapping` interface and have functions called `values`, `keys`, `items`, `get`, etc.
What would be the preferred way of dealing with a quantitity or subsection that collides with these fu...The metainfo sections follow the `collections.abs.Mapping` interface and have functions called `values`, `keys`, `items`, `get`, etc.
What would be the preferred way of dealing with a quantitity or subsection that collides with these function names? In my case, I would prefer to assign something to `values`, but can't. I guess it is not an option to use some other name for these functions (e.g. m_values), since that will break the interface. Should I just use another name in this case?