nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2023-10-30T14:40:46Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1692Group-Based Access Control for Uploads2023-10-30T14:40:46ZAdam FeketeGroup-Based Access Control for UploadsIt would be beneficial to implement a feature allowing users to define groups, where members of these groups can either view or edit specific uploads. This would provide a more flexible way for users to share and manage access to their u...It would be beneficial to implement a feature allowing users to define groups, where members of these groups can either view or edit specific uploads. This would provide a more flexible way for users to share and manage access to their uploads.
## Possible Implementation:
1. **Group Management in Keyclock**
- While this is possible, this will mean that the group management will reside outside NOMAD.
2. **Group Management within NOMAD:**
- Instead of using external systems like Keycloak for group management, integrate group creation and modification directly into the NOMAD application.
- Each group should have designated roles like 'group_admin' and 'members'.
- **Admins**: Can create, edit, and delete the group, add/remove members.
- **Members**: Have read/write permissions based on the group settings.
- Similar to platforms like GitLab, members should be able to leave a group on their own.
2. **Upload Visibility and Access:**
- By default, an upload should only be visible to the uploader at creation time.
- While uploading or editing an upload, there should be a dropdown menu allowing users to select which groups or individuals should have access to the specific upload. An upload can be shared with multiple groups.
- On the main upload page, there should be a clear indication that the upload is visible to a user/group (i.e., through which group membership). If an upload is visible due to multiple group memberships, all relevant groups should be displayed.
3. **Ideas for the User Interface:**
- On the top right of the Nomad interface, incorporate a button for users to manage their group memberships.
- The design and user experience should be intuitive and align with familiar platforms like GitLab, ensuring ease of use and understanding.
## Use-Case:
- If multiple groups or projects are using the same Nomad Oasis deployment, each could have their dedicated shared uploads.
- Before making an AI toolkit publicly available, it could first be shared with a select group of contributors.
## Additional Notes:
- It's mentioned that there's another feature request regarding upload visibility ([Issue #1691](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1691)). It might be advantageous to address these features together, ensuring consistent and coherent implementation.
---
Note: This feature suggestion was inspired by input from @josma and further enhanced by suggestions from @g-michaelgoette based on his use cases.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1790Improved searchbar2024-01-15T09:07:50ZLauri HimanenImproved searchbarAs brought up by Oasis users (e.g. @alfu) in the project meeting, we should add support for a fully free-text search. This means that you can type in any text in the search bar, and get the best matches without having to have exact match...As brought up by Oasis users (e.g. @alfu) in the project meeting, we should add support for a fully free-text search. This means that you can type in any text in the search bar, and get the best matches without having to have exact matches or having to target a specific field in the metainfo.
My suggestion would be to have a generic text mapping in our metadata section that would be reserved for this purpose. This would generalize across all entries, and filling it could be even done through the ES annotation, with some intelligent default (all string fields in a schema...?). This field could also be used to test some better suggestion mechanisms that could work with filters.
This addition has a big impact on the search bar logic: we need to fully refactor how suggestions are prioritized and shown, and make sure that the UX does not suffer. Also, we need a mechanism for deciding which types of queries are available in the apps: free-text search should be initially disabled in the most basic apps as it will create confusion due to not returning any results when dealing with computational and not-reprocessed data.
Tasks:
- [x] Add new metainfo that stores the free-text data(e.g. `metadata.keywords`)
- [x] Setup the ES config with an annotation. Should be annotated with `text`, queried with `match`. Default analyzer should be used for now.
- [x] Add logic for populating the free-text field: all string quantities in custom schemas should be considered. Every _unique_ string (whitespace in beginning and end removed) field should be added as a new entry into the free-text search. By using unique values we can avoid populating terms that are used multiple times e.g. in repeating sections.
- [x] The apps should be able to define whether they want to enable free-text queries. It is impossible to automate this decision reliably, because an app may work with any kind of data which may or may not have the `keywords` field. This functionality can be added in a generic way, which can later be used to control/add new kinds of query syntaxes for the search bar.
- [x] Sorting relevance logic should be changed:
- [x] By default sorting happens by what is set in the app
- [x] When a free-text query is executed, sorting should be changed to relevance automatically
- [x] If a user explicitly sets sorting, this is used until a new free-text query is created
- [x] Improve search bar UX:
- [x] The suggestions type should be shown, and the priority of the suggestions should be logical
- [x] Executed queries should be stored into `localStorage` so that we can show suggestions from history
- [x] Items in search history can be deleted
- [x] Autoselect text when navigating suggestions (helps with e.g. modifying a query in history)
- [x] Add a better explanation of the available search queries
- [x] Add tests that check that correct string quantities are added to the new quantity
- [x] Add search tests that check that the API does the correct queries on this field
- [x] Extend search bar tests
In the future, this feature could also be extended using some LLM model: the `metadata.keywords` field would be translated into a latent space vector of an LLM and queries would be performed against it.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1691Defining visibility for upload2023-10-30T14:40:39ZAdam FeketeDefining visibility for uploadThere are certain cases when you want to share an upload with everybody without making it published, or inmutable.
## Possible solutions:
- Probably the simplest solution would be to add an option to make an upload visible to everybody...There are certain cases when you want to share an upload with everybody without making it published, or inmutable.
## Possible solutions:
- Probably the simplest solution would be to add an option to make an upload visible to everybody.
- Slightly more complex: A group of users (like a specific group or everybody) can access an upload with some permissions (like read-only or edit). Described here: #1692
## Use cases:
- There is a WIP implementation to integrate AI toolkits into nomad interface:
https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1357
- An inventory type upload in the experimental lab for chemicals or instruments.
- A curation process of a database.
In the case of AI toolkits we need a way to share the notebooks with everybody but still have the option to edit the metadata of the entries. This is needed because we might need to add a new author who did some improvement or we need to update the packages because it is not compatible with the new environment... Unfortunately, it cannot just be published every time when there are some small changes. From the FAIR point of view, every AI toolkit has its own git repo where we can keep track of all of the changes.
If any of you (eg. @josma) has another use-case where this feature would be useful please to add it to here.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1554Plot support refactor2023-10-17T14:37:33ZLauri HimanenPlot support refactorOur current plotting functionality is quite limited. In customs schemas one can define `PlotAnnotation`s for sections or quantities, but they don't offer enough functionality in many cases.
# Problems
- Only static configuration is all...Our current plotting functionality is quite limited. In customs schemas one can define `PlotAnnotation`s for sections or quantities, but they don't offer enough functionality in many cases.
# Problems
- Only static configuration is allowed. I guess the origin of this is the use in yaml schemas which can't really contain code. What if the layout or plotted traces should change based on data that is being plotted? What if we need to perform some postprocessing to the data before plotting? There was already a request coming from @jschumann about being able to determine the visualization of different lines when it is not known how many lines will be plotted.
- The attributes in our `PlotAnnotation` is not completely aligned with the way Plotly plots are created.
- Plot needs to be associated with a section or quantity. What if I want to define it in a completely custom location?
# Possible solution
Plotly plots can be configured with JSON: Each plot takes in `data`, `layout` and `config`, more info [here](https://plotly.com/javascript/plotlyjs-function-reference/#plotlynewplot). If we are able to produce a single JSON dictionary:
```json
{
data: {...},
layout: {...},
config: {...}
}
```
This could be loaded from our API and visualized directly. It would only be a matter of how this JSON data is created and rendered:
- It could similar to a `derived` value: The data would be created on the fly when requesting it from the API.
- It could be a quantity with JSON dtype that is filled by a normalize-function of a custom schema. This would mean it is persisted always and there is data duplication. There could be a special annotation that tells that Plotly should be used to view this JSON file within an ELN.
- Maybe we could create a special type of `Quantity` for plots? This might be easier for users to grasp. This special quantity could then maybe be configured to either store the plot, or always recreate it on the fly.
This solution would only be available in python schemas, since it requires specifying python code. But I think this should be an acceptable limitation.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1493Small ELN improvements2023-12-21T15:55:19ZMarkus ScheidgenSmall ELN improvements### Bugs that need to be fixed
- [x] When a numeric field filled and the value is removed afterwards, an error occurs (#1362). There seems to be different problems with both int and floats. The error is happening in the processing, not t...### Bugs that need to be fixed
- [x] When a numeric field filled and the value is removed afterwards, an error occurs (#1362). There seems to be different problems with both int and floats. The error is happening in the processing, not the UI.
- [unable to reproduce] You cannot remove the value from a ReferenceEditQuantity field.
- [x] Only the `main_author` should be allowed to remove an upload. Currently all co-authors can. This needs to be fixed in the API and in the UI.
- [x] When the `Datatable` component shows table rows that are wider than the table, you have to scroll to reach the table actions (e.g. the "->" button). We need to make sure that the table actions are always visible.
### Simple things and low hanging fruits
- [x] Duplicate functionality for entries. Entries can already be duplicated in the file browser. For convinience, we need to add a "duplicate" button (there is a well-known icon in the MUI icon set). This button should be added to the entry overview page and as an additional table action to the entry table on the upload overview page. (#1099)
- [x] For convenience we should add a "search" button to the upload page. Clicking the button should open a menu similar to the "explore" menu. Clicking on a menu item should navigate to the respective search app and preselect the respective upload id (#1100, #1143)
- [x] Currently, plots generated by the plot annotation use the units from the global units system. If there is a display unit on the respective quantity, this should be used instead. (#1059)
- [x] FileEditQuantity on a list-shaped quantity does not allow to navigate to the file preview. This only works for scalar-shaped file quantities (#1033)
- [x] When uploading a file with the FileEditQuantity, no upload progress is shown. We need to show the same progress model (and disable the ui) as we used for uploading files to an upload
- [x] It should be possible to upload multiple files to a list-shaped FileEditQuantity.
- [x] The checkbox options in the ReferenceEditQuantity dialog should all be unchecked by default. We need other options: "From this upload", "From my uploads", "From my (co)-authored uploads", "From published uploads"
- [x] The RichTextEditor annotation needs an option to define the default hight of the text editor
- [x] ReferenceEditQuantities that target an entry with processing error, do not allow to navigate to this entry (even if an archive exists).
- [x] We need to visually tell that an entry has a (processing) error. It should be visible on all entry pages, in the upload page entry list, in the file browser, when opening a reference to an entry with error.
- [x] It is really hard to see list-shape edit quantities. They should be foldable and unfolded edit field should be indented. Folding like with repeating sub-sections.
- [x] Clicking the (+) button on folded repeated sub-sections, should unfold the sub-section and highlight the new sub-section.
- [x] Indicate that there are co-authors in the uploads list (and on the upload page). The "publish" icon could show this.
- [x] We might also offer co-authors and reviewers as table columns (similar to authors on entry tables).
- [x] If upload and entry names are available, use them in the breadcrumbs. Maybe fall back to file name for entries. (#1558)
### Maybe not too hard
- [ ] A button EditQuantity for boolean quantities that is automatically set to False after processing. (#1242)
- [ ] Show (incoming) references as a table that shows, allows to sort, and/or filter based on the reference quantity definition
- [ ] Allow to parameterize error and warning logs with a path to the quantity or section that causes the error. The logs can be taken from the processing logs to show the error on the quantity or section.
- [ ] API+UI to create+delete groups. Co-authors and reviewers can be added or removed from groups. Groups can be added/removed to uploads. Needs more discussion. (#1016)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1435Skip parser matching when parser is already specified2023-08-22T13:08:54ZNathan DaelmanSkip parser matching when parser is already specifiedFrom a discussion with @ladinesa we came with the following proposal:
atm, running `nomad parse --parser <parser name> --show-archive` will still perform the parser matching.
We know this, since some auxiliary files (e.g. `OUTCAR`) are n...From a discussion with @ladinesa we came with the following proposal:
atm, running `nomad parse --parser <parser name> --show-archive` will still perform the parser matching.
We know this, since some auxiliary files (e.g. `OUTCAR`) are not being picked up on in the presence of the main file (e.g. `vasprun.xml`).
`nomad` then just returns `assert parser is not None, 'there is no parser matching %s' % mainfile`.
I'd argue that this behavior is counterintuitive in light of the `--parser <parser name>` option.
If we change this, I'd also request to add further specification in `nomad parse --help` on how to formulate `<parser name>`.
Trial-and-error doesn't work in guessing the naming format, and I eventually only figured it out thx to @ladinesa .
If the naming can be customized via a settings file, just pointing to its location is also fine.
@mscheidg I'd really like your feedback, given that this touches on the command line behavior.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1384The "delete mainfile and folder" option does more harm than good2023-03-31T10:02:53ZMarkus ScheidgenThe "delete mainfile and folder" option does more harm than goodThe option is kinda necessary for mass deleting theory entries. However, the option could be expressed and placed differently to make it less likely to use it by accident.The option is kinda necessary for mass deleting theory entries. However, the option could be expressed and placed differently to make it less likely to use it by accident.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1383Warning when deleting uploads or entries that are referenced2023-05-11T13:06:17ZMarkus ScheidgenWarning when deleting uploads or entries that are referencedWith all references (and schema uses) stored in the searchable metadata, it should be possible to implement a check and warning when deleting uploads and entries.
This might be a bit harder when a folder is deleted. But, using the text-...With all references (and schema uses) stored in the searchable metadata, it should be possible to implement a check and warning when deleting uploads and entries.
This might be a bit harder when a folder is deleted. But, using the text-based path search on `mainfile` even this should be feasible.
If we do a search on delete anyways, the delete confirm dialog could also provide information on how much files/data is about to be deleted.Mohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1355Visually consolidate related entries and respective files2023-12-21T16:05:38ZMarkus ScheidgenVisually consolidate related entries and respective filesThere are many cases, where multiple mainfiles (and aux-files) and respectively multiple entries form a logical whole. A few examples:
- parsed files (e.g. nexus) that are augmented with user-entered information from a "sister" ELN entry...There are many cases, where multiple mainfiles (and aux-files) and respectively multiple entries form a logical whole. A few examples:
- parsed files (e.g. nexus) that are augmented with user-entered information from a "sister" ELN entry
- workflow entries and the tasks, input, outputs, etc. they reference
- entries that are coming from an .xls (or other tabular file)
It is important to show this connected (or even hide the parts) from the user, because in their minds these things belong to each other and form a semantic unit.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1305Customization bugs in the gui2023-04-21T11:40:10ZMichael GötteCustomization bugs in the guiDear Nomad,
I had over the time some issues with the customization of the gui. I want to collect them all in one issue because i feel that there is not much attention towards them.
- Hiding quantities only works in leaf (Entrydata) cla...Dear Nomad,
I had over the time some issues with the customization of the gui. I want to collect them all in one issue because i feel that there is not much attention towards them.
- Hiding quantities only works in leaf (Entrydata) classes.
- Hiding of subsections does not work (see #1109).
- Reordering the order of quantities is only possible by repeating all parent quantities in the child classes.
- Overwriting the label of a parent class quantity in a child class does not work.
These are all small things, but as a whole they highly limit the customization for specific labs. The workaorund would be not to use the parentbase classes and restate the classes with the right labels, order and quantities and subsections, but this is way harder to maintain.
Best MichaMohammad NakhaeeMohammad Nakhaeehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1099Duplicate Archive in upload2023-12-21T16:04:41ZMichael GötteDuplicate Archive in uploadHey nomad,
i have not found this feature request yet. It is not a key feature but could be a quality of life feature for some scientists.
The appility to duplicate an Archive can save much work, if for example only some fields need to...Hey nomad,
i have not found this feature request yet. It is not a key feature but could be a quality of life feature for some scientists.
The appility to duplicate an Archive can save much work, if for example only some fields need to be changed. I could imagine that you tick an archive in the upload overview and besides deletion there is a duplicate button.
Best Michahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1074System links and ASE atoms for results section2023-12-21T16:06:49ZMarkus ScheidgenSystem links and ASE atoms for results sectionThe results section's material does not reference the structure section it is based on.
The result's sections structure property is not referencing the section system/atoms it is based on.
Both the results structure sections and the sys...The results section's material does not reference the structure section it is based on.
The result's sections structure property is not referencing the section system/atoms it is based on.
Both the results structure sections and the system/atoms section could benefit from a function that is producing a respective ase.Atoms object.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/947Support for displaying values as percent '%'2023-12-21T15:55:18ZJose Marquez PrietoSupport for displaying values as percent '%'From several sources, we have heard multiple times the need to support in some kind of way to display values in percentage. One possibility could be to introduce it as a unit of ratio. @himanel1 what is your opinion on this?
@aalbino, @...From several sources, we have heard multiple times the need to support in some kind of way to display values in percentage. One possibility could be to introduce it as a unit of ratio. @himanel1 what is your opinion on this?
@aalbino, @dflor Any thoughts about this?Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1953Topology visualization improvement2024-03-25T11:57:34ZLauri HimanenTopology visualization improvementDue to limitations built into the GUI code, certain topologies cannot be shown. The following limitations should be lifted:
- [ ] If a system in topology does not contain atoms, we simply don't display the ngl viewer. This allows logica...Due to limitations built into the GUI code, certain topologies cannot be shown. The following limitations should be lifted:
- [ ] If a system in topology does not contain atoms, we simply don't display the ngl viewer. This allows logical groups that don't have their own visualization.
- [ ] The topology should not be required to have a single root node.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1937add reference to elabftw parser2024-03-15T00:49:10ZAmir Golparvaradd reference to elabftw parserAmir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1886Packages aliases in metainfo2024-03-01T11:03:51ZMarkus ScheidgenPackages aliases in metainfoWe are currently moving a lot of metainfo packages. It is necessary to have some infrastructure that allows to gracefully deprecate old package names. The goal is to not have to reprocess or otherwise migrate data instantly.
"Package al...We are currently moving a lot of metainfo packages. It is necessary to have some infrastructure that allows to gracefully deprecate old package names. The goal is to not have to reprocess or otherwise migrate data instantly.
"Package aliases" would allow to add old package names to a package definition:
- `m_from_dict` will consider package aliases when looking for definition referenced in `m_def`
- the gui will consider package aliases when looking for definitions referenced in `m_def`
- searches targeting `section_defs` will expand into an "or" on all package aliases
- facilities will be made available to clients to easily work with package names and consider package aliases in the process
Property aliases (espsecially for sub sections) is a related functionality, not covered in the issue.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1866Config option to disable optimade API.2024-02-26T11:46:23ZMarkus ScheidgenConfig option to disable optimade API.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1849Converter for json to and from pd dataframe2024-02-15T09:12:26ZAmir GolparvarConverter for json to and from pd dataframeflatten json into a pandas dataframe
reconstruct dataframe to json
options: depth of the json to flatten
options: order resultant dataframe alphabetically or by a list of stringsflatten json into a pandas dataframe
reconstruct dataframe to json
options: depth of the json to flatten
options: order resultant dataframe alphabetically or by a list of stringsAmir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1845Attribute inheritance2024-02-29T11:46:44ZMarkus ScheidgenAttribute inheritanceThe current metainfo does not support attribute inheritance. Attributes are primarily used in nexus. Here, the lack of attribute inheritance causes an excessive duplication of attribute definitions.
- [x] add inheritance to the metainf...The current metainfo does not support attribute inheritance. Attributes are primarily used in nexus. Here, the lack of attribute inheritance causes an excessive duplication of attribute definitions.
- [x] add inheritance to the metainfo python implementation
- [x] show attribute inheritance in the metainfo browser
- [x] adapt the generated nexus metainfoMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1812Scaffolding for the plugin example projects.2024-03-13T10:24:56ZMarkus ScheidgenScaffolding for the plugin example projects.We could add more features to the example plugin projects:
- [x] code style checks
- [ ] autoformatting checks and autoformatting pre-commit hooks (both opt-in)
- [x] linting
- [x] an empty mkdocs + github actions to deploy as gh-pages
...We could add more features to the example plugin projects:
- [x] code style checks
- [ ] autoformatting checks and autoformatting pre-commit hooks (both opt-in)
- [x] linting
- [x] an empty mkdocs + github actions to deploy as gh-pages
- [x] test coverage badge, pipeline badge
The mkdocs should include the theming/styling from the main project. We still need to figure out what should in the docs by default.
Some other ideas:
- maybe gh actions to build pypi package (is there something like a package/release registry in github?)
- maybe gh actions to build the docker file (gh does not have a docker resgistry? this would only make sense as opt-in where users define a registry to use)
The projects are :
https://github.com/nomad-coe/nomad-schema-plugin-example
https://github.com/nomad-coe/nomad-parser-plugin-example
@ladinesa is about to create a third one for normalizers
Just start with one project. Create PR there on github.Markus ScheidgenMarkus Scheidgen