nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2024-03-25T11:57:34Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1953Topology visualization improvement2024-03-25T11:57:34ZLauri HimanenTopology visualization improvementDue to limitations built into the GUI code, certain topologies cannot be shown. The following limitations should be lifted:
- [ ] If a system in topology does not contain atoms, we simply don't display the ngl viewer. This allows logica...Due to limitations built into the GUI code, certain topologies cannot be shown. The following limitations should be lifted:
- [ ] If a system in topology does not contain atoms, we simply don't display the ngl viewer. This allows logical groups that don't have their own visualization.
- [ ] The topology should not be required to have a single root node.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1937add reference to elabftw parser2024-03-15T00:49:10ZAmir Golparvaradd reference to elabftw parserAmir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1886Packages aliases in metainfo2024-03-01T11:03:51ZMarkus ScheidgenPackages aliases in metainfoWe are currently moving a lot of metainfo packages. It is necessary to have some infrastructure that allows to gracefully deprecate old package names. The goal is to not have to reprocess or otherwise migrate data instantly.
"Package al...We are currently moving a lot of metainfo packages. It is necessary to have some infrastructure that allows to gracefully deprecate old package names. The goal is to not have to reprocess or otherwise migrate data instantly.
"Package aliases" would allow to add old package names to a package definition:
- `m_from_dict` will consider package aliases when looking for definition referenced in `m_def`
- the gui will consider package aliases when looking for definitions referenced in `m_def`
- searches targeting `section_defs` will expand into an "or" on all package aliases
- facilities will be made available to clients to easily work with package names and consider package aliases in the process
Property aliases (espsecially for sub sections) is a related functionality, not covered in the issue.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1866Config option to disable optimade API.2024-02-26T11:46:23ZMarkus ScheidgenConfig option to disable optimade API.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1849Converter for json to and from pd dataframe2024-02-15T09:12:26ZAmir GolparvarConverter for json to and from pd dataframeflatten json into a pandas dataframe
reconstruct dataframe to json
options: depth of the json to flatten
options: order resultant dataframe alphabetically or by a list of stringsflatten json into a pandas dataframe
reconstruct dataframe to json
options: depth of the json to flatten
options: order resultant dataframe alphabetically or by a list of stringsAmir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1845Attribute inheritance2024-02-29T11:46:44ZMarkus ScheidgenAttribute inheritanceThe current metainfo does not support attribute inheritance. Attributes are primarily used in nexus. Here, the lack of attribute inheritance causes an excessive duplication of attribute definitions.
- [x] add inheritance to the metainf...The current metainfo does not support attribute inheritance. Attributes are primarily used in nexus. Here, the lack of attribute inheritance causes an excessive duplication of attribute definitions.
- [x] add inheritance to the metainfo python implementation
- [x] show attribute inheritance in the metainfo browser
- [x] adapt the generated nexus metainfoMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1812Scaffolding for the plugin example projects.2024-03-13T10:24:56ZMarkus ScheidgenScaffolding for the plugin example projects.We could add more features to the example plugin projects:
- [x] code style checks
- [ ] autoformatting checks and autoformatting pre-commit hooks (both opt-in)
- [x] linting
- [x] an empty mkdocs + github actions to deploy as gh-pages
...We could add more features to the example plugin projects:
- [x] code style checks
- [ ] autoformatting checks and autoformatting pre-commit hooks (both opt-in)
- [x] linting
- [x] an empty mkdocs + github actions to deploy as gh-pages
- [x] test coverage badge, pipeline badge
The mkdocs should include the theming/styling from the main project. We still need to figure out what should in the docs by default.
Some other ideas:
- maybe gh actions to build pypi package (is there something like a package/release registry in github?)
- maybe gh actions to build the docker file (gh does not have a docker resgistry? this would only make sense as opt-in where users define a registry to use)
The projects are :
https://github.com/nomad-coe/nomad-schema-plugin-example
https://github.com/nomad-coe/nomad-parser-plugin-example
@ladinesa is about to create a third one for normalizers
Just start with one project. Create PR there on github.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1810Set co-author metadata for multiple uploads in one action2023-12-21T15:55:50ZAndrea AlbinoSet co-author metadata for multiple uploads in one actionI'm coming up with this question because some user would like to be able to share (or unshare) multiple uploads with some co-author or reviewer, when the uploads are more than 20 it can be tedious to go inside each of them to modify the ...I'm coming up with this question because some user would like to be able to share (or unshare) multiple uploads with some co-author or reviewer, when the uploads are more than 20 it can be tedious to go inside each of them to modify the metadata.
Would we face particular drawbacks implementing this action on a list of ticked uploads in the dashboard?
![Screenshot_from_2023-12-05_14-25-00](/uploads/b8efdf521bcbe555121a76b6b015ab66/Screenshot_from_2023-12-05_14-25-00.png)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1805Separating the plugin's options from the nomad.yaml file2024-03-20T10:08:09ZAdam FeketeSeparating the plugin's options from the nomad.yaml fileIt would be nice to separate ui/plugin options in the plugin and nomad.yaml. It would be nice to define `ui > apps > options ...` in nomad_plugin.yaml.
- options' attribute of ui/plugin could be part of nomad_plugin.yaml file
- it would ...It would be nice to separate ui/plugin options in the plugin and nomad.yaml. It would be nice to define `ui > apps > options ...` in nomad_plugin.yaml.
- options' attribute of ui/plugin could be part of nomad_plugin.yaml file
- it would behave as defaults so you can still override with nomad.yaml
- nomad.yaml would mostly control exclude/include attribute
Use cases:
- @sbrueck 's use case: installing multiple plugins using in a nomad oasis installation. some of the plugins depends on python packages not available in the main image so a custom image needs to built which would contain all plugins and them configuration.
- @afekete 's use case: the support for jupyter notebooks will be implemented as an external parser but should be still part of the core nomad package. It would be nice to use the plugin's yaml file to store its configs instead of putting them as default values of the main configuration.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1804Support for differeent wrap modes in the systems API2023-12-04T08:10:24ZLauri HimanenSupport for differeent wrap modes in the systems APIWe should provide support for handling different wrapping modes directly in the API.
Tasks:
- [ ] Modify API request model to include new `wrap_mode` parameter with fixed options: `wrap`, `unwrap`, or 'raw'.
- [ ] Add functions for `wra...We should provide support for handling different wrapping modes directly in the API.
Tasks:
- [ ] Modify API request model to include new `wrap_mode` parameter with fixed options: `wrap`, `unwrap`, or 'raw'.
- [ ] Add functions for `wrap` and `unwrap` directly in `atomutils`. These should work on `positions`, `cell` and `pbc`.
- [ ] Create tests
- [ ] Add GUI option in the system download menu. By default the mode should be 'raw', but can be selected from a dropdown. Needs good descriptions.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1790Improved searchbar2024-01-15T09:07:50ZLauri HimanenImproved searchbarAs brought up by Oasis users (e.g. @alfu) in the project meeting, we should add support for a fully free-text search. This means that you can type in any text in the search bar, and get the best matches without having to have exact match...As brought up by Oasis users (e.g. @alfu) in the project meeting, we should add support for a fully free-text search. This means that you can type in any text in the search bar, and get the best matches without having to have exact matches or having to target a specific field in the metainfo.
My suggestion would be to have a generic text mapping in our metadata section that would be reserved for this purpose. This would generalize across all entries, and filling it could be even done through the ES annotation, with some intelligent default (all string fields in a schema...?). This field could also be used to test some better suggestion mechanisms that could work with filters.
This addition has a big impact on the search bar logic: we need to fully refactor how suggestions are prioritized and shown, and make sure that the UX does not suffer. Also, we need a mechanism for deciding which types of queries are available in the apps: free-text search should be initially disabled in the most basic apps as it will create confusion due to not returning any results when dealing with computational and not-reprocessed data.
Tasks:
- [x] Add new metainfo that stores the free-text data(e.g. `metadata.keywords`)
- [x] Setup the ES config with an annotation. Should be annotated with `text`, queried with `match`. Default analyzer should be used for now.
- [x] Add logic for populating the free-text field: all string quantities in custom schemas should be considered. Every _unique_ string (whitespace in beginning and end removed) field should be added as a new entry into the free-text search. By using unique values we can avoid populating terms that are used multiple times e.g. in repeating sections.
- [x] The apps should be able to define whether they want to enable free-text queries. It is impossible to automate this decision reliably, because an app may work with any kind of data which may or may not have the `keywords` field. This functionality can be added in a generic way, which can later be used to control/add new kinds of query syntaxes for the search bar.
- [x] Sorting relevance logic should be changed:
- [x] By default sorting happens by what is set in the app
- [x] When a free-text query is executed, sorting should be changed to relevance automatically
- [x] If a user explicitly sets sorting, this is used until a new free-text query is created
- [x] Improve search bar UX:
- [x] The suggestions type should be shown, and the priority of the suggestions should be logical
- [x] Executed queries should be stored into `localStorage` so that we can show suggestions from history
- [x] Items in search history can be deleted
- [x] Autoselect text when navigating suggestions (helps with e.g. modifying a query in history)
- [x] Add a better explanation of the available search queries
- [x] Add tests that check that correct string quantities are added to the new quantity
- [x] Add search tests that check that the API does the correct queries on this field
- [x] Extend search bar tests
In the future, this feature could also be extended using some LLM model: the `metadata.keywords` field would be translated into a latent space vector of an LLM and queries would be performed against it.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1768OpenBIS Integration2023-12-07T09:17:35ZMarkus ScheidgenOpenBIS IntegrationThere are potential use cases with FAIRmat partners.
- [x] What apis are available
- [x] Can the zenode export be used
- [x] Basic OpenBIS schema
- [x] Mapping custom OpenBIS entities to NOMAD schemaThere are potential use cases with FAIRmat partners.
- [x] What apis are available
- [x] Can the zenode export be used
- [x] Basic OpenBIS schema
- [x] Mapping custom OpenBIS entities to NOMAD schemaAmir GolparvarAmir Golparvarhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1760Groups and upload visibility2024-01-29T13:47:24ZMarkus ScheidgenGroups and upload visibilityThis replaces #1691, #1692
Currently uploads have a `main_author: User` (owner), `coauthors: User[]`, and `reviewers: User[]`. At least the `coauthors` and `reviewers` roles should also be assignable via user groups.
### First steps
...This replaces #1691, #1692
Currently uploads have a `main_author: User` (owner), `coauthors: User[]`, and `reviewers: User[]`. At least the `coauthors` and `reviewers` roles should also be assignable via user groups.
### First steps
- [x] add `reviewer_groups: str` and `coauthor_groups: str` to the upload mongo model (`nomad/processing/data.py::Upload`)
- [x] add a mongo collection `groups` with keys `owner: User`, `group_name`, `group_id`, `members: User[]`
- [x] add `reviewer_groups: str` and `coauthor_groups: str` to the EntryMetadata (`nomad/datamodel/datamodel.py::EntryMetadata`)
- [x] populate the new EntryMetadata fields (somewhere in `nomad/processing/data.py`)
- [x] extend queries to mongo, e.g. in `nomad/app/v1/routers/uploads.py`
- [x] add tests (`tests/app/v1/routers/test_uploads.py`, `tests/app/v1/routers/test_groups.py`)
- [x] extend queries to elastic search in `nomad/search.py::_owner_es_query`
- [x] add some basic `groups` CRUD API
### Manage visibility with groups
- introduce a special group_id 'all' to make uploads visible or editable to everyone
- this could replace visibility rules constructed around `published` and reduced the published state to be only concerned with immutability
### Further steps
- a UI to create and manage groups
- add groups to the UI for managing coauthors and reviewers
### Potentially managing groups with keycloak:
- Keycloak groups are global groups with no owner or any other scope but the realm. It seams unlikely that this can be used, especially not for the central keycloak. Conclusively, we cannot use any group management UI and have to build our own.
- Keycloak users can have attributes. Group membership could be persisted as a `groups` attribute for users. However, additional group metadata, like name, owner, etc. still needs to be stored somewhere. We also still need to write our own management UI. It will also be hard to distinguish between different groups on different oases.
Conclusively, keycloak does not seem to be helpful for group management.
### Questions
- Should coauthor/reviewer (groups) be split into access right concepts and authorshop/reference concepts?
- Should the coauthor/reviewer groups of an upload be converted into immutable coauthors/reviewers when an upload is published?
- Should the coauthors list (returned) also includes users from the coauthor groups or be kept separate?Sascha KlawohnSascha Klawohnhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1758Unit selection component2023-12-01T06:01:19ZLauri HimanenUnit selection componentFor #1713, #1666, and #960 we need an implementation of a good unit selection component. We already have something implemented for the ELNs, just needs to be refactored so it can be used elsewhere in the GUI.For #1713, #1666, and #960 we need an implementation of a good unit selection component. We already have something implemented for the ELNs, just needs to be refactored so it can be used elsewhere in the GUI.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1740Search interface improvements2023-11-01T06:55:19ZLauri HimanenSearch interface improvements- Change label for Structure to be Structure / Symmetry
- Add a default dashboard for the "Calculations" app:
- Periodic table
- Structure type
- Crystal system
- Space group name- Change label for Structure to be Structure / Symmetry
- Add a default dashboard for the "Calculations" app:
- Periodic table
- Structure type
- Crystal system
- Space group nameLauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1738JWT ImmatureSignatureError2023-10-18T11:36:58ZLauri HimanenJWT ImmatureSignatureErrorOne of our Oasis users has reported an [issue with JWT validation](https://github.com/nomad-coe/nomad/issues/88), which might be caused by a small difference in the timestamp of two physical machines.
In order to fix this, it might be r...One of our Oasis users has reported an [issue with JWT validation](https://github.com/nomad-coe/nomad/issues/88), which might be caused by a small difference in the timestamp of two physical machines.
In order to fix this, it might be reasonable to introduce a 'leeway' config in nomad.yaml that specified how much wiggling room there is when checking the ia``t, `nbf`, `exp` fields.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1732Scatter plots with multiple values per entry2024-02-27T15:46:38ZLauri HimanenScatter plots with multiple values per entryThe scatter plot widget is too limited at the moment: it can only be used to display scalar values from non-repeated structures.
To improve this, the following should be done:
- [x] Extend the widget pydantic model to include
- [x] I...The scatter plot widget is too limited at the moment: it can only be used to display scalar values from non-repeated structures.
To improve this, the following should be done:
- [x] Extend the widget pydantic model to include
- [x] Instead of providing a string for `x`, `y`, and `color`, we should use more complex configuration objects that can include units, labels etc.
- [x] Document the possibility of using JMESPath syntax on `quantity`
- [x] Add `root_validator` that ensures backward compatibility with old models.
- [x] Create a new function utility function `parseJMESPath` for parsing JMESPath syntax
- [x] Create tests for `parseJMESPath`:
- [x] Raise JMESPath errors correctly
- [x] Handle functions
- [x] Handle selectors
- [x] Handle quantity names with schema + dtype
- [x] Create new input component `InputJMESPath` for JMESPath queries
- [x] Create tests for `InputJMESPath`:
- [x] Raise JMESPath errors correctly
- [x] Only allow values from current `SearchContext`
- [x] Create tests for `WidgetScatterPlot`:
- [x] Test that using quantity names that target scalar values works as normal
- [x] Test that custom axis labels are shown correctly
- [x] Test that custom units are handled correctly.
- [x] Test that using (subset) of JMESPath syntax works:
- [x] selection: `results.material.topology[0].cell.a`
- [x] functions: `max(results.material.topology[*].cell.a)`
- [x] Create tests for `WidgetScatterPlotEdit`:
- [x] Test that using invalid JMESPath syntax gives a warning
- [x] Test that targeting unavailable quantities gives warning
- [x] Test that invalid units gives warning
- [x] Test that incompatible units gives warning
- [x] Test that missing required values units gives warning
- [x] Test that targeting repeatable data with no JMESPath syntax gives a warning.Lauri HimanenLauri Himanenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1708Add upload_name to search index for references2023-10-03T05:34:16ZMichael GötteAdd upload_name to search index for referencesHey @himanel1 @mscheidg
I noticed, that you can not filter by upload_name although it is in the metadata?
![image](/uploads/490abbcbe85360ff952dec1cfebf819e/image.png)
You can filter by entry_name.
Is there a reason for this?
Best...Hey @himanel1 @mscheidg
I noticed, that you can not filter by upload_name although it is in the metadata?
![image](/uploads/490abbcbe85360ff952dec1cfebf819e/image.png)
You can filter by entry_name.
Is there a reason for this?
Best Michahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1692Group-Based Access Control for Uploads2023-10-30T14:40:46ZAdam FeketeGroup-Based Access Control for UploadsIt would be beneficial to implement a feature allowing users to define groups, where members of these groups can either view or edit specific uploads. This would provide a more flexible way for users to share and manage access to their u...It would be beneficial to implement a feature allowing users to define groups, where members of these groups can either view or edit specific uploads. This would provide a more flexible way for users to share and manage access to their uploads.
## Possible Implementation:
1. **Group Management in Keyclock**
- While this is possible, this will mean that the group management will reside outside NOMAD.
2. **Group Management within NOMAD:**
- Instead of using external systems like Keycloak for group management, integrate group creation and modification directly into the NOMAD application.
- Each group should have designated roles like 'group_admin' and 'members'.
- **Admins**: Can create, edit, and delete the group, add/remove members.
- **Members**: Have read/write permissions based on the group settings.
- Similar to platforms like GitLab, members should be able to leave a group on their own.
2. **Upload Visibility and Access:**
- By default, an upload should only be visible to the uploader at creation time.
- While uploading or editing an upload, there should be a dropdown menu allowing users to select which groups or individuals should have access to the specific upload. An upload can be shared with multiple groups.
- On the main upload page, there should be a clear indication that the upload is visible to a user/group (i.e., through which group membership). If an upload is visible due to multiple group memberships, all relevant groups should be displayed.
3. **Ideas for the User Interface:**
- On the top right of the Nomad interface, incorporate a button for users to manage their group memberships.
- The design and user experience should be intuitive and align with familiar platforms like GitLab, ensuring ease of use and understanding.
## Use-Case:
- If multiple groups or projects are using the same Nomad Oasis deployment, each could have their dedicated shared uploads.
- Before making an AI toolkit publicly available, it could first be shared with a select group of contributors.
## Additional Notes:
- It's mentioned that there's another feature request regarding upload visibility ([Issue #1691](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1691)). It might be advantageous to address these features together, ensuring consistent and coherent implementation.
---
Note: This feature suggestion was inspired by input from @josma and further enhanced by suggestions from @g-michaelgoette based on his use cases.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1691Defining visibility for upload2023-10-30T14:40:39ZAdam FeketeDefining visibility for uploadThere are certain cases when you want to share an upload with everybody without making it published, or inmutable.
## Possible solutions:
- Probably the simplest solution would be to add an option to make an upload visible to everybody...There are certain cases when you want to share an upload with everybody without making it published, or inmutable.
## Possible solutions:
- Probably the simplest solution would be to add an option to make an upload visible to everybody.
- Slightly more complex: A group of users (like a specific group or everybody) can access an upload with some permissions (like read-only or edit). Described here: #1692
## Use cases:
- There is a WIP implementation to integrate AI toolkits into nomad interface:
https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/merge_requests/1357
- An inventory type upload in the experimental lab for chemicals or instruments.
- A curation process of a database.
In the case of AI toolkits we need a way to share the notebooks with everybody but still have the option to edit the metadata of the entries. This is needed because we might need to add a new author who did some improvement or we need to update the packages because it is not compatible with the new environment... Unfortunately, it cannot just be published every time when there are some small changes. From the FAIR point of view, every AI toolkit has its own git repo where we can keep track of all of the changes.
If any of you (eg. @josma) has another use-case where this feature would be useful please to add it to here.