nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2019-08-09T07:00:34Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/81Reduce the responsibilities of the repository db2019-08-09T07:00:34ZMarkus ScheidgenReduce the responsibilities of the repository dbAfter creating file structures that allow to archive repository metadata #80, we could move some data from the repository db to files and (partially)create the repository search index from there.
This requires to adopt the existing repo...After creating file structures that allow to archive repository metadata #80, we could move some data from the repository db to files and (partially)create the repository search index from there.
This requires to adopt the existing repository search API
- [ ] coe->fairdi migration of repoTool
- [ ] coe->fairdi migration of repoWebservice
Also
- [ ] Implementation of user metadata change notification API endpoint
or better
- [ ] User metadata API endpointsMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/74Configuration file based approach to config2019-03-20T15:40:28ZMarkus ScheidgenConfiguration file based approach to configCurrent we use a python module to provide configuration. The only option to override configuration in different environments is through environment variables. This is cumbersome and limited, since environment variables have no structure ...Current we use a python module to provide configuration. The only option to override configuration in different environments is through environment variables. This is cumbersome and limited, since environment variables have no structure or types. It leads to ugly config scripts (e.g. helm charts).
We should refactor to a solution that:
- [ ] is based on an external configuration file
- [ ] allows partial overrides from other files
- [ ] still allows overrides from environment variablesMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/59Refactor git project and pypi package structure2020-04-14T09:47:42ZMarkus ScheidgenRefactor git project and pypi package structure# goals
- a nomad pypi package with various *setuptools* extensions: nomad, nomad[parser], nomad[client], nomad[infra], nomad[dev]
- fast cli startup
- fast tests
## cleaning up the internal module structure
This is what the nomad modul...# goals
- a nomad pypi package with various *setuptools* extensions: nomad, nomad[parser], nomad[client], nomad[infra], nomad[dev]
- fast cli startup
- fast tests
## cleaning up the internal module structure
This is what the nomad module structure and its inter-dependencies should look like:
![modules_and_dependencies](/uploads/3bd266dad3d0b0cfec59035b7c96e827/modules_and_dependencies.png)
## experimenting with lazy imports
There are various ways in python, especially newer versions 3.7+, to lazy import modules. We should explore solutions for these two problems:
- How to implement a nomad module in a way that it is not directly loaded when imported
- How to lazily import other modules (dependencies)
With both problems solutions should:
- minimize impact on "normal" programming practices
- not break pylint and mypy !!!
- allow to monkeypatch (where necessary)
- allow 'partial' imports (optional)
Furthermore, it would be beneficial to use tools to analyse import and startup times.
## implement the setup.py
If the packages are prepared in the right way, this should be trivial. Instead of having `python-common` and the parsers as their own packages, we should include them in the nomad package.
## a proper CI for tests
This would require modularised tests, an image without prior pip installs, respective installs between tests of various nomad extensions. There is still the need for a complete image to deploy nomad.
## other (old) tasks
- [ ] clean up the python common to the bare minimum
- [x] move parser specific metainfo to parsersAlvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1680Annotations as definition properties2024-01-08T07:13:53ZHampus NaesstroemAnnotations as definition propertiesAnnotations are additional key-value pairs that can be added to any metainfo object. In practice however, there are predominantly used on definitions. Them not being definition properties seems to be confusion and creates more issues tha...Annotations are additional key-value pairs that can be added to any metainfo object. In practice however, there are predominantly used on definitions. Them not being definition properties seems to be confusion and creates more issues than it solves.
Topic was changed from the following, based on the discussion below:
### Adding ELN Annotation in Inheriting Section Fails
Trying to add a ELN annotation to an inheriting section using `m_copy()` fails (the NumberEditQuantity is not showing up for entries of type `B`):
```python
class A(EntryData):
some_property = Quantity(type=float)
class B(A):
some_property = A.some_property.m_copy()
some_property.a_eln = ELNAnnotation(component=ELNComponentEnum.NumberEditQuantity)
```
This does work for specifying the type of a sub section.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1634Upload page refactor2023-12-21T15:55:49ZLauri HimanenUpload page refactorWe want to make the upload page more intuitive and also more capable in terms of filtering and interacting with the entries. This refactoring may also affect the terminology (upload vs project?) and how datasets and DOIs are handled.
He...We want to make the upload page more intuitive and also more capable in terms of filtering and interacting with the entries. This refactoring may also affect the terminology (upload vs project?) and how datasets and DOIs are handled.
Here are some initial ideas to keep track of them:
- The user cannot really search for certain entries within an upload directly in the upload page. This might be useful when e.g. deleting or otherwise interacting with the entries. Suggested by @jlehm.
- Add `method_name` and `workflow_name` from results in the table. I really think this should be by default, at least from my side it is complicated to clearly see which entry comes from where, but tbh I think then the layout should change a bit to allocate for this extra info. Suggested by @pizarroj
- Ctrl+click to open tabs into entries. Suggested by @pizarroj
- Drop-zone component for dropping or adding files instead of a button. Suggested by @josmahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1630Deprecate or extend metadata.domain2023-12-21T16:06:52ZAlvin Noe LadinesDeprecate or extend metadata.domainEntryArchive.metadata.domain currently is either dft/ems. This obviously need to be extended (see e.g. https://github.com/nomad-coe/nomad/issues/64). However, I am not quite sure if this is still relevant and therefore be deprecated. @hi...EntryArchive.metadata.domain currently is either dft/ems. This obviously need to be extended (see e.g. https://github.com/nomad-coe/nomad/issues/64). However, I am not quite sure if this is still relevant and therefore be deprecated. @himanel1 @mscheidg what do you think?https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1498Bundle format test and documentation2023-12-21T15:39:10ZMarkus ScheidgenBundle format test and documentationWe should try our bundle import and export CLI functionality on our first two example uploads. David implemented this, but I am not sure what we actually have here. I want to see what this can already do and what not. We need this as a s...We should try our bundle import and export CLI functionality on our first two example uploads. David implemented this, but I am not sure what we actually have here. I want to see what this can already do and what not. We need this as a starting point for further discussions on data transfer.
As a side effect, please extend the documentation with a brief "how to import and export uploads". This should cover the commands and a rundown on the bundle format, e.g. what files it contains, whats in the nomad.json, etc. We can put this under "NOMAD Oasis" for now.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1479Normalization refactor2023-12-21T16:06:50ZLauri HimanenNormalization refactorWe currently have a concept of "normalization" that is separate from parsing. For this we have separate classes that are run in a pre-determined order based on what is configured in nomad.yaml. In addition, schemas may define a "normaliz...We currently have a concept of "normalization" that is separate from parsing. For this we have separate classes that are run in a pre-determined order based on what is configured in nomad.yaml. In addition, schemas may define a "normalize" function that specified schema-specific functionality that is run by a special `MetainfoNormalizer`, typically at the very end.
There are certain limitations to this approach:
- A single global order for the normalizers does not work for every parser and schema.
- The distinction between parsing and normalizing is often blurred.
- It is hard to track the execution order of normalizers
- It is hard to understand the difference between a `Normalizer` and a `normalize`-function.
- It is hard for a plugin developer to make use of the existing normalization code in `nomad.normalizing`.
In order to simplify this, we could think of a simpler strategy, such as:
- All processing of an entry could be moved into a single `normalize`/`parse` function of the schema. In the case of python schemas this is almost the case already. In the case of a parser, we would explicitly define a schema as well that would just combine the contents of the current `metainfo` folder and `parser.py` module. Essentially parser == schema.
- There would no longer be a set of separate `Normalizers` that would be run after parsing: each schema can still make use of their functionaliy, but they would be in full control of the call order.
- For parsers/schemas that behave very similarly (such as all electronic structure parsers) there could be base classes that trigger a common set of normalization procedures, but the schema could overwrite this at any point.
- The current code in `nomad.normalizing` is based on classes that operate on archives. The classes typically contain different methods for processing different parts of the data. It is hard for a plugin developer to use the existing normalization functionality since it is abstracted inside these monolithic classes. The situation could be improved by providing several smaller pure functions inside the `nomad.normalizing.<name>` modules.
I feel that this would bridge the gap between parsers and Python schemas and would make plugin development much more intuitive.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1440Reordering the order of quantities and sections in the ELN2023-12-21T16:04:41ZAndrea AlbinoReordering the order of quantities and sections in the ELNTesting the implementation accomplished after #1305, I noticed that we cannot mix quantities and sections.
Basically, the section are always flushed together at the bottom.
I think the main reason to have this custom ordering touching b...Testing the implementation accomplished after #1305, I noticed that we cannot mix quantities and sections.
Basically, the section are always flushed together at the bottom.
I think the main reason to have this custom ordering touching both sections and quantities is to bring the sections at the beginning of our ELN visualization, and especially to flush at the bottom the giant RichTextEditQuantity, because it hinders the view on everything that is below.
@g-michaelgoette you may comment whether you may find this useful as well or not really.
![Screenshot_from_2023-04-21_13-33-45](/uploads/5ccf66547781482c81d2d1f5cc73e952/Screenshot_from_2023-04-21_13-33-45.png)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1375Deprecate browse annotation for explicit JSON and HTML metainfo type.2023-12-21T16:04:41ZMarkus ScheidgenDeprecate browse annotation for explicit JSON and HTML metainfo type.This would remove a relatively unknown and superflous annotation. Explicit types would give us more options to check values (e.g. sanitize) on deserialisation.This would remove a relatively unknown and superflous annotation. Explicit types would give us more options to check values (e.g. sanitize) on deserialisation.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1272Formula rendering improvement2023-12-21T16:06:49ZLauri HimanenFormula rendering improvementAll of the chemical formulas we display within non-editable fields should be rendered with subscripts. For editable fields (URLs, search boxes etc.) the plain-text version will still be used.
We need a function to transform a regular pl...All of the chemical formulas we display within non-editable fields should be rendered with subscripts. For editable fields (URLs, search boxes etc.) the plain-text version will still be used.
We need a function to transform a regular plain-text formula to proper formatting + a possible react component for displaying formula. These should be used in several locations, including overview page, search chips, table rows etc.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1261Improve the release process2023-12-21T16:06:49ZMarkus ScheidgenImprove the release process- Don't know how to create an actual release with a NOMAD version that matches the tag. You'll always get a version like `1.1.7.dev0+g2ecdd77f8.d20221223`. How to i create a `1.1.6` image and python package?
- There is no docker tag crea...- Don't know how to create an actual release with a NOMAD version that matches the tag. You'll always get a version like `1.1.7.dev0+g2ecdd77f8.d20221223`. How to i create a `1.1.6` image and python package?
- There is no docker tag created for the gitlab tag
- The tag pipeline is not building an image. So I have to re-run the latest `develop` pipeline to build with the new git generated version
- The integrations test don't work. Probably a file is missing from the image that is used to run the integration test client.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1206Switching to src layout2022-12-19T10:17:38ZAdam FeketeSwitching to src layouthttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1184Use CLI to start nomad services in docker-compose2023-12-21T15:38:47ZMarkus ScheidgenUse CLI to start nomad services in docker-composeThe docker-compose using different scripts and commands to start the services (app, worker, north). Instead, it should simply use `python -m nomad.cli admin run ...`. This would give us more control and flexibility to change how the serv...The docker-compose using different scripts and commands to start the services (app, worker, north). Instead, it should simply use `python -m nomad.cli admin run ...`. This would give us more control and flexibility to change how the services are started without needing to update installations. Would also be more consistent with how we run services in development and the bare-metal installation.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1014Remove domain from parsers2022-09-01T07:16:00ZMarkus ScheidgenRemove domain from parsersThis is obsolete and not used anymore. These kinda *entry types* are now differentiated by different methods in the section results. We should remove the domain attribute from the parsers (and potentially the search index and metadata)This is obsolete and not used anymore. These kinda *entry types* are now differentiated by different methods in the section results. We should remove the domain attribute from the parsers (and potentially the search index and metadata)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1009Metainfo data type improvements2023-12-21T15:38:46ZLauri HimanenMetainfo data type improvementsOur metainfo has some inconsistencies in how the different data types are handled. Here are some observations which could be broken down into separate issues if we decide to act on them:
- Units are ignored for primitive python numeric ...Our metainfo has some inconsistencies in how the different data types are handled. Here are some observations which could be broken down into separate issues if we decide to act on them:
- Units are ignored for primitive python numeric types (#1006).
- Units cannot be assigned to non-numpy 1D arrays (`pint` does not allow assigning units to regular python 1D lists).
- Units can be assigned to non-numeric quantities (maybe a `@constraint` would help here).
- Possible precision loss in downcasting is not handled consistently. For primitive types, one cannot assign values whose values do not match the target data type. This ensures that no precision is lost, but is too strict (cannot assign int to float). For numpy data types, there is no check for precision loss, and values can be casted freely. E.g. storing a float array to an int array does not raise any errors.
- The differences between some of the data types are quite vague. We do support both 32 and 64 bit `int` and `float` numbers, but is this difference realized when we are saving the archive to disk? How does the fact that Javascript only deals with one numeric 64 bit numeric type (`Number`) affect things when reading/saving data in an ELN?
- Arrays with dimensions > 1 can only be created for quantities that have a numpy data type. This is causing some confusion especially for people who do not know what numpy is.
My gut feeling would be that many of these issues would if we used numpy internally when storing all numeric values, no matter what shape (0D, 1D, ND). This way our metainfo would have one "normalized" form for all numeric data, but the user could use any compatible data type in assignments (we would check the compatibility automatically). This would also mean that upon accessing numeric data, the user would always get a numpy data back. Whether this is confusing or not is hard to say. This change would also mean that we could simplify the supported numeric data types. How far we simplify them is an open question. One extreme is to only support two numeric types: `int` (64 bit integer) and `float` (64 bit float). Another option would be to keep support for different variants (`int32`, `int64`, `float32`, `float64`, `uint32`, `uint64` etc.) but maybe use simple strings instead of the actual numpy data type objects, as it may be confusing for ELN users.
All of this is open for discussion and any comments are welcome.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/666Refactor "owner"2023-12-21T15:39:03ZDavid SikterRefactor "owner"The "owner" field, used in many api calls, uses terminology that doesn't fit so well after the changes done to the permission structure. A suggestion for improvement would be to call it "scope", rather than "owner" and change some of the...The "owner" field, used in many api calls, uses terminology that doesn't fit so well after the changes done to the permission structure. A suggestion for improvement would be to call it "scope", rather than "owner" and change some of the values, for example something like this:
|old name |suggested new name |
|---------|-------------------|
| public | public |
| staging | staging |
| all | **visible** |
| visible | **fully_visible** |
| user | **main_author** |
| | **writer** |
| shared | **viewer** |
| admin | admin |
The new values could first be introduced as synonyms for the existing values, to make the transition in steps.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/627Consistent component for "loading ..." status2023-12-21T15:39:03ZMarkus ScheidgenConsistent component for "loading ..." statusAt some point we discussed that we should use some consistent mechanism for placeholders while data is loading.
This was moved from #619At some point we discussed that we should use some consistent mechanism for placeholders while data is loading.
This was moved from #619https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/617Use current ASE2023-12-21T15:38:43ZAlvin Noe LadinesUse current ASEWe are currently using ase==3.19.0 but I need ase==3.22.0 for ASR to work. I tried to fix the dependency issues on the parser side but there seems to be an interdependency issue in the packages when installing nomad.We are currently using ase==3.19.0 but I need ase==3.22.0 for ASR to work. I tried to fix the dependency issues on the parser side but there seems to be an interdependency issue in the packages when installing nomad.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/616Use pydantic settings management for nomad.config2023-12-21T15:40:00ZMarkus ScheidgenUse pydantic settings management for nomad.configPydantic has build in support for [settings management](https://pydantic-docs.helpmanual.io/usage/settings/). This supports everything that our `NomadConfig`-obj system provides (but better):
- nested config objects
- override from first...Pydantic has build in support for [settings management](https://pydantic-docs.helpmanual.io/usage/settings/). This supports everything that our `NomadConfig`-obj system provides (but better):
- nested config objects
- override from first env (limited), than config file, than default values
Here is an example of how it could work:
```python
from typing import Dict, Any
from pydantic import BaseModel, Field, BaseSettings, HttpUrl
import yaml
import os.path
import os
class Client(BaseModel):
user: str = Field('test', description='The username.')
password: str = '*'
url: HttpUrl = Field('http://nomad-lab.eu')
class NomadConfig(BaseSettings):
client: Client = Client()
not_nested: str = 'hello'
class Config:
env_prefix = 'nomad_'
case_sensitive = False
@classmethod
def customise_sources(
cls,
init_settings,
env_settings,
file_secret_settings):
return (
init_settings,
env_settings,
yaml_config_settings_source, file_secret_settings)
def yaml_config_settings_source(settings: BaseSettings) -> Dict[str, Any]:
if not os.path.exists('nomad.yaml'):
return {}
try:
with open('nomad.yaml') as f:
data = yaml.load(f, Loader=yaml.FullLoader)
if data is None:
return {}
return data
except yaml.YAMLError as e:
raise e
config = NomadConfig()
```
The support of env vars is limited as they only apply to the first level (e.g. NomadConfig). To set nested fields, the top level env var has to be set with a json value:
```
export NOMAD_CLIENT='{"password":"mypassword"}'
```
If necessary this could be replaced by a custom settings source.
```