nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2023-12-21T15:55:48Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/234Improvements to the Edit dialog2023-12-21T15:55:48ZMarkus ScheidgenImprovements to the Edit dialog- [x] the user completion with all users is slow, jumpy, and inconsistent
- [ ] we need a solution for new users
- [ ] ideally, we have more complex actions like set comment, add reference, remove author, etc.
- [ ] the presentation of t...- [x] the user completion with all users is slow, jumpy, and inconsistent
- [ ] we need a solution for new users
- [ ] ideally, we have more complex actions like set comment, add reference, remove author, etc.
- [ ] the presentation of the changes is not prominent enoughhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/198PIDs for entries2023-12-21T15:38:37ZMarkus ScheidgenPIDs for entrieshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/208Overhaul the file handling2022-05-13T09:26:41ZMarkus ScheidgenOverhaul the file handlingThere should be the following concepts
- `mainfile`, the main output file, as usual
- `codefiles`, other files generated or inputted to the code
- `auxfiles`, other files in directory (and subdirectories?) respecting some heuristics
- `...There should be the following concepts
- `mainfile`, the main output file, as usual
- `codefiles`, other files generated or inputted to the code
- `auxfiles`, other files in directory (and subdirectories?) respecting some heuristics
- `allfiles`, files in same directory (currently `auxfiles` + `mainfile`)
`auxfiles` and `allfiles` should not be part of processing, but deduced by the API on request.
- [ ] "codefiles" based on filename regexps from parser
- [ ] allow to load more files in GUI
- [ ] include sub directories
- [ ] download all, dataset download, etc. use "allfiles"
- [ ] heuristicts to detect meaningful `auxfiles`https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/591Remove old search index and v0.10.x (flask) API, also clean the CLI2021-09-29T17:43:33ZMarkus ScheidgenRemove old search index and v0.10.x (flask) API, also clean the CLIWe will hardfork NOMAD and run v1.0.0 on new databases that will not support the old search index (and therefore most of the old flask API).
- [x] remove v0 search and unify the search module once again
- [x] remove the old metainfo se...We will hardfork NOMAD and run v1.0.0 on new databases that will not support the old search index (and therefore most of the old flask API).
- [x] remove v0 search and unify the search module once again
- [x] remove the old metainfo search extension that drove the v0 index
- [x] remove all of the old v0 API and its tests
- [x] remove obsolete commands from client CLI (mirror, statistics?)
- [x] migrate the integrationtests to v1 API
- [x] migrate upload CLI to v1 API
- [x] migrate local CLI to v1 API
- [x] migrate aflow update to v1 API
- [x] migrate uploads CLI to v1 search
- [x] migrate Optimade API
- [x] migrate dcat API
- [x] migrate `client.py` to v1 API
- [x] move lots of tiny API functions to v1 (done in #607)
- [x] temporarily add the old edit API to v1 (done in #607)
- [x] migrate all examples to requests and v1 API
- [x] remove `.dft` (and other domains)
- [x] remove `nomad.cli.client.legacy._create_client`
- [x] "test" the GUI after refactors
- [x] test the deployment
This depends on #588 and #607.v1.0.0-betaMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/432Distinguish weak and strong code support2021-06-17T15:55:43ZMarkus ScheidgenDistinguish weak and strong code supportThe quality of most parsers is not good. We should publicly distinguish three types of parsers:
- full parsers with archive support that create an archive and all searchable metadata
- metadata parsers that do not create a real archive a...The quality of most parsers is not good. We should publicly distinguish three types of parsers:
- full parsers with archive support that create an archive and all searchable metadata
- metadata parsers that do not create a real archive and only provide the section metadata or parts of it
- parsers that do not exist and where we only do the mainfile detection
| parser | complex. | state | entries | TODO (priority) |
| ------------ | -------- | -------------- | --------- | ------------------------------------------------ |
| abinit | 2 | new + tests | 18,781 | update metainfo(10) |
| amber | 3 | basic + tests | 1 | implement full parser |
| asap | 1 | new + tests | 3 | |
| atk | 1 | new + tests | 0 | |
| band | 2 | new + tests | 51 | add code-specific metainfo |
| bigdft | 1 | new + tests | 0 | |
| castep | 3 | new + tests | 1820 | |
| charmm | 3 | basic + tests | 7 | |
| cp2k | 3 | new + tests | 3451 | test other versions, calculation types(10) |
| cpmd | 1 | basic + tests | 1 | implement full parser |
| crystal | 2 | new + tests | 4228 | |
| dftbplus | 2 | basic + tests | 1 | implement full parser |
| dl-poly | 3 | basic + tests | 2 | implement full parser |
| dmol3 | 2 | basic + tests | 2 | implement full parser |
| eelsdb | 1 | new crawler | ? | ? |
| elastic | 2 | new + tests | 18 | |
| elk | 2 | basic + tests | 3 | implement full parser |
| exciting | 3 | new + tests | 17851 | add support for other outputs (5) |
| fhi-aims | 3 | new + tests | 978899 | update metainfo(10), add props.(6) |
| fhi-vibes | 1 | new + tests | 0 | include support for other codes, metainfo |
| fleur | 2 | basic + tests | 11 | implement full parser |
| fplo | 3 | basic + tests | 2 | implement full parser |
| gamess | 2 | basic + tests | 73 | implement full parser |
| gaussian | 2 | new + tests | 1641540 | update metainfo(10) |
| gpaw | 2 | new + tests | 13175 | add props.(7) |
| gromacs | 2 | new + tests | 12 | add support for other formats(7) |
| gromos | 2 | basic + tests | 3 | implement full parser |
| gulp | 2 | basic + tests | 3 | implement full parser |
| lammps | 2 | new + tests | 3106 | |
| lib-atoms | 2 | basic + tests | 7 | implement full parser |
| molcas | 2 | basic + tests | 4 | implement full parser |
| mopac | 2 | basic + tests | 2 | implement full parser |
| mpes | 1 | new json | ? | ? |
| namd | 2 | basic + tests | 7 | implement full parser |
| nwchem | 1 | new + tests | 1896 | |
| octopus | 2 | new + tests | 108283 | update metainfo(10) |
| onetep | 3 | basic + tests | 8 | implement full parser |
| openkim | 2 | new + tests | 1 | |
| orca | 2 | new + tests | 90081 | update metainfo(10) |
| phonopy | 2 | new + tests | 401 | |
| qbox | 3 | basic + tests | 30 | implement full parser |
| quantum-espre| 3 | new + tests | 98038 | |
| siesta | 2 | basic + tests | 1 | implement full parser |
| tinker | 2 | basic + tests | 5 | implement full parser |
| turbomole | 3 | new + tests | 846 | |
| vasp | 3 | new + tests | 7983375 | |
| wien2k | 2 | new + tests | 1743 | parse more quantities |Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/474Mover parsers to GitHub2021-03-30T14:05:27ZMarkus ScheidgenMover parsers to GitHubThis should probably done via script. If we keep the script, we might use it for future updates.
- skip if already on github
- ensure that a proper license file exists in the repo
- replace the copyright headers in all source files
- a ...This should probably done via script. If we keep the script, we might use it for future updates.
- skip if already on github
- ensure that a proper license file exists in the repo
- replace the copyright headers in all source files
- a minimal github actions pipeline for minimal test and (commented out) linting
- create a repo on GitHub with name, description, license ... otherwise start with defaults
- push the repo with all branches/tags
- change the mother project submodulev0.10.0Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/396Phonopy Workflows2020-12-18T16:07:58ZMarkus ScheidgenPhonopy Workflows- Phonopy parser to be an actual Phonopy parser and not just an FHI-aim phonopy parser
- Phonopy parser to uses the new workflow system- Phonopy parser to be an actual Phonopy parser and not just an FHI-aim phonopy parser
- Phonopy parser to uses the new workflow systemAlvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/363VASP geometry optimizations as workflows2020-12-18T16:07:57ZMarkus ScheidgenVASP geometry optimizations as workflows- [x] metainfo for workflows
- [x] adapt VASP parser
- [ ] normalizer (system, dos, encyclopedia, ?) and section_metadata use the workflow result system/calculation
![image](/uploads/aaa6b494cf0afc03d96372e3a338c973/image.png)- [x] metainfo for workflows
- [x] adapt VASP parser
- [ ] normalizer (system, dos, encyclopedia, ?) and section_metadata use the workflow result system/calculation
![image](/uploads/aaa6b494cf0afc03d96372e3a338c973/image.png)Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/350Parallel archive access2020-12-18T16:07:57ZMarkus ScheidgenParallel archive accessThe performance when accessing large archive query result sets is poor. We identified this is mostly due to many small serial file system accesses to GPFS. The endpoint `nomad.app.api.archive.ArchiveQueryResource` should use threading to...The performance when accessing large archive query result sets is poor. We identified this is mostly due to many small serial file system accesses to GPFS. The endpoint `nomad.app.api.archive.ArchiveQueryResource` should use threading to read for many calcs simultaneously.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/403Clean up the experimental data2020-11-10T13:45:24ZMarkus ScheidgenClean up the experimental dataFor show-casing purposes the experimental section of NOMAD needs to be "cleaned"
- [x] rename CMS/EMS -> computational/experimental
- [x] fix the uploader and co-author names
- [x] fix other metadata like locations and dates
- [x] bette...For show-casing purposes the experimental section of NOMAD needs to be "cleaned"
- [x] rename CMS/EMS -> computational/experimental
- [x] fix the uploader and co-author names
- [x] fix other metadata like locations and dates
- [x] better metadata and experiment names
- [x] more data (e.g. automatised EELS indexing)
- [ ] EELS preview?
- [ ] maybe Markus Kühbach has real preview figs for his set
- [x] a disclaimer (in the search) about the "show-case" nature of the experimental section
- [ ] more databases to "index"
## show-cases the indexing of external databse
We could use [EELS](https://eelsdb.eu/spectra/) to show that NOMAD could crawl web-based databases to index. Simply Python web-scraping techniques should suffice to create an upload consisting of respective web-pages that "parsers" can convert into respective NOMAD metainfo data.
## show-case the indexing of external repositories
We could use zenoodo and its API to improve the metadata in NOMAD/experimental by downloading titles, descriptions, authors, etc.
## authors
There is a difference between the person uploading the metadata, i.e. the person providing the reference to the data and the authors of the data. The later usually given by an external database or repository. We need to reflect this in NOMAD user datamodel and add support for non NOMAD user authors: #404v0.10.0Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/407Workflows for elastic constants via elastic parser.2020-10-06T10:46:20ZMarkus ScheidgenWorkflows for elastic constants via elastic parser.Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/309v0.7.x to v0.8.x migration2020-05-18T16:27:28ZMarkus Scheidgenv0.7.x to v0.8.x migrationVersion 0.8Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/233"Overview" in nomad GUI homepage2020-05-05T13:03:12ZMarkus Scheidgen"Overview" in nomad GUI homepageIt would be informative to have an overview with basic statistics, supported codes, used normalizer, etc. It should show
- [x] repo statistics: number of entries, datasets
- [ ] archive statistics: number of quantities, number of sccs, T...It would be informative to have an overview with basic statistics, supported codes, used normalizer, etc. It should show
- [x] repo statistics: number of entries, datasets
- [ ] archive statistics: number of quantities, number of sccs, TBs
- [ ] raw statistics (?): TBs
- [ ] parsers: list with URL to code
- [ ] normalizer: list with description
- [ ] metainfo quantity statistics: number in public, number overall
- [x] nomad versionVersion 0.8Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/317Improving the search bar2020-04-30T11:51:33ZMarkus ScheidgenImproving the search barCurrently the searchbar only autocompletes on values from preset quantities and these values are requested per statistics from the API on each search. This has two issues:
- we need to aggregate all values all the time
- we don't allow t...Currently the searchbar only autocompletes on values from preset quantities and these values are requested per statistics from the API on each search. This has two issues:
- we need to aggregate all values all the time
- we don't allow the user any completion on other quantities
A new searchbar would first provide completions on search quantities and then completion on values from the chosen quantity. The search quantities can be preset (e.g. from `api/info` which gets them from `datamodel.py`) and autocomplete could be offered right when the user focuses the searchbar without typing anything. If the quantity is selected, the searchbar can use the api's quantity search to retrieve values for the quantities and show to top n values with the highest entry count. Selecting quantity and value can result in a chip like before.
The various searchtabs now have to set/unset the shown statistics and cannot rely on the fact that most statistics are loaded all the time.Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/232Visual representation of user metadata and quantities in search2020-04-06T07:22:52ZMarkus ScheidgenVisual representation of user metadata and quantities in searchVersion 0.8Alvin Noe LadinesAlvin Noe Ladineshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/298Move data from mongo to the archive2020-03-24T22:44:45ZMarkus ScheidgenMove data from mongo to the archiveCurrently all of `EntryMetadata` is stored in mongodb (in `proc.Calc.metadata`). This makes the entries quite big and some bulk operations on mongo are becoming quite slow. This might become even worse, if we think about adding encyclope...Currently all of `EntryMetadata` is stored in mongodb (in `proc.Calc.metadata`). This makes the entries quite big and some bulk operations on mongo are becoming quite slow. This might become even worse, if we think about adding encyclopedia metadata as well.
Only some quantities of `EntryMetadata` should be stored in mongo (especially what is editable by the user). The rest should be part of the archive. Downside is that optimade API and indexing elastic will need to read from the archive to work.
Tasks:
- [x] allow to store filtered `EntryMetadata` in archive with working references to optimade (encyclopedia)
- [x] new index function
- [x] adopt processing
- [x] adopt optimade API
![image](/uploads/2351ef278c25372040f5b046ff6619fa/image.png)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/286Multi domain support in API and GUI2020-02-25T13:11:50ZMarkus ScheidgenMulti domain support in API and GUI- [x] use prefix on domain relevant repo metadata
- [x] use multiple domains at the same time in API
- [x] use multiple domains at the same time in GUI- [x] use prefix on domain relevant repo metadata
- [x] use multiple domains at the same time in API
- [x] use multiple domains at the same time in GUImulti-domain support/metainfo 2.0Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/197Create API DOI assign capabilities2019-11-12T15:40:22ZMarkus ScheidgenCreate API DOI assign capabilitiesA replacement for the NOMAD CoE RepositoryMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/207Pypi parser packages2019-10-16T19:27:53ZMarkus ScheidgenPypi parser packagesWe must allow to easily use and install the NOMAD parser to make them more useful. We need
- stand-alone pypi packages for each parser, and one pypi package with all parser without unnecessary dependencies
- each parser needs consistent ...We must allow to easily use and install the NOMAD parser to make them more useful. We need
- stand-alone pypi packages for each parser, and one pypi package with all parser without unnecessary dependencies
- each parser needs consistent development instructions
In order to achieve this, we need to:
- reduce `python_common` to what is actually common: backend + parsing + metainfo .. rename it in the process
- move the specific meta-info to the parsers
- put each parser under the same skeleton (this is almost done by integrating parsers into nomad@fairdi)
Package structure could be:
- nomad (infrastructure + parsers)
- nomad-parsers (matching, simple cli)
- xxxx-parser (single command)
- nomad-parsing (old python common, current nomad.parsing), nomad-metainfo (+ old metainfo parser), nomad-normalizing (optional)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/195Create a new user management based on keycloak (OAuth)2019-09-02T12:13:30ZMarkus ScheidgenCreate a new user management based on keycloak (OAuth)- [x] 'user' replace the implementation
- [x] new auth API endpoints
- [x] test runs based on a keycloak mockup
- [x] local keycloak via docker compose
- [x] gui uses keycloak for login, logout, links to register, forget password, edit u...- [x] 'user' replace the implementation
- [x] new auth API endpoints
- [x] test runs based on a keycloak mockup
- [x] local keycloak via docker compose
- [x] gui uses keycloak for login, logout, links to register, forget password, edit user, etc.
- [x] install a keycloak on labdev
- [x] backup restore for keycloak
- [x] some tests on the real keycloak
- [x] theme keycloak
- [x] additional keycloak attributes
- [x] a cache for get_user
- [x] copy script for existing users to keycloakA replacement for the NOMAD CoE RepositoryMarkus ScheidgenMarkus Scheidgen