nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2019-02-04T14:52:13Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/91Symmetry Normalizer Not Finding Crystal Structure2019-02-04T14:52:13ZDaniel SpeckhardSymmetry Normalizer Not Finding Crystal StructureThe symmetry normalizer currently doesn't always assign several values in the symmetry section of the backend. This means important variables such as the crystal structure are sometimes missing after normalizing on main files. The symmet...The symmetry normalizer currently doesn't always assign several values in the symmetry section of the backend. This means important variables such as the crystal structure are sometimes missing after normalizing on main files. The symmetry normalizer calculates the dimension of the .dependencies/normalizers/symmetry/normalizer/normalizer-symmetry/symmetrynormalizer/symmetry_analysis.py and when it doesn't find the dimensions are equal to three it simply returns without assigning some key values. The structure normalizer calculates the bravais lattice value in the symmetry section of the backend. This can be used to find the crystal structure by a simple mapping. Ideally, one should move the symmetry section assigning to the symmetry normalizer and deal with the lower dimensional cases appropriately. This issue came up as the test_normalizing.py tests failed to find a crystal_structure value for several parsers (NWChem, BigDFT, CPMD). More information is found in the Normalizer Test failure section of this document (https://docs.google.com/document/d/1fTjSJIuRtbVWqS4CoXVlrf94MzYDWNqU-jtxkYOVAoQ/edit#).repo/archive consolidationDaniel SpeckhardDaniel Speckhardhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/89Adopt the nomad@FAIRDI prototype GUI to become the statging GUI2019-01-31T22:27:38ZMarkus ScheidgenAdopt the nomad@FAIRDI prototype GUI to become the statging GUICreate a second app based on the existing components with the following features
- [ ] only one view
- [x] allow raw file download
- [x] allow archive file download
- [x] modal for showing processed data and its meta info
- [x] ad public...Create a second app based on the existing components with the following features
- [ ] only one view
- [x] allow raw file download
- [x] allow archive file download
- [x] modal for showing processed data and its meta info
- [x] ad public/restricted on the auto commit
- [x] provide a proper login that connects to the repository login
- [x] ad "got it"-table documentation
- [ ] rework the shown hashes
- [ ] allow multiple uploads of slightly different .zip versions where existing files are skipped
- [ ] action to remove faulty calculationsrepo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/75Swagger API2018-12-23T16:02:10ZMarkus ScheidgenSwagger APIThe API should provide a generated swagger.json and respective swagger UI. This is already provided by the flask-restful clone flask-restplus. We simply need to migrate and implement all endpoints as flask-restplus resources. All users o...The API should provide a generated swagger.json and respective swagger UI. This is already provided by the flask-restful clone flask-restplus. We simply need to migrate and implement all endpoints as flask-restplus resources. All users of the API should migrate to swagger afterwards.
- [x] move to flask-restplus
- [x] refactor all endpoints to flask-restplus
- [x] client.py uses swagger (via bravado)
- [x] GUI uses swagger (via swagger-client/swagger-js)repo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/124Optimisations2019-03-14T08:54:12ZMarkus ScheidgenOptimisations### io
- [x] **(1)** skip bagit
- [x] **(1)** pack rawfiles directly into zips
- [x] **(1)** allow staging and public buckets to be on different fs
- [ ] (3) remove package upload zips after successful migration, configure what successfu...### io
- [x] **(1)** skip bagit
- [x] **(1)** pack rawfiles directly into zips
- [x] **(1)** allow staging and public buckets to be on different fs
- [ ] (3) remove package upload zips after successful migration, configure what successful means (e.g. no failures or missing calcs)
### computing related
- *(not doing this)* copy system normalisation if likely to be the same
- [x] system normalisation only for some systems with old heuristics
### parallelisation
- [ ] **(1)** complete/test package migration parallelisation
- [x] **(1)** catch/backoff/retry API Gateway timeouts (the api is just overworked)
- [x] rollback publish in repo db, if publish failed for errors
### analysis
- [x] (2) add 'source->target' value to quantity diff log message
- [x] (3) lower more of the log levels
- [x] **(1)** rethink errors for migration log outputrepo/archive consolidationhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/88Remove staging elastic search2019-01-13T14:42:19ZMarkus ScheidgenRemove staging elastic search.. or replace with repository elastic search.
The elastic search in staging was only necessary for the "all nomad" prototype. If we put uploads to the existing repository, its elastic search should do the job or replace the current stag..... or replace with repository elastic search.
The elastic search in staging was only necessary for the "all nomad" prototype. If we put uploads to the existing repository, its elastic search should do the job or replace the current staging elastic search. This will disable the "search" in the staging GUI (#89). Repository data and archive data in the staging GUI need to be read from files (#80).repo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/109Refactoring the fixtures2019-02-12T21:06:17ZMarkus ScheidgenRefactoring the fixtures- The test fixtures are confusing.
- There are parameterizations that are unnecessary.
- Establish clear hierarchy
- documentation
- synergies between common test patters for e.g. coe_repo vs search vs proc- The test fixtures are confusing.
- There are parameterizations that are unnecessary.
- Establish clear hierarchy
- documentation
- synergies between common test patters for e.g. coe_repo vs search vs procrepo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/108Refactoring of the datamodel module2019-02-12T14:42:58ZMarkus ScheidgenRefactoring of the datamodel moduleThis got a little out of hand with transformations spread through out the code.
- There should be a clear definition of upload and calc with their respective attributes.
- Instead of registering mappings, the necessary transformations s...This got a little out of hand with transformations spread through out the code.
- There should be a clear definition of upload and calc with their respective attributes.
- Instead of registering mappings, the necessary transformations should be implemented as simple member functions of the respective entity representations, e.g. within coe_repo.Calc, processing.Calc, etc.
- common transformation function names should be `to_calc_with_metadata`, `update_from_calc_with_metadata`,
`add_calc_with_metadata`, etc.repo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/98Documentation overhaul2019-02-05T16:06:27ZMarkus ScheidgenDocumentation overhaulA lot happened in the last month. The documentation (especially the none code reference parts) have diverged from reality alot. Read through it and adopt it to recent changes.A lot happened in the last month. The documentation (especially the none code reference parts) have diverged from reality alot. Read through it and adopt it to recent changes.repo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/116Multi mainfile directories, again2019-03-01T13:30:50ZMarkus ScheidgenMulti mainfile directories, againThere might be terribly many mainfiles in a single directory. Therefore we should evaluate, if
- [x] we should apply a general limit to the number of aux files
- [ ] do not have mainfiles being aux files of other mainfiles in the same di...There might be terribly many mainfiles in a single directory. Therefore we should evaluate, if
- [x] we should apply a general limit to the number of aux files
- [ ] do not have mainfiles being aux files of other mainfiles in the same directoryrepo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/83Perform upload delete with a celery task2019-01-15T15:25:50ZMarkus ScheidgenPerform upload delete with a celery taskDeleting an upload in staging is a potentially long running task, especially if raw/archive files are (re)-packed/bagged with #80.Deleting an upload in staging is a potentially long running task, especially if raw/archive files are (re)-packed/bagged with #80.repo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/82Perform unstage with a celery task2019-01-15T15:26:04ZMarkus ScheidgenPerform unstage with a celery taskMoving out of staging is a potentially long running task, especially if raw/archive files are (re)-packed/bagged with #80.Moving out of staging is a potentially long running task, especially if raw/archive files are (re)-packed/bagged with #80.repo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/80Transparent, restriction aware staging and public file structures2019-01-14T09:37:25ZMarkus ScheidgenTransparent, restriction aware staging and public file structuresWe need an abstraction for handling all files related to an *upload*.
There are two different structures for uploads in two different states: *staging* and *public*. Possible operations on uploads differ based on this state. Staging is...We need an abstraction for handling all files related to an *upload*.
There are two different structures for uploads in two different states: *staging* and *public*. Possible operations on uploads differ based on this state. Staging is used for processing, heavily editing, creating hashes, etc. Public is supposed to be a
almost readonly (beside metadata) storage.
```
fs/staging/<upload>/metadata/<calc>.json
/raw/**
/archive/<calc>.hdf5
/.frozen
/.public
/.restricted
fs/public/<upload>/metadata.json.gz
/raw-public.bagit.zip
/raw-restricted.bagit.zip
/archive-public.hdf5.zip
/archive-restricted.hdf5.zip
```
This is an complex issue with multiple sub-tasks
- [x] implementing/testing an respective the uploads.py module
- [x] adopt processing and api accordingly
- [x] make the raw file API restriction aware
- [x] do potential adaptations to the staging GUI
This is a prerequisite for building reasonable mirror functionality.
It would allow to move calc (and potentially also user) metadata out of the *repository db*, which also requires to adopt the *repoTool*.repo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/76Implicit upload create on PUT file2018-12-23T16:03:24ZMarkus ScheidgenImplicit upload create on PUT fileWe started with a more complex upload entity that could exist without an uploaded file. This makes things to complicated and is unnecessary.We started with a more complex upload entity that could exist without an uploaded file. This makes things to complicated and is unnecessary.repo/archive consolidationMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/131Make domain specific metadata configurable2019-04-12T09:02:52ZMarkus ScheidgenMake domain specific metadata configurable- [x] elastic mapping
- [x] datamodel
- [x] search api
- [x] gui- [x] elastic mapping
- [x] datamodel
- [x] search api
- [x] guiExperimental nomadMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/130Replace metadata.json with mongodb2019-03-21T17:40:46ZMarkus ScheidgenReplace metadata.json with mongodbbecause of reasons:
- we already have entries in mongo before publish
- these are additional small files that have to be packed
- if we allow editing metadata and this is the source of truth, we are effectively implementing a terrible da...because of reasons:
- we already have entries in mongo before publish
- these are additional small files that have to be packed
- if we allow editing metadata and this is the source of truth, we are effectively implementing a terrible database
- the only need for the files is mirroring and we can provide the json out of mongo for thatExperimental nomadMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/129Allow to disable postgres2019-03-20T15:41:17ZMarkus ScheidgenAllow to disable postgresIt is not needed for experimental data, the source of truth is the metadata (either mongo, or metadata.json).It is not needed for experimental data, the source of truth is the metadata (either mongo, or metadata.json).Experimental nomadMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/171Raw file API to read files and file sizes of calculation2019-07-19T09:47:30ZMarkus ScheidgenRaw file API to read files and file sizes of calculationCurrent auxfiles strategy is not good. Using a simple cutoff on auxfiles does not seam to be a good idea, since many calcs have >50 auxfiles. Storing all the auxfile paths seems also be a bad idea? Maybe when we just store last bit? Or d...Current auxfiles strategy is not good. Using a simple cutoff on auxfiles does not seam to be a good idea, since many calcs have >50 auxfiles. Storing all the auxfile paths seems also be a bad idea? Maybe when we just store last bit? Or do not store them at all, but read them via separate API.
This is related to #169migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/191A better and more "standard" help system2019-08-07T09:51:25ZMarkus ScheidgenA better and more "standard" help systemThe current help on the GUI is very intrusive and confusing. This should be replaced with more traditional help buttons, or simple hovers.The current help on the GUI is very intrusive and confusing. This should be replaced with more traditional help buttons, or simple hovers.Production uploadsMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/212Show calculation by PIDs2019-09-10T10:20:31ZMarkus ScheidgenShow calculation by PIDs- [x] refactor the path routing, see below
- [x] add an API endpoint that resolves PIDs
The path routing should work with the following paths
- `/gui/search`
- `/gui/dataset/doi/<doi>`
- `/gui/dataset/pid/<pid>`
- `/gui/entry/pid/<pid>...- [x] refactor the path routing, see below
- [x] add an API endpoint that resolves PIDs
The path routing should work with the following paths
- `/gui/search`
- `/gui/dataset/doi/<doi>`
- `/gui/dataset/pid/<pid>`
- `/gui/entry/pid/<pid>`
- `/gui/entry/id/<upload_id>/<calc_id>`A replacement for the NOMAD CoE RepositoryMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/222Parser use the new Metainfo2020-03-22T14:04:55ZMarkus ScheidgenParser use the new MetainfoAfter #59 and #221, we should move all parsers to use the new Metainfo. It already works (but is not used) for the VASP parser.
This includes an updated API that serves the metainfo (moved as part from #221).After #59 and #221, we should move all parsers to use the new Metainfo. It already works (but is not used) for the VASP parser.
This includes an updated API that serves the metainfo (moved as part from #221).multi-domain support/metainfo 2.0Markus ScheidgenMarkus Scheidgen