nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2019-08-08T13:31:39Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/183Re-processing does not update calc where processing failed2019-08-08T13:31:39ZMarkus ScheidgenRe-processing does not update calc where processing failedmigration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/182Cut of file names in raw file gui2019-08-08T13:31:39ZMarkus ScheidgenCut of file names in raw file guimigration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/174Re-process the migration2019-07-24T12:01:52ZMarkus ScheidgenRe-process the migrationImplement a function to reprocess everything.Implement a function to reprocess everything.migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/171Raw file API to read files and file sizes of calculation2019-07-19T09:47:30ZMarkus ScheidgenRaw file API to read files and file sizes of calculationCurrent auxfiles strategy is not good. Using a simple cutoff on auxfiles does not seam to be a good idea, since many calcs have >50 auxfiles. Storing all the auxfile paths seems also be a bad idea? Maybe when we just store last bit? Or d...Current auxfiles strategy is not good. Using a simple cutoff on auxfiles does not seam to be a good idea, since many calcs have >50 auxfiles. Storing all the auxfile paths seems also be a bad idea? Maybe when we just store last bit? Or do not store them at all, but read them via separate API.
This is related to #169migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/170Transfer uploads to uploader during migration2019-07-24T12:03:20ZMarkus ScheidgenTransfer uploads to uploader during migrationOnly the calcs are given to the respective uploader, not the uploads.Only the calcs are given to the respective uploader, not the uploads.migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/161A "script" that quickly allows to migrate new data to a certain deployment.2019-08-05T09:52:46ZMarkus ScheidgenA "script" that quickly allows to migrate new data to a certain deployment.The scenario is that we use a deployment of nomad-FAIRDI alongside the old python API. Once the migration of old data is completed and we want to discontinue the old python API, we need a script to move new data (produced by either the o...The scenario is that we use a deployment of nomad-FAIRDI alongside the old python API. Once the migration of old data is completed and we want to discontinue the old python API, we need a script to move new data (produced by either the old python API or the mimicing FARIDI deployment) to the fully migrated deployment.
## How to do it
### requirements
- a CLI *admin* command to pull data from a different source deployment via the source deployments API
- only available for the admin user (for now) and he can also pull restricted data
- it is enough if the upload files are provides via path; the command can make local copy (or even mv); no need to stream the actual data yet
- this can be extended to a actual mirror functionality later
### tasks
The 'mirror' API endpoint to export data
- [x] endpoint uses upload as param
- [x] endpoint provides metadata of upload + all calcs + paths of upload files
The admin command to import data
- [x] the *admin* command that uses the search endpoint to aggregate over all uploads
- [x] the *admin* command uses the 'mirror' endpoint to get data
- [x] the *admin* command puts the data into mongo, es, and the file system
### 'mirror' endpoint results
```json
{
"upload_id": "<id>",
"upload": {
/* ... */
),
"calcs": [
/* ... */
]
"upload_files": "<path to public upload files>"
}
```migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/155Migration admin should find embargoed calcs2019-05-07T07:54:12ZMarkus ScheidgenMigration admin should find embargoed calcsEven the admin user will not get published calcs with embargo when searchingEven the admin user will not get published calcs with embargo when searchingmigration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/154Exception in migration upload2023-09-05T15:38:32ZMarkus ScheidgenException in migration upload```
Traceback (most recent call last):
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 875, in migrate_package
package_report = self.migrate_package(package, delete_failed=delete_failed, ...```
Traceback (most recent call last):
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 875, in migrate_package
package_report = self.migrate_package(package, delete_failed=delete_failed, republish=republish)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 1194, in migrate_package
for _, source_calc_with_metadata in source_calcs.values()]
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 1194, in <listcomp>
for _, source_calc_with_metadata in source_calcs.values()]
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 1248, in _to_api_metadata
_name=ds.get('name', None)) for ds in calc_with_metadata.datasets],
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 1248, in <listcomp>
_name=ds.get('name', None)) for ds in calc_with_metadata.datasets],
TypeError: 'NoneType' object is not subscriptable
```
```json
{
"_index": "logstash-2019.05.07",
"_type": "doc",
"_id": "ooXfkGoB0XR4BDyb_E7L",
"_version": 1,
"_score": null,
"_source": {
"event": "unexpected exception while migrating packages",
"logger_name": "nomad.migration",
"nomad.migration.package_id": "_ttAlUeOSDyKXf2ZvPNVFQ",
"level": "ERROR",
"host": "nomad-flink-01",
"type": "Logstash",
"path": "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py",
"@timestamp": "2019-05-07T05:59:19.396Z",
"@version": "1",
"nomad.migration.migration_version": 8,
"nomad.service": "client",
"stack_info": null,
"port": 47934,
"message": "unexpected exception while migrating packages",
"digest": "_metadata.datasets],\n File \"/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py\", line 1248, in <listcomp>\n _name=ds.get('name', None)) for ds in calc_with_metadata.datasets],\nTypeError: 'NoneType' object is not subscriptable",
"tags": [
"nomad",
"devel"
],
"exception": "Traceback (most recent call last):\n File \"/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py\", line 875, in migrate_package\n package_report = self.migrate_package(package, delete_failed=delete_failed, republish=republish)\n File \"/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py\", line 1194, in migrate_package\n for _, source_calc_with_metadata in source_calcs.values()]\n File \"/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py\", line 1194, in <listcomp>\n for _, source_calc_with_metadata in source_calcs.values()]\n File \"/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py\", line 1248, in _to_api_metadata\n _name=ds.get('name', None)) for ds in calc_with_metadata.datasets],\n File \"/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py\", line 1248, in <listcomp>\n _name=ds.get('name', None)) for ds in calc_with_metadata.datasets],\nTypeError: 'NoneType' object is not subscriptable",
"nomad.migration.source_upload_id": "uiByHdhTkP0PJOya9Im7odCoX-ZEy06D6cy0V7QFI",
"nomad.release": "devel"
},
"fields": {
"@timestamp": [
"2019-05-07T05:59:19.396Z"
]
},
"highlight": {
"nomad.service.keyword": [
"@kibana-highlighted-field@client@/kibana-highlighted-field@"
],
"level.keyword": [
"@kibana-highlighted-field@ERROR@/kibana-highlighted-field@"
]
},
"sort": [
1557208759396
]
}
```migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/152Exception on building migration index2019-06-25T12:14:16ZMarkus ScheidgenException on building migration index```
ERROR nomad.migration 2019-04-30T05:03:58 could not index
- exception: Traceback (most recent call last):
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 512, in index
...```
ERROR nomad.migration 2019-04-30T05:03:58 could not index
- exception: Traceback (most recent call last):
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 512, in index
source_calc.metadata = calc.to_calc_with_metadata().__dict__
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/coe_repo/calc.py", line 329, in to_calc_with_metadata
if topic.cid == base.topic_code:
AttributeError: 'NoneType' object has no attribute 'cid'
- nomad.migration.pid: 5398526
- nomad.release: devel
- nomad.service: client
```migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/151Catch migration exception2019-06-25T08:17:42ZMarkus ScheidgenCatch migration exception```
Traceback (most recent call last):
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/bin/nomad", line 11, in <module>
load_entry_point('nomad', 'console_scripts', 'nomad')()
File "/nomad/u/mscheidg/Docu...```
Traceback (most recent call last):
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/bin/nomad", line 11, in <module>
load_entry_point('nomad', 'console_scripts', 'nomad')()
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/client/migration.py", line 228, in upload
create_packages=create_packages, only_republish=republish, wait=wait)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/nomad/migration.py", line 840, in migrate
for arg in args:
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/mongoengine/queryset/queryset.py", line 99, in _iter_results
self._populate_cache()
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/mongoengine/queryset/queryset.py", line 118, in _populate_cache
self._result_cache.append(six.next(self))
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/mongoengine/queryset/base.py", line 1495, in __next__
raw_doc = six.next(self._cursor)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/pymongo/cursor.py", line 1189, in next
if len(self.__data) or self._refresh():
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/pymongo/cursor.py", line 1126, in _refresh
self.__send_message(g)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/pymongo/cursor.py", line 982, in __send_message
helpers._check_command_response(first)
File "/nomad/u/mscheidg/Documents/Projects/nomad/3-git/nomad-FAIR/.pyenv/lib/python3.6/site-packages/pymongo/helpers.py", line 152, in _check_command_response
raise CursorNotFound(errmsg, code, response)
pymongo.errors.CursorNotFound: cursor id 12091759078 not found
```migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/150Strip POTCAR Vasp files2019-04-30T11:13:07ZMarkus ScheidgenStrip POTCAR Vasp filesThe agreement with Gerd Kresse says something about only publicing a *stripped* version of POTCAR files.The agreement with Gerd Kresse says something about only publicing a *stripped* version of POTCAR files.migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/145Mongo documents get 'too large' for big uploads (100k+ calcs)2019-05-03T10:42:02ZMarkus ScheidgenMongo documents get 'too large' for big uploads (100k+ calcs)This happened while publishing `/nomad/repository/data/extracted/bda3ef9f8dd1f2eadcfdcdcdaa496e3` (package id `81I7ZF14SBKYNvKk6NXEoQ`).
```
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py"...This happened while publishing `/nomad/repository/data/extracted/bda3ef9f8dd1f2eadcfdcdcdaa496e3` (package id `81I7ZF14SBKYNvKk6NXEoQ`).
```
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1813, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1799, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_restplus/api.py", line 325, in wrapper
resp = resource(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/views.py", line 88, in view
return self.dispatch_request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restplus/resource.py", line 44, in dispatch_request
resp = meth(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_restplus/marshalling.py", line 136, in wrapper
resp = f(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_httpauth.py", line 104, in decorated
return f(*args, **kwargs)
File "/app/nomad/api/auth.py", line 106, in wrapper
return func(*args, **kwargs)
File "/app/nomad/api/auth.py", line 125, in wrapper
return func(*args, **kwargs)
File "/app/nomad/api/upload.py", line 373, in post
upload.publish_upload()
File "/app/nomad/processing/base.py", line 506, in wrapper
self.save()
File "/usr/local/lib/python3.6/site-packages/mongoengine/document.py", line 392, in save
write_concern)
File "/usr/local/lib/python3.6/site-packages/mongoengine/document.py", line 510, in _save_update
upsert=upsert, **write_concern)
File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 3194, in update
write_concern, collation=collation)
File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 851, in _update_retryable
_update, session)
File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1248, in _retryable_write
return self._retry_with_session(retryable, func, s, None)
File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1201, in _retry_with_session
return func(session, sock_info, retryable)
File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 847, in _update
retryable_write=retryable_write)
File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 817, in _update
retryable_write=retryable_write).copy()
File "/usr/local/lib/python3.6/site-packages/pymongo/pool.py", line 584, in command
self._raise_connection_failure(error)
File "/usr/local/lib/python3.6/site-packages/pymongo/pool.py", line 745, in _raise_connection_failure
raise error
File "/usr/local/lib/python3.6/site-packages/pymongo/pool.py", line 579, in command
unacknowledged=unacknowledged)
File "/usr/local/lib/python3.6/site-packages/pymongo/network.py", line 128, in command
name, size, max_bson_size + message._COMMAND_OVERHEAD)
File "/usr/local/lib/python3.6/site-packages/pymongo/message.py", line 965, in _raise_document_too_large
raise DocumentTooLarge("%r command document too large" % (operation,))
pymongo.errors.DocumentTooLarge: 'update' command document too large
```migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/143SystemNormalizer needs to check atom labels2019-05-03T12:47:54ZMarkus ScheidgenSystemNormalizer needs to check atom labelsThere are atom labels in the data that are definitely not atom labels. We should reject parser produced atom labels that cannot properly build an ase.Atoms objects.There are atom labels in the data that are definitely not atom labels. We should reject parser produced atom labels that cannot properly build an ase.Atoms objects.migration finishedDaniel SpeckhardDaniel Speckhardhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/140Setup for coe GUI2019-07-11T07:44:08ZMarkus ScheidgenSetup for coe GUIdocker compose (?) based setup to run nomad coe services necessary for the coe GUIdocker compose (?) based setup to run nomad coe services necessary for the coe GUImigration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/139Allow to reprocess calculations (via API)2019-05-03T12:47:43ZMarkus ScheidgenAllow to reprocess calculations (via API)This is important
- for migration to redo failed processings after fixes
- as a predecossor for an actual (automatic) re-process on normalizer/parser updates.This is important
- for migration to redo failed processings after fixes
- as a predecossor for an actual (automatic) re-process on normalizer/parser updates.migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/138Allow to update the coe db2019-04-26T14:58:24ZMarkus ScheidgenAllow to update the coe dbIf we do the migration with disabled coe db, we need a function to update the coe db after the fact.If we do the migration with disabled coe db, we need a function to update the coe db after the fact.migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/136Further parallise packaging to allow package of large aflowlib uploads2019-04-01T13:38:35ZMarkus ScheidgenFurther parallise packaging to allow package of large aflowlib uploadsmigration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/135Added parser/normalizer/nomad version+git hash to calc metadata2019-04-01T09:24:23ZMarkus ScheidgenAdded parser/normalizer/nomad version+git hash to calc metadatamigration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/134Allow to publish failed calcs.2019-03-25T14:04:53ZMarkus ScheidgenAllow to publish failed calcs.We ignore failed calcs. But they should be added published to the repo with just their user metadata.We ignore failed calcs. But they should be added published to the repo with just their user metadata.migration finishedMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/133Add handles2019-05-06T13:23:27ZMarkus ScheidgenAdd handlesIts not just PIDs, but handles.Its not just PIDs, but handles.migration finishedMarkus ScheidgenMarkus Scheidgen