nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2024-03-19T09:37:50Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1944Config does not allow string loglevel like "INFO" anymore2024-03-19T09:37:50ZMarkus ScheidgenConfig does not allow string loglevel like "INFO" anymoreMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1812Scaffolding for the plugin example projects.2024-03-13T10:24:56ZMarkus ScheidgenScaffolding for the plugin example projects.We could add more features to the example plugin projects:
- [x] code style checks
- [ ] autoformatting checks and autoformatting pre-commit hooks (both opt-in)
- [x] linting
- [x] an empty mkdocs + github actions to deploy as gh-pages
...We could add more features to the example plugin projects:
- [x] code style checks
- [ ] autoformatting checks and autoformatting pre-commit hooks (both opt-in)
- [x] linting
- [x] an empty mkdocs + github actions to deploy as gh-pages
- [x] test coverage badge, pipeline badge
The mkdocs should include the theming/styling from the main project. We still need to figure out what should in the docs by default.
Some other ideas:
- maybe gh actions to build pypi package (is there something like a package/release registry in github?)
- maybe gh actions to build the docker file (gh does not have a docker resgistry? this would only make sense as opt-in where users define a registry to use)
The projects are :
https://github.com/nomad-coe/nomad-schema-plugin-example
https://github.com/nomad-coe/nomad-parser-plugin-example
@ladinesa is about to create a third one for normalizers
Just start with one project. Create PR there on github.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1778The nomad's appworker doesn't run outside of the nomad's root folder2023-12-21T15:40:07ZAdam FeketeThe nomad's appworker doesn't run outside of the nomad's root folderrunning nomad appworker other place than the nomad's root folder fails:
```
$ nomad admin run appworker
-------------- celery@fekad-mbp2015.local v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- macOS-12.3.1-x86_64-i386-64bit 202...running nomad appworker other place than the nomad's root folder fails:
```
$ nomad admin run appworker
-------------- celery@fekad-mbp2015.local v5.2.7 (dawn-chorus)
--- ***** -----
-- ******* ---- macOS-12.3.1-x86_64-i386-64bit 2023-11-13 17:23:49
- *** --- * ---
- ** ---------- [config]
- ** ---------- .> app: nomad.processing:0x125987310
- ** ---------- .> transport: amqp://rabbitmq:**@localhost:5672//
- ** ---------- .> results: disabled://
- *** --- * --- .> concurrency: 4 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. nomad.app.resources.routers.resources.retrieve_resources
. nomad.processing.base.proc_task
INFO: Started server process [90059]
INFO: Waiting for application startup.
ERROR: Traceback (most recent call last):
File "/Users/fekad/Work/nomad-1549-notebook-plugin/.venv/lib/python3.9/site-packages/starlette/routing.py", line 671, in lifespan
async with self.lifespan_context(app):
File "/Users/fekad/Work/nomad-1549-notebook-plugin/.venv/lib/python3.9/site-packages/starlette/routing.py", line 566, in __aenter__
await self._router.startup()
File "/Users/fekad/Work/nomad-1549-notebook-plugin/.venv/lib/python3.9/site-packages/starlette/routing.py", line 648, in startup
await handler()
File "/Users/fekad/Work/nomad-1549-notebook-plugin/nomad/app/main.py", line 152, in startup_event
GuiFiles.gui_artifacts_data = get_gui_artifacts_js()
File "/Users/fekad/Work/nomad-1549-notebook-plugin/nomad/cli/dev.py", line 79, in get_gui_artifacts_js
'exampleUploads': _generate_example_upload_metadata(),
File "/Users/fekad/Work/nomad-1549-notebook-plugin/nomad/cli/dev.py", line 259, in _generate_example_upload_metadata
with open('examples/data/uploads/example_uploads.yml') as infile:
FileNotFoundError: [Errno 2] No such file or directory: 'examples/data/uploads/example_uploads.yml'
ERROR: Application startup failed. Exiting.
```
Expected: behaviour: The appworker should run independently what folder you are.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1704App is always running just one worker2023-10-04T07:26:19ZMarkus ScheidgenApp is always running just one worker- the `--workers` setting to the `nomad run app` command does not work, because it runs the uvicorn dev server
- we are not running a proper process manager like gunicorn (as recommended by uvicorn)
Consequently all oasis and our produc...- the `--workers` setting to the `nomad run app` command does not work, because it runs the uvicorn dev server
- we are not running a proper process manager like gunicorn (as recommended by uvicorn)
Consequently all oasis and our production system is only running one process. With some loads timeouts and restarts are unavoidable.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1685Getting complaint in installation of nomad with python3.102023-12-21T15:38:52ZRubel MozumderGetting complaint in installation of nomad with python3.10In the dev environment installation process having a complaint from mdtraj package while `python version 3.10`:
```
File "/home/rubel/NOMAD-FAIRmat/GL/nomad-FAIR/.NF_1674_xrd/lib/python3.10/site-packages/pip/_vendor/pyproject_hoo...In the dev environment installation process having a complaint from mdtraj package while `python version 3.10`:
```
File "/home/rubel/NOMAD-FAIRmat/GL/nomad-FAIR/.NF_1674_xrd/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/home/rubel/NOMAD-FAIRmat/GL/nomad-FAIR/.NF_1674_xrd/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/rubel/NOMAD-FAIRmat/GL/nomad-FAIR/.NF_1674_xrd/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 396, in prepare_metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 507, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "<string>", line 301, in <module>
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
cythonize_one(*args)
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: mdtraj/geometry/src/_geometry.pyx
```
But no issues are found with `python 3.9`.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1631Publishing fails due to IndexError2023-08-16T11:40:26ZJoseph RudzinskiPublishing fails due to IndexErrorI am working to publish a dataset of molecular dynamics simulations (datsetid: k0FIIN93TDqcDatfaBWpXQ, not yet published) and recently made a series of somewhat large uploads (not exceeding the 30GB limit, but lots of entries, ~180 per u...I am working to publish a dataset of molecular dynamics simulations (datsetid: k0FIIN93TDqcDatfaBWpXQ, not yet published) and recently made a series of somewhat large uploads (not exceeding the 30GB limit, but lots of entries, ~180 per upload, which take a long time to process). Because of this, I worked with @mscheidg to make these uploads. This was done in the following way:
1. I stored my zip files on the nomad mpcdf computer and used the administrator option "local_path" with the "uploads" Post API endpoint.
2. I used a specialized deployment to make the upload at `https://nomad-lab.eu/prod/v1/util/`
3. The maximum allowed uploads for my account was extended
4. After the uploads were successful (all uploads and entries were parsed successfully except for one outlier entry in a single upload), I used the API to edit the Metadata of each upload.
Now, when I try to publish any of the uploads, either via API or in the GUI. I get the following error message: "Process publish_upload failed: IndexError: index out of range". See, e.g., uploadids: tgzOKVOfRNSCmHprfXnOaA, rOL3BcUrS6CdFjtBTN6NEQ.
On the upload page itself, on top of the entries it says: "Processing failed IndexError: index out of range, 180/180 entries processed". Could it be that there is a maximum number of entries per upload? Or could it have to do with my number of non-published uploads exceeding the normal limit?
I am not sure how to obtain any other information about the problem, since all the entries seemed to process without issues, so there is no relevant log information inside the entries. Is there a way to access logs for the upload?Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1598The dev_python image cannot be used as a replacement for the production image2024-01-11T23:35:14ZMarkus ScheidgenThe dev_python image cannot be used as a replacement for the production imageThe dev_python image has at least two differences to the production image that prohibit me to use it in a deployment
- the user is not set to nomad (1000)
- the run.sh script is not available in the default working directory.
Maybe the...The dev_python image has at least two differences to the production image that prohibit me to use it in a deployment
- the user is not set to nomad (1000)
- the run.sh script is not available in the default working directory.
Maybe there is more.
It must be possible to use the dev_python image in real deployments, because for certain debugging things the prod image is too limited.
I didn't noticed the user thing and accidentally screwed up a few file ownerships during the reprocessing.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1597A c library cannot be used in the nomad image2023-07-26T08:57:58ZMarkus ScheidgenA c library cannot be used in the nomad imageThe new soap normalizers uses a package called `quippy-ase`, which contains some gfortran compiled libraries. It imports and tests fine in the `dev_python` image, but when imported in the production image, this error is thrown:
```
Impo...The new soap normalizers uses a package called `quippy-ase`, which contains some gfortran compiled libraries. It imports and tests fine in the `dev_python` image, but when imported in the production image, this error is thrown:
```
ImportError: /usr/local/lib/python3.9/site-packages/quippy/_quippy.cpython-39-x86_64-linux-gnu.so: ELF load command address/offset not page-aligned
```
To reproduce: This works:
```
docker run --rm gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair/dev_python:process python -c 'import quippy'
```
This does not work:
```
docker run --rm gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:process python -c 'import quippy'
```
This happens on the MPCDF nodes. Maybe also in other environments.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1526Mongo and elasticsearch authentication2023-08-24T11:10:58ZMarkus ScheidgenMongo and elasticsearch authenticationWe should allow to use NOMAD with mongo and elastic that have security on.We should allow to use NOMAD with mongo and elastic that have security on.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1492small typo in the jupyterhub config script2023-05-17T11:54:02ZAdam Feketesmall typo in the jupyterhub config scriptBecause !1117 will not be merged there is a small fix to fix the typo.Because !1117 will not be merged there is a small fix to fix the typo.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1452The NORTH page is broken on the latest staging deployment2023-05-12T15:46:31ZAdam FeketeThe NORTH page is broken on the latest staging deploymentIt seems that the the hub and the api works but there is an issue on the gui application.It seems that the the hub and the api works but there is an issue on the gui application.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1415cannot run the pipeline manually2023-12-21T15:40:10ZMohammad Nakhaeecannot run the pipeline manuallyNormally we should be able to run the pipeline manually for the branches that failed because of unexpected server errors. But the rules do not allow.
![Screenshot_from_2023-03-30_13-39-37](/uploads/1587ce4269d10bb6c03851950b9d4978/Scree...Normally we should be able to run the pipeline manually for the branches that failed because of unexpected server errors. But the rules do not allow.
![Screenshot_from_2023-03-30_13-39-37](/uploads/1587ce4269d10bb6c03851950b9d4978/Screenshot_from_2023-03-30_13-39-37.png)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1404Operatin Nomad Oasis, manage Rabbitmq storage2023-12-21T15:38:53ZMichael GötteOperatin Nomad Oasis, manage Rabbitmq storageHey Nomad,
I am running a nomad oasis and for some reason rabbitmq has docker volumes with 33GB, that does not seem to be reasonable to me. If this grows in a similar rate it quickly becomes too big.
Here you see what i mean.
![image](...Hey Nomad,
I am running a nomad oasis and for some reason rabbitmq has docker volumes with 33GB, that does not seem to be reasonable to me. If this grows in a similar rate it quickly becomes too big.
Here you see what i mean.
![image](/uploads/9dd36d90a3520b557c4cf1a0619b84fb/image.png)
Do you have an idea what to do?
Best MichaMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1360North/gui: Allow custom tools.json for deployments2023-05-17T13:17:44ZFlorian DobenerNorth/gui: Allow custom tools.json for deploymentsIt would be good to allow custom tools.json files for deployments. It is possible to mount a custom `tools.json` to change the properties of existing containers. However, the changes are only picked up by jupyter hub and not by the gui a...It would be good to allow custom tools.json files for deployments. It is possible to mount a custom `tools.json` to change the properties of existing containers. However, the changes are only picked up by jupyter hub and not by the gui as the `northTools.json` is compiled into the gui js. Therefore, it's not possible to add or remove containers in a deployment.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1358North fiji tool crashing at stratup2023-02-23T11:08:01ZAdam FeketeNorth fiji tool crashing at stratupAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1357Running nomad-oasis on the root path2023-12-21T15:40:06ZAdam FeketeRunning nomad-oasis on the root pathissues/workarounds:
- [ ] update nginx.conf
- [x] update north's health check
- [ ] check on mounted folder ownerships
- [x] fixing GUI's double leading slashes("//api/v1/north/apmtools")issues/workarounds:
- [ ] update nginx.conf
- [x] update north's health check
- [ ] check on mounted folder ownerships
- [x] fixing GUI's double leading slashes("//api/v1/north/apmtools")Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1350.volumes folder structure is missing from nomad-oasis.zip2023-02-21T12:53:13ZAdam Fekete.volumes folder structure is missing from nomad-oasis.zipAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1347NORTH: the updated tool image has not been updated on the deployment2023-02-21T12:53:53ZAdam FeketeNORTH: the updated tool image has not been updated on the deploymentAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1345NORTH: Websocket connection (webtop based tools) failure on Nginx reverse proxy2023-02-16T15:51:11ZAdam FeketeNORTH: Websocket connection (webtop based tools) failure on Nginx reverse proxyhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1343Issue: unavailable persistent volume claim of north tools in the staging depl...2023-02-16T13:27:45ZAdam FeketeIssue: unavailable persistent volume claim of north tools in the staging deploymentAdam FeketeAdam Fekete