nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2023-12-21T15:38:47Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1184Use CLI to start nomad services in docker-compose2023-12-21T15:38:47ZMarkus ScheidgenUse CLI to start nomad services in docker-composeThe docker-compose using different scripts and commands to start the services (app, worker, north). Instead, it should simply use `python -m nomad.cli admin run ...`. This would give us more control and flexibility to change how the serv...The docker-compose using different scripts and commands to start the services (app, worker, north). Instead, it should simply use `python -m nomad.cli admin run ...`. This would give us more control and flexibility to change how the services are started without needing to update installations. Would also be more consistent with how we run services in development and the bare-metal installation.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1084Mix nomad.yaml based north config with user based hub config2023-12-21T15:38:45ZMarkus ScheidgenMix nomad.yaml based north config with user based hub configWe provide some configuration to JupyterHub based on settings in our `nomad.yaml`. This configuration also connects JupyterHUB to our UI, user-management, mounts volumes, etc.
We should still provide a way for Oasis admins to add more ...We provide some configuration to JupyterHub based on settings in our `nomad.yaml`. This configuration also connects JupyterHUB to our UI, user-management, mounts volumes, etc.
We should still provide a way for Oasis admins to add more configuration to JupyterHUB.
Either of those?
- Maybe two configs can be merged somehow and we allow users to mount their own JupyterHUB config.
- A dict-valued key in `nomad.yaml` that we pass to JupyterHUB via nomad's jupyterconfig.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/989Improvement to docker-compose to support dev work2023-12-21T15:38:43ZSherjeel ShabihImprovement to docker-compose to support dev workWe would like to have a docker-compose file that spins up needed docker containers and mounts our Nomad dev folder from the host machine into these containers. This means we run the local version of our Nomad source code rather than the ...We would like to have a docker-compose file that spins up needed docker containers and mounts our Nomad dev folder from the host machine into these containers. This means we run the local version of our Nomad source code rather than the one compiled into the container image.
This allows us to have almost the same dev environment as a production Nomad Oasis install.
Here are some recommendations or ideas we have already discussed in Area B:
- Make use of the docker-compose from ops/docker-compose/nomad-oasis
- Mount the dev folder into all containers that are not just infrastructure i.e. the app, hub, gui, etc.
- Run yarn in hot reload on the gui folder instead of using the production build for the gui container
- Maybe it is a good idea to check if we need to change/add some helper scripts to reload things like generate_gui_artifacts.sh. We would guess for scripts that just create new files on disk this wouldn't be an issue.
@dflor @sanbrock @kuehbachm @ecarohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/634Refactor config "api" keys2023-12-21T15:38:38ZMarkus ScheidgenRefactor config "api" keysThe configuration of API related keys is very confusing. The path is always the same, but host/port have three different meanings that need to be supported by three different keys:
- to use by nomad client
- to return for external use
- ...The configuration of API related keys is very confusing. The path is always the same, but host/port have three different meanings that need to be supported by three different keys:
- to use by nomad client
- to return for external use
- to access the API via the internal server side network between different app container
Another thing is GUI vs API (which makes a difference in dev mode).
Similar internal vs external distinction is necessary for files (inside container, outside container).https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1704App is always running just one worker2023-10-04T07:26:19ZMarkus ScheidgenApp is always running just one worker- the `--workers` setting to the `nomad run app` command does not work, because it runs the uvicorn dev server
- we are not running a proper process manager like gunicorn (as recommended by uvicorn)
Consequently all oasis and our produc...- the `--workers` setting to the `nomad run app` command does not work, because it runs the uvicorn dev server
- we are not running a proper process manager like gunicorn (as recommended by uvicorn)
Consequently all oasis and our production system is only running one process. With some loads timeouts and restarts are unavoidable.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1526Mongo and elasticsearch authentication2023-08-24T11:10:58ZMarkus ScheidgenMongo and elasticsearch authenticationWe should allow to use NOMAD with mongo and elastic that have security on.We should allow to use NOMAD with mongo and elastic that have security on.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1631Publishing fails due to IndexError2023-08-16T11:40:26ZJoseph RudzinskiPublishing fails due to IndexErrorI am working to publish a dataset of molecular dynamics simulations (datsetid: k0FIIN93TDqcDatfaBWpXQ, not yet published) and recently made a series of somewhat large uploads (not exceeding the 30GB limit, but lots of entries, ~180 per u...I am working to publish a dataset of molecular dynamics simulations (datsetid: k0FIIN93TDqcDatfaBWpXQ, not yet published) and recently made a series of somewhat large uploads (not exceeding the 30GB limit, but lots of entries, ~180 per upload, which take a long time to process). Because of this, I worked with @mscheidg to make these uploads. This was done in the following way:
1. I stored my zip files on the nomad mpcdf computer and used the administrator option "local_path" with the "uploads" Post API endpoint.
2. I used a specialized deployment to make the upload at `https://nomad-lab.eu/prod/v1/util/`
3. The maximum allowed uploads for my account was extended
4. After the uploads were successful (all uploads and entries were parsed successfully except for one outlier entry in a single upload), I used the API to edit the Metadata of each upload.
Now, when I try to publish any of the uploads, either via API or in the GUI. I get the following error message: "Process publish_upload failed: IndexError: index out of range". See, e.g., uploadids: tgzOKVOfRNSCmHprfXnOaA, rOL3BcUrS6CdFjtBTN6NEQ.
On the upload page itself, on top of the entries it says: "Processing failed IndexError: index out of range, 180/180 entries processed". Could it be that there is a maximum number of entries per upload? Or could it have to do with my number of non-published uploads exceeding the normal limit?
I am not sure how to obtain any other information about the problem, since all the entries seemed to process without issues, so there is no relevant log information inside the entries. Is there a way to access logs for the upload?Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1597A c library cannot be used in the nomad image2023-07-26T08:57:58ZMarkus ScheidgenA c library cannot be used in the nomad imageThe new soap normalizers uses a package called `quippy-ase`, which contains some gfortran compiled libraries. It imports and tests fine in the `dev_python` image, but when imported in the production image, this error is thrown:
```
Impo...The new soap normalizers uses a package called `quippy-ase`, which contains some gfortran compiled libraries. It imports and tests fine in the `dev_python` image, but when imported in the production image, this error is thrown:
```
ImportError: /usr/local/lib/python3.9/site-packages/quippy/_quippy.cpython-39-x86_64-linux-gnu.so: ELF load command address/offset not page-aligned
```
To reproduce: This works:
```
docker run --rm gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair/dev_python:process python -c 'import quippy'
```
This does not work:
```
docker run --rm gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:process python -c 'import quippy'
```
This happens on the MPCDF nodes. Maybe also in other environments.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1360North/gui: Allow custom tools.json for deployments2023-05-17T13:17:44ZFlorian DobenerNorth/gui: Allow custom tools.json for deploymentsIt would be good to allow custom tools.json files for deployments. It is possible to mount a custom `tools.json` to change the properties of existing containers. However, the changes are only picked up by jupyter hub and not by the gui a...It would be good to allow custom tools.json files for deployments. It is possible to mount a custom `tools.json` to change the properties of existing containers. However, the changes are only picked up by jupyter hub and not by the gui as the `northTools.json` is compiled into the gui js. Therefore, it's not possible to add or remove containers in a deployment.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1492small typo in the jupyterhub config script2023-05-17T11:54:02ZAdam Feketesmall typo in the jupyterhub config scriptBecause !1117 will not be merged there is a small fix to fix the typo.Because !1117 will not be merged there is a small fix to fix the typo.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1452The NORTH page is broken on the latest staging deployment2023-05-12T15:46:31ZAdam FeketeThe NORTH page is broken on the latest staging deploymentIt seems that the the hub and the api works but there is an issue on the gui application.It seems that the the hub and the api works but there is an issue on the gui application.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1358North fiji tool crashing at stratup2023-02-23T11:08:01ZAdam FeketeNorth fiji tool crashing at stratupAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1347NORTH: the updated tool image has not been updated on the deployment2023-02-21T12:53:53ZAdam FeketeNORTH: the updated tool image has not been updated on the deploymentAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1350.volumes folder structure is missing from nomad-oasis.zip2023-02-21T12:53:13ZAdam Fekete.volumes folder structure is missing from nomad-oasis.zipAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1345NORTH: Websocket connection (webtop based tools) failure on Nginx reverse proxy2023-02-16T15:51:11ZAdam FeketeNORTH: Websocket connection (webtop based tools) failure on Nginx reverse proxyhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1343Issue: unavailable persistent volume claim of north tools in the staging depl...2023-02-16T13:27:45ZAdam FeketeIssue: unavailable persistent volume claim of north tools in the staging deploymentAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1223Add a automatically generated changelog to our git2023-01-18T12:30:48ZMarkus ScheidgenAdd a automatically generated changelog to our gitGitlab supports this via this feature: https://docs.gitlab.com/ee/api/repositories.html#add-changelog-data-to-a-changelog-file
- [x] test how the API works
- [ ] setup the project accordingly
- [ ] add a respective gitlab action, e.g. w...Gitlab supports this via this feature: https://docs.gitlab.com/ee/api/repositories.html#add-changelog-data-to-a-changelog-file
- [x] test how the API works
- [ ] setup the project accordingly
- [ ] add a respective gitlab action, e.g. when generating version tagshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1247K8s doesn't work due to the missing rabbitmq chart2022-12-20T15:06:25ZAdam FeketeK8s doesn't work due to the missing rabbitmq chartThe version (8.30.1) of the rabbitmq chart is not available anymore:
```
Error: can't get a valid version for repositories rabbitmq. Try changing the version constraint in Chart.yaml
```
list of available versions:
```bash
$ helm search...The version (8.30.1) of the rabbitmq chart is not available anymore:
```
Error: can't get a valid version for repositories rabbitmq. Try changing the version constraint in Chart.yaml
```
list of available versions:
```bash
$ helm search repo rabbitmq/rabbitmq -l
NAME CHART VERSION APP VERSION DESCRIPTION
rabbitmq/rabbitmq 11.2.2 3.11.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.2.1 3.11.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.2.0 3.11.4 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.5 3.11.4 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.4 3.11.3 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.3 3.11.3 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.2 3.11.3 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.1 3.11.2 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.0 3.11.2 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.4 3.11.2 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.3 3.11.2 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.2 3.11.1 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.1 3.11.1 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.0 3.11.0 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.9 3.10.8 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.8 3.10.8 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.7 3.10.8 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.6 3.10.8 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.5 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.4 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.3 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.2 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.1 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.0 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.2.1 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.2.0 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.19 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.18 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.17 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.16 3.10.6 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.15 3.10.6 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.14 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.13 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.12 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.11 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.10 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.9 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.8 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.7 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.6 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.5 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.4 3.10.5 RabbitMQ is an open source general-purpose mess...
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/856Revise version information2022-12-19T11:42:42ZMarkus ScheidgenRevise version information- The current NOMAD version is stated a many places throughout the source code. This is annoying to change.
- The documentation has no version statement
- The GUI could have a more prominent statement
- The API does not give any metadata...- The current NOMAD version is stated a many places throughout the source code. This is annoying to change.
- The documentation has no version statement
- The GUI could have a more prominent statement
- The API does not give any metadata with its responsesAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1207Building docs independently from the python package2022-12-19T10:18:27ZAdam FeketeBuilding docs independently from the python packageThe "compiled" documentation should not be part of the python source distribution. Reasoning:
* we need the nomad package to build the documentation which actually should also contain the documentation... Currently the only way to build...The "compiled" documentation should not be part of the python source distribution. Reasoning:
* we need the nomad package to build the documentation which actually should also contain the documentation... Currently the only way to build the python sdist by building it twice.
* most of the other links on the gui are also pointing to external link
* it is more just a decimation of the usage of the gui...
* the link of the logo of the documentation redirect to a another page (central deployment) and not to the one where it was actually deployed.Adam FeketeAdam Fekete