nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2023-12-21T16:06:49Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1261Improve the release process2023-12-21T16:06:49ZMarkus ScheidgenImprove the release process- Don't know how to create an actual release with a NOMAD version that matches the tag. You'll always get a version like `1.1.7.dev0+g2ecdd77f8.d20221223`. How to i create a `1.1.6` image and python package?
- There is no docker tag crea...- Don't know how to create an actual release with a NOMAD version that matches the tag. You'll always get a version like `1.1.7.dev0+g2ecdd77f8.d20221223`. How to i create a `1.1.6` image and python package?
- There is no docker tag created for the gitlab tag
- The tag pipeline is not building an image. So I have to re-run the latest `develop` pipeline to build with the new git generated version
- The integrations test don't work. Probably a file is missing from the image that is used to run the integration test client.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1247K8s doesn't work due to the missing rabbitmq chart2022-12-20T15:06:25ZAdam FeketeK8s doesn't work due to the missing rabbitmq chartThe version (8.30.1) of the rabbitmq chart is not available anymore:
```
Error: can't get a valid version for repositories rabbitmq. Try changing the version constraint in Chart.yaml
```
list of available versions:
```bash
$ helm search...The version (8.30.1) of the rabbitmq chart is not available anymore:
```
Error: can't get a valid version for repositories rabbitmq. Try changing the version constraint in Chart.yaml
```
list of available versions:
```bash
$ helm search repo rabbitmq/rabbitmq -l
NAME CHART VERSION APP VERSION DESCRIPTION
rabbitmq/rabbitmq 11.2.2 3.11.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.2.1 3.11.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.2.0 3.11.4 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.5 3.11.4 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.4 3.11.3 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.3 3.11.3 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.2 3.11.3 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.1 3.11.2 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.1.0 3.11.2 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.4 3.11.2 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.3 3.11.2 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.2 3.11.1 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.1 3.11.1 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 11.0.0 3.11.0 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.9 3.10.8 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.8 3.10.8 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.7 3.10.8 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.6 3.10.8 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.5 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.4 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.3 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.2 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.1 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.3.0 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.2.1 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.2.0 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.19 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.18 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.17 3.10.7 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.16 3.10.6 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.15 3.10.6 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.14 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.13 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.12 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.11 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.10 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.9 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.8 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.7 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.6 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.5 3.10.5 RabbitMQ is an open source general-purpose mess...
rabbitmq/rabbitmq 10.1.4 3.10.5 RabbitMQ is an open source general-purpose mess...
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1230quick fix: initialise all git submodules during setup process2022-12-13T11:26:17ZAdam Feketequick fix: initialise all git submodules during setup processAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1223Add a automatically generated changelog to our git2023-01-18T12:30:48ZMarkus ScheidgenAdd a automatically generated changelog to our gitGitlab supports this via this feature: https://docs.gitlab.com/ee/api/repositories.html#add-changelog-data-to-a-changelog-file
- [x] test how the API works
- [ ] setup the project accordingly
- [ ] add a respective gitlab action, e.g. w...Gitlab supports this via this feature: https://docs.gitlab.com/ee/api/repositories.html#add-changelog-data-to-a-changelog-file
- [x] test how the API works
- [ ] setup the project accordingly
- [ ] add a respective gitlab action, e.g. when generating version tagshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1220Keycloak redirect problem with nomad-oasis behind ssl (traefik) proxy2023-12-21T15:38:48ZAdam FeketeKeycloak redirect problem with nomad-oasis behind ssl (traefik) proxyThe url returns with "502 Bad gateway nginx/1.13.9"
```
https://nomad.eln.data.fau.de/keycloak/auth/realms/nomad/protocol/openid-connect/auth?response_type=code&redirect_uri=http%3A%2F%2Fnomad.eln.data.fau.de%2Fnomad-oasis%2Fnorth%2Fhub%...The url returns with "502 Bad gateway nginx/1.13.9"
```
https://nomad.eln.data.fau.de/keycloak/auth/realms/nomad/protocol/openid-connect/auth?response_type=code&redirect_uri=http%3A%2F%2Fnomad.eln.data.fau.de%2Fnomad-oasis%2Fnorth%2Fhub%2Foauth_callback&client_id=nomad_public&state=eyJzdGF0ZV9pZCI6ICJmOTMxYWIyMWM3NDU0MWUwYTRkNzczYTVmZDU1Y2FiNCIsICJuZXh0X3VybCI6ICIvbm9tYWQtb2FzaXMvbm9ydGgvaHViL2FwaS9vYXV0aDIvYXV0aG9yaXplP2NsaWVudF9pZD1qdXB5dGVyaHViLXVzZXItYWRtaW4tanVweXRlciZyZWRpcmVjdF91cmk9JTJGbm9tYWQtb2FzaXMlMkZub3J0aCUyRnVzZXIlMkZhZG1pbiUyRmp1cHl0ZXIlMkZvYXV0aF9jYWxsYmFjayZyZXNwb25zZV90eXBlPWNvZGUmc3RhdGU9ZXlKMWRXbGtJam9nSW1VME9EVTJOVFU0WkRrNFpUUXhOemc0TkRWaVltUTVNamt6TXpCallXTXlJaXdnSW01bGVIUmZkWEpzSWpvZ0lpOXViMjFoWkMxdllYTnBjeTl1YjNKMGFDOTFjMlZ5TDJGa2JXbHVMMnAxY0hsMFpYSXZkSEpsWlNKOSJ9&scope=openid+profile
```Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1211minor fixes for the pipeline2022-12-02T23:09:04ZAdam Feketeminor fixes for the pipelinehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1207Building docs independently from the python package2022-12-19T10:18:27ZAdam FeketeBuilding docs independently from the python packageThe "compiled" documentation should not be part of the python source distribution. Reasoning:
* we need the nomad package to build the documentation which actually should also contain the documentation... Currently the only way to build...The "compiled" documentation should not be part of the python source distribution. Reasoning:
* we need the nomad package to build the documentation which actually should also contain the documentation... Currently the only way to build the python sdist by building it twice.
* most of the other links on the gui are also pointing to external link
* it is more just a decimation of the usage of the gui...
* the link of the logo of the documentation redirect to a another page (central deployment) and not to the one where it was actually deployed.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1206Switching to src layout2022-12-19T10:17:38ZAdam FeketeSwitching to src layouthttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1186Possible improvements on python packaging2023-12-21T16:06:49ZAdam FeketePossible improvements on python packaging- [ ] using src layout
- [ ] ease (>=) on package dependencies in the submodules and constrain them in the main (~=)
- [ ] python version ?? 3.10 or 3.11 (local oasis)
- [ ] test matrix ?
- [ ] convert parsers into plugins/extensions ...- [ ] using src layout
- [ ] ease (>=) on package dependencies in the submodules and constrain them in the main (~=)
- [ ] python version ?? 3.10 or 3.11 (local oasis)
- [ ] test matrix ?
- [ ] convert parsers into plugins/extensions VS monolithic package (Distribution package VS individual packages)
- [ ] implicit namespaces
```python
/src/nomad_lab/parser
/src/nomad_lab/gui
import nomad_lab.parser
import nomad_lab.gui
```
- [ ] linting: we might ignoring too much warnings (ignore them in the code instead)
alternative: black
- [ ] run `gui` and `jupyterhub` independently
- gui: is an independent application which depends on some artefacts
- [ ] adding `linter` and `pytest` configs to `myproject.toml`
- [ ] examples: storing examples in a separate repository using [git LFS](https://docs.gitlab.com/ee/topics/git/lfs/) for the zipped files like: https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-fair-examples
- [ ] docs: The "compiled" documentation should not be part of the python source distribution. Reasoning:
* we need the nomad package to build the documentation which actually should also contain the documentation... Currently the only way to build the python sdist by building it twice.
* most of the other links on the gui are also pointing to external link
* the link of the logo of the documentation redirect to a another page (central deployment) and not to the one where it was actually deployed.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1184Use CLI to start nomad services in docker-compose2023-12-21T15:38:47ZMarkus ScheidgenUse CLI to start nomad services in docker-composeThe docker-compose using different scripts and commands to start the services (app, worker, north). Instead, it should simply use `python -m nomad.cli admin run ...`. This would give us more control and flexibility to change how the serv...The docker-compose using different scripts and commands to start the services (app, worker, north). Instead, it should simply use `python -m nomad.cli admin run ...`. This would give us more control and flexibility to change how the services are started without needing to update installations. Would also be more consistent with how we run services in development and the bare-metal installation.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1136Deployment is broken due to celery update2022-10-20T10:35:48ZMarkus ScheidgenDeployment is broken due to celery updateMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1133DFT and ELN compatible search config2022-10-20T05:57:45ZMarkus ScheidgenDFT and ELN compatible search configIn our current v1.0/v1.1 and prod/staging situation the entries search results always look bad because of the mix of v1.0 and v1.1 processed data. We should
- make ui configurable through helm templates
- add a specialized "ELN entries...In our current v1.0/v1.1 and prod/staging situation the entries search results always look bad because of the mix of v1.0 and v1.1 processed data. We should
- make ui configurable through helm templates
- add a specialized "ELN entries search"Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1084Mix nomad.yaml based north config with user based hub config2023-12-21T15:38:45ZMarkus ScheidgenMix nomad.yaml based north config with user based hub configWe provide some configuration to JupyterHub based on settings in our `nomad.yaml`. This configuration also connects JupyterHUB to our UI, user-management, mounts volumes, etc.
We should still provide a way for Oasis admins to add more ...We provide some configuration to JupyterHub based on settings in our `nomad.yaml`. This configuration also connects JupyterHUB to our UI, user-management, mounts volumes, etc.
We should still provide a way for Oasis admins to add more configuration to JupyterHUB.
Either of those?
- Maybe two configs can be merged somehow and we allow users to mount their own JupyterHUB config.
- A dict-valued key in `nomad.yaml` that we pass to JupyterHUB via nomad's jupyterconfig.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1079Docker image version tags are wrong2022-09-28T12:48:55ZMarkus ScheidgenDocker image version tags are wrongThis is probably because we use CI_COMMIT_REF_SLUG for all images build from branches or tags. Resulting in v1-1-3 instead of v1.1.3.This is probably because we use CI_COMMIT_REF_SLUG for all images build from branches or tags. Resulting in v1-1-3 instead of v1.1.3.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1048Improve on python setup process2022-09-28T12:52:03ZAdam FeketeImprove on python setup process- [x] using new standard `myproject.toml`
- [x] resolve package conflicts (closes: #934)
- [x] use `pip-compile` for generating the `requirement.txt` file
- [ ] use `setuptool_scm` to generate version automatically
- [ ] documentation ab...- [x] using new standard `myproject.toml`
- [x] resolve package conflicts (closes: #934)
- [x] use `pip-compile` for generating the `requirement.txt` file
- [ ] use `setuptool_scm` to generate version automatically
- [ ] documentation about how to bump the version of packages up
- [ ] regularly rebuild the image from scratch to avoid issues like #937 (--no-cache)Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1017Customize NOMAD installations2023-12-21T15:38:47ZMarkus ScheidgenCustomize NOMAD installationsOasis admins might want to disable some features. This can be archived with filters on
- available search filters
- available schemas (e.g. for eln, but also in the metainfo)
- available parsers
- available examples
- available north too...Oasis admins might want to disable some features. This can be archived with filters on
- available search filters
- available schemas (e.g. for eln, but also in the metainfo)
- available parsers
- available examples
- available north tools
This is pure on an installation or run level and should not require a build or separate images.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1008Plugin mechanism2023-12-21T16:06:49ZMichael GöttePlugin mechanismOasis administrators need to add more functionality to NOMAD without requiring a new image.
NOMAD develops want to separate non core functionality from the nomad-FAIR codebase and NOMAD images.
Potential functionality that can be added...Oasis administrators need to add more functionality to NOMAD without requiring a new image.
NOMAD develops want to separate non core functionality from the nomad-FAIR codebase and NOMAD images.
Potential functionality that can be added via plugins:
- schemas
- parsers
- normalisers
- APIs
- CLI commands
--
The original report by Micha: *Loading individual ELN code into an existing docker image*
Hey all,
this is a feature request. Imagine you want to use a fresh and tested docker image from your gitlab registry but you are not happy with the ELN normalizers in that image. Is it possible to develop and commit the ELN part in a different git project and then load this into the image of the registry?
An idea would be to do this in the ```docker-compose```file, e.g. add:
```
volumes:
- ./configs/nomad.yaml:/app/nomad.yaml
- ./eln/:/app/nomad/datamodel/metainfo/eln/
```
I will test this and let you know!
Best Michahttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1005Fine-tuning develop and dev deployments2022-09-12T14:56:28ZAdam FeketeFine-tuning develop and dev deploymentshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/995Mail delivery failed: returning message to sender2023-12-21T15:39:05ZAdam FeketeMail delivery failed: returning message to senderit looks like all the emails that should got to the specific users ends up in support@nomad-lab.hu (and in my spam folder)
For example:
```
Mail Delivery System <Mailer-Daemon@www153.your-server.de>
A message that you sent could not b...it looks like all the emails that should got to the specific users ends up in support@nomad-lab.hu (and in my spam folder)
For example:
```
Mail Delivery System <Mailer-Daemon@www153.your-server.de>
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
lauri.himanen@gmail.com
(ultimately generated from support@nomad-lab.eu)
host gmail-smtp-in.l.google.com [142.251.5.27]
SMTP error from remote mail server after pipelined end of data:
550-5.7.1 [213.133.104.153] Messages missing a valid messageId header are not
550 5.7.1 accepted. t1-20020adfe101000000b0021f157ae467si6385629wrz.115 - gsmtp
---------- Forwarded message ----------
From: support@nomad-lab.eu
To: Lauri Himanen
Cc: The nomad team <support@nomad-lab.eu>
Bcc:
Date:
Subject: Processing completed
Dear Lauri Himanen,
your data uploaded at 2022-08-23T11:36:32.722000 has completed processing. You can review your data on your upload page: https://nomad-lab.eu/prod/v1/staging/gui/uploads
If you encounter any issues with your upload, please let us know and reply to this email.
The nomad team
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/991Random failing issues during the `gui tests`2022-09-13T08:18:04ZAdam FeketeRandom failing issues during the `gui tests`Quite regularly the gui test fails. As a workaround, we can restart the job and hope it will pass.
Here is the summary of a [failed test](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/1799899):
```
Summary of all failing tests
...Quite regularly the gui test fails. As a workaround, we can restart the job and hope it will pass.
Here is the summary of a [failed test](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/1799899):
```
Summary of all failing tests
FAIL src/components/uploads/UploadPage.spec.js (354.586 s)
● Delete selected entries from table
thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."
252 | })
253 |
> 254 | test('Delete selected entries from table', async () => {
| ^
255 | await startAPI('tests.states.uploads.multiple_entries', 'tests/data/uploads/delete_entries_from_table', 'test', 'password')
256 | render(<UploadPage uploadId="dft_upload_1"/>)
257 |
at Object.<anonymous> (src/components/uploads/UploadPage.spec.js:254:1)
Test Suites: 1 failed, 25 passed, 26 total
Tests: 1 failed, 285 passed, 286 total
Snapshots: 0 total
Time: 2822.837 s
error Command failed with exit code 1.
```
There is another error message earlier in the log which might be the source of the actual problem:
```
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
```
Note: The tests' running time (54 minutes 1 second) is actually very close to the hard timeout limit (1h)Adam FeketeAdam Fekete