nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2023-12-21T15:38:43Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/989Improvement to docker-compose to support dev work2023-12-21T15:38:43ZSherjeel ShabihImprovement to docker-compose to support dev workWe would like to have a docker-compose file that spins up needed docker containers and mounts our Nomad dev folder from the host machine into these containers. This means we run the local version of our Nomad source code rather than the ...We would like to have a docker-compose file that spins up needed docker containers and mounts our Nomad dev folder from the host machine into these containers. This means we run the local version of our Nomad source code rather than the one compiled into the container image.
This allows us to have almost the same dev environment as a production Nomad Oasis install.
Here are some recommendations or ideas we have already discussed in Area B:
- Make use of the docker-compose from ops/docker-compose/nomad-oasis
- Mount the dev folder into all containers that are not just infrastructure i.e. the app, hub, gui, etc.
- Run yarn in hot reload on the gui folder instead of using the production build for the gui container
- Maybe it is a good idea to check if we need to change/add some helper scripts to reload things like generate_gui_artifacts.sh. We would guess for scripts that just create new files on disk this wouldn't be an issue.
@dflor @sanbrock @kuehbachm @ecarohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1685Getting complaint in installation of nomad with python3.102023-12-21T15:38:52ZRubel MozumderGetting complaint in installation of nomad with python3.10In the dev environment installation process having a complaint from mdtraj package while `python version 3.10`:
```
File "/home/rubel/NOMAD-FAIRmat/GL/nomad-FAIR/.NF_1674_xrd/lib/python3.10/site-packages/pip/_vendor/pyproject_hoo...In the dev environment installation process having a complaint from mdtraj package while `python version 3.10`:
```
File "/home/rubel/NOMAD-FAIRmat/GL/nomad-FAIR/.NF_1674_xrd/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
main()
File "/home/rubel/NOMAD-FAIRmat/GL/nomad-FAIR/.NF_1674_xrd/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/rubel/NOMAD-FAIRmat/GL/nomad-FAIR/.NF_1674_xrd/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
return hook(metadata_directory, config_settings)
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 396, in prepare_metadata_for_build_wheel
self.run_setup()
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 507, in run_setup
super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup
exec(code, locals())
File "<string>", line 301, in <module>
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
cythonize_one(*args)
File "/tmp/pip-build-env-ghmqe1s6/overlay/lib/python3.10/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: mdtraj/geometry/src/_geometry.pyx
```
But no issues are found with `python 3.9`.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1005Fine-tuning develop and dev deployments2022-09-12T14:56:28ZAdam FeketeFine-tuning develop and dev deploymentshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/402Enforcing HTTPS across all services2023-12-21T15:40:02ZLauri HimanenEnforcing HTTPS across all servicesCurrently many of the resources that our production machine serves can be retrieved both with http or https protocols.
As far as I understand, the standard these days is to always use https, no matter what the resource is. This ensures ...Currently many of the resources that our production machine serves can be retrieved both with http or https protocols.
As far as I understand, the standard these days is to always use https, no matter what the resource is. This ensures that all outgoing data is always properly secured (we don't have to selectively enable https, as that is prone to mistakes), and that there will be no issue with resources interacting with different protocols.
Doing this should be relative easy: we will still accept incoming requests through http, but these requests will always be redirected to https. This can be done through our nginx server, [with something like this.](https://serversforhackers.com/c/redirect-http-to-https-nginx) You can see that most sites do something similar, e.g. if you try to load this issue page with http, Gitlab will automatically switch to https.
What do you think?https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1079Docker image version tags are wrong2022-09-28T12:48:55ZMarkus ScheidgenDocker image version tags are wrongThis is probably because we use CI_COMMIT_REF_SLUG for all images build from branches or tags. Resulting in v1-1-3 instead of v1.1.3.This is probably because we use CI_COMMIT_REF_SLUG for all images build from branches or tags. Resulting in v1-1-3 instead of v1.1.3.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1133DFT and ELN compatible search config2022-10-20T05:57:45ZMarkus ScheidgenDFT and ELN compatible search configIn our current v1.0/v1.1 and prod/staging situation the entries search results always look bad because of the mix of v1.0 and v1.1 processed data. We should
- make ui configurable through helm templates
- add a specialized "ELN entries...In our current v1.0/v1.1 and prod/staging situation the entries search results always look bad because of the mix of v1.0 and v1.1 processed data. We should
- make ui configurable through helm templates
- add a specialized "ELN entries search"Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1136Deployment is broken due to celery update2022-10-20T10:35:48ZMarkus ScheidgenDeployment is broken due to celery updateMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1017Customize NOMAD installations2023-12-21T15:38:47ZMarkus ScheidgenCustomize NOMAD installationsOasis admins might want to disable some features. This can be archived with filters on
- available search filters
- available schemas (e.g. for eln, but also in the metainfo)
- available parsers
- available examples
- available north too...Oasis admins might want to disable some features. This can be archived with filters on
- available search filters
- available schemas (e.g. for eln, but also in the metainfo)
- available parsers
- available examples
- available north tools
This is pure on an installation or run level and should not require a build or separate images.https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1944Config does not allow string loglevel like "INFO" anymore2024-03-19T09:37:50ZMarkus ScheidgenConfig does not allow string loglevel like "INFO" anymoreMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1415cannot run the pipeline manually2023-12-21T15:40:10ZMohammad Nakhaeecannot run the pipeline manuallyNormally we should be able to run the pipeline manually for the branches that failed because of unexpected server errors. But the rules do not allow.
![Screenshot_from_2023-03-30_13-39-37](/uploads/1587ce4269d10bb6c03851950b9d4978/Scree...Normally we should be able to run the pipeline manually for the branches that failed because of unexpected server errors. But the rules do not allow.
![Screenshot_from_2023-03-30_13-39-37](/uploads/1587ce4269d10bb6c03851950b9d4978/Screenshot_from_2023-03-30_13-39-37.png)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1207Building docs independently from the python package2022-12-19T10:18:27ZAdam FeketeBuilding docs independently from the python packageThe "compiled" documentation should not be part of the python source distribution. Reasoning:
* we need the nomad package to build the documentation which actually should also contain the documentation... Currently the only way to build...The "compiled" documentation should not be part of the python source distribution. Reasoning:
* we need the nomad package to build the documentation which actually should also contain the documentation... Currently the only way to build the python sdist by building it twice.
* most of the other links on the gui are also pointing to external link
* it is more just a decimation of the usage of the gui...
* the link of the logo of the documentation redirect to a another page (central deployment) and not to the one where it was actually deployed.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/628Better user-magement2023-12-21T15:55:38ZMarkus ScheidgenBetter user-magement- upgrade keycloak (to version 8?), keeping our custom interface
- connect via OAuth (e.g. with github, google)
- connect via SAML (e.g. with DFN AAI)
- allow local keycloaks for Oasis (why?)
- common user id (e.g. ORCID)
- consolidate u...- upgrade keycloak (to version 8?), keeping our custom interface
- connect via OAuth (e.g. with github, google)
- connect via SAML (e.g. with DFN AAI)
- allow local keycloaks for Oasis (why?)
- common user id (e.g. ORCID)
- consolidate users (e.g. automatically based on ORCID, CLI functions to migrate users)
- properly use (or don't use at all) affiliations
- allow authors that are not usershttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1704App is always running just one worker2023-10-04T07:26:19ZMarkus ScheidgenApp is always running just one worker- the `--workers` setting to the `nomad run app` command does not work, because it runs the uvicorn dev server
- we are not running a proper process manager like gunicorn (as recommended by uvicorn)
Consequently all oasis and our produc...- the `--workers` setting to the `nomad run app` command does not work, because it runs the uvicorn dev server
- we are not running a proper process manager like gunicorn (as recommended by uvicorn)
Consequently all oasis and our production system is only running one process. With some loads timeouts and restarts are unavoidable.Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1223Add a automatically generated changelog to our git2023-01-18T12:30:48ZMarkus ScheidgenAdd a automatically generated changelog to our gitGitlab supports this via this feature: https://docs.gitlab.com/ee/api/repositories.html#add-changelog-data-to-a-changelog-file
- [x] test how the API works
- [ ] setup the project accordingly
- [ ] add a respective gitlab action, e.g. w...Gitlab supports this via this feature: https://docs.gitlab.com/ee/api/repositories.html#add-changelog-data-to-a-changelog-file
- [x] test how the API works
- [ ] setup the project accordingly
- [ ] add a respective gitlab action, e.g. when generating version tagshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1597A c library cannot be used in the nomad image2023-07-26T08:57:58ZMarkus ScheidgenA c library cannot be used in the nomad imageThe new soap normalizers uses a package called `quippy-ase`, which contains some gfortran compiled libraries. It imports and tests fine in the `dev_python` image, but when imported in the production image, this error is thrown:
```
Impo...The new soap normalizers uses a package called `quippy-ase`, which contains some gfortran compiled libraries. It imports and tests fine in the `dev_python` image, but when imported in the production image, this error is thrown:
```
ImportError: /usr/local/lib/python3.9/site-packages/quippy/_quippy.cpython-39-x86_64-linux-gnu.so: ELF load command address/offset not page-aligned
```
To reproduce: This works:
```
docker run --rm gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair/dev_python:process python -c 'import quippy'
```
This does not work:
```
docker run --rm gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:process python -c 'import quippy'
```
This happens on the MPCDF nodes. Maybe also in other environments.Adam FeketeAdam Fekete