nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2022-12-19T10:17:38Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1206Switching to src layout2022-12-19T10:17:38ZAdam FeketeSwitching to src layouthttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1230quick fix: initialise all git submodules during setup process2022-12-13T11:26:17ZAdam Feketequick fix: initialise all git submodules during setup processAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/905Kubernetes: switching to namespaces instead of using prefixes2022-12-10T00:44:47ZAdam FeketeKubernetes: switching to namespaces instead of using prefixesThe goal is to use namespaces to separate all the dev deployments from each other. Right now there is a single `nomad` namespace and all the pods/services are using a prefix to avoid any conflict.
Benefits:
- slightly more reasonable te...The goal is to use namespaces to separate all the dev deployments from each other. Right now there is a single `nomad` namespace and all the pods/services are using a prefix to avoid any conflict.
Benefits:
- slightly more reasonable testing environment
- different deployments cannot affect each other
- simplified pod/service names
- possibility to use the same release name
- in the case of a single namespace the explosion of the Pod Information to Containers Through Environment Variables could be dangerous
Disadvantage:
- the explosion of namespaces (cleanup is slightly more tricky...)
Known issues:
- `Error: secret “nomad-keycloak-password” not found`: Right now the secret has been defined manually for a specific namespaceAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1211minor fixes for the pipeline2022-12-02T23:09:04ZAdam Feketeminor fixes for the pipelinehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/906Improve on the build process of the docker image2022-11-29T16:38:33ZAdam FeketeImprove on the build process of the docker image1. [x] caching layers for multi-stage build
2. [x] The build process of the docker image can be accelerated by caching layers for multi-stage build:
https://testdriven.io/blog/faster-ci-builds-with-docker-cache/
TLDR: Each stage n...1. [x] caching layers for multi-stage build
2. [x] The build process of the docker image can be accelerated by caching layers for multi-stage build:
https://testdriven.io/blog/faster-ci-builds-with-docker-cache/
TLDR: Each stage needs to be built (using `--target ...`) and pushed separately.
2. [x] The building of "node" stage should be independent of the "python" stage.
3. [x] Many steps can be grouped together to avoid a lot of tiny layers
4. [x] installing python packages without versioning is useless because the setup.py script will uninstall most of them and replace them with a versioned package from the requirements.txt
5. [x] As far as I know there are more modern ways to handle requirements (citation needed)
6. [x] ~~mismatch in node versions (build:14.8 vs final 16.)~~
7. [x] ~~Consider (no hard feelings about this :smile: ) building separate images (still using the single Dockerfile but with multiple targets) for separate services (GUI, worker, north).~~
8. [x] ~~Inconsistent package versions during installation (closes: #934)~~
9. [x] Creating of a docker image for the development as mentioned in https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/989Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1136Deployment is broken due to celery update2022-10-20T10:35:48ZMarkus ScheidgenDeployment is broken due to celery updateMarkus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1133DFT and ELN compatible search config2022-10-20T05:57:45ZMarkus ScheidgenDFT and ELN compatible search configIn our current v1.0/v1.1 and prod/staging situation the entries search results always look bad because of the mix of v1.0 and v1.1 processed data. We should
- make ui configurable through helm templates
- add a specialized "ELN entries...In our current v1.0/v1.1 and prod/staging situation the entries search results always look bad because of the mix of v1.0 and v1.1 processed data. We should
- make ui configurable through helm templates
- add a specialized "ELN entries search"Markus ScheidgenMarkus Scheidgenhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1048Improve on python setup process2022-09-28T12:52:03ZAdam FeketeImprove on python setup process- [x] using new standard `myproject.toml`
- [x] resolve package conflicts (closes: #934)
- [x] use `pip-compile` for generating the `requirement.txt` file
- [ ] use `setuptool_scm` to generate version automatically
- [ ] documentation ab...- [x] using new standard `myproject.toml`
- [x] resolve package conflicts (closes: #934)
- [x] use `pip-compile` for generating the `requirement.txt` file
- [ ] use `setuptool_scm` to generate version automatically
- [ ] documentation about how to bump the version of packages up
- [ ] regularly rebuild the image from scratch to avoid issues like #937 (--no-cache)Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1079Docker image version tags are wrong2022-09-28T12:48:55ZMarkus ScheidgenDocker image version tags are wrongThis is probably because we use CI_COMMIT_REF_SLUG for all images build from branches or tags. Resulting in v1-1-3 instead of v1.1.3.This is probably because we use CI_COMMIT_REF_SLUG for all images build from branches or tags. Resulting in v1-1-3 instead of v1.1.3.Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/969Testing data compatibility with metainfo schema2022-09-28T07:33:47ZLauri HimanenTesting data compatibility with metainfo schemaWe would like to add some form of testing how the production data complies with the current metainfo schema. This will give some important insights into how well the real data complies with our schema and lets us examine any compatibili...We would like to add some form of testing how the production data complies with the current metainfo schema. This will give some important insights into how well the real data complies with our schema and lets us examine any compatibility problems.
I would envision this as a manually triggered CI task. Would be written in python since we can easily load the data with python libs, and also perform the schema tests easily by loading the metainfo.Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/934Inconsistent package versions during installation2022-09-22T08:23:48ZAdam FeketeInconsistent package versions during installationRunning `setup.sh` more precisely `dependencies.sh` changes the version of some of the python packages. In the end, you have an environment you have different packages that you "required" the first time. Note: as long as there is no brak...Running `setup.sh` more precisely `dependencies.sh` changes the version of some of the python packages. In the end, you have an environment you have different packages that you "required" the first time. Note: as long as there is no braking changes in the packages themself it is just an inconsistency...
Inconsistent versions in (and probably in more):
- dependencies/parsers/nexus/requirements.txt
- dependencies/parsers/workflow/requirements.txthttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/908Sporadically failing python-tests during CI2022-09-15T09:05:27ZMarkus ScheidgenSporadically failing python-tests during CI
This is a list of test cases, where we observed that they failed or errored, but the problem is gone when simply repeating the tests.
- `tests/app/v1/routers/test_uploads.py::test_post_upload[invalid-credentials]`
- `tests/app/v1/route...
This is a list of test cases, where we observed that they failed or errored, but the problem is gone when simply repeating the tests.
- `tests/app/v1/routers/test_uploads.py::test_post_upload[invalid-credentials]`
- `tests/app/v1/routers/test_uploads.py::test_put_upload_raw_path[zip-to-subfolder]`
- `tests/app/v1/routers/test_uploads.py::test_post_upload[stream-no-embargo]`
- `tests/app/v1/routers/test_uploads.py::test_editing_raw_file[conflict_in_concurrent_editing]`
These are probably race-conditions with a singular cause.David SikterDavid Sikterhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/935Improve on gitlab ci/cd script2022-09-13T08:21:03ZAdam FeketeImprove on gitlab ci/cd script- [x] define environments for `staging` (aka `develop`) similarly to `dev`
- [x] checking the rules because some of them does not make sense anymore (like: `$CI_COMMIT_REF_NAME =~ /^dev-.*$/`)
- [ ] checking the way how initialisation of...- [x] define environments for `staging` (aka `develop`) similarly to `dev`
- [x] checking the rules because some of them does not make sense anymore (like: `$CI_COMMIT_REF_NAME =~ /^dev-.*$/`)
- [ ] checking the way how initialisation of submodules works
- [x] get rid of docker login waning (`echo $CI_BUILD_TOKEN | docker login --username foo --password-stdin`)
- [x] cleanup (eg. using `before_script` more often)
- [ ] regularly rebuild the image from scratch to avoid issues like #937 (--no-cache)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/991Random failing issues during the `gui tests`2022-09-13T08:18:04ZAdam FeketeRandom failing issues during the `gui tests`Quite regularly the gui test fails. As a workaround, we can restart the job and hope it will pass.
Here is the summary of a [failed test](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/1799899):
```
Summary of all failing tests
...Quite regularly the gui test fails. As a workaround, we can restart the job and hope it will pass.
Here is the summary of a [failed test](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/1799899):
```
Summary of all failing tests
FAIL src/components/uploads/UploadPage.spec.js (354.586 s)
● Delete selected entries from table
thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."
252 | })
253 |
> 254 | test('Delete selected entries from table', async () => {
| ^
255 | await startAPI('tests.states.uploads.multiple_entries', 'tests/data/uploads/delete_entries_from_table', 'test', 'password')
256 | render(<UploadPage uploadId="dft_upload_1"/>)
257 |
at Object.<anonymous> (src/components/uploads/UploadPage.spec.js:254:1)
Test Suites: 1 failed, 25 passed, 26 total
Tests: 1 failed, 285 passed, 286 total
Snapshots: 0 total
Time: 2822.837 s
error Command failed with exit code 1.
```
There is another error message earlier in the log which might be the source of the actual problem:
```
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
```
Note: The tests' running time (54 minutes 1 second) is actually very close to the hard timeout limit (1h)Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1005Fine-tuning develop and dev deployments2022-09-12T14:56:28ZAdam FeketeFine-tuning develop and dev deployments