nomad-FAIR issueshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues2022-09-12T14:56:28Zhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/1005Fine-tuning develop and dev deployments2022-09-12T14:56:28ZAdam FeketeFine-tuning develop and dev deploymentshttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/995Mail delivery failed: returning message to sender2023-12-21T15:39:05ZAdam FeketeMail delivery failed: returning message to senderit looks like all the emails that should got to the specific users ends up in support@nomad-lab.hu (and in my spam folder)
For example:
```
Mail Delivery System <Mailer-Daemon@www153.your-server.de>
A message that you sent could not b...it looks like all the emails that should got to the specific users ends up in support@nomad-lab.hu (and in my spam folder)
For example:
```
Mail Delivery System <Mailer-Daemon@www153.your-server.de>
A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:
lauri.himanen@gmail.com
(ultimately generated from support@nomad-lab.eu)
host gmail-smtp-in.l.google.com [142.251.5.27]
SMTP error from remote mail server after pipelined end of data:
550-5.7.1 [213.133.104.153] Messages missing a valid messageId header are not
550 5.7.1 accepted. t1-20020adfe101000000b0021f157ae467si6385629wrz.115 - gsmtp
---------- Forwarded message ----------
From: support@nomad-lab.eu
To: Lauri Himanen
Cc: The nomad team <support@nomad-lab.eu>
Bcc:
Date:
Subject: Processing completed
Dear Lauri Himanen,
your data uploaded at 2022-08-23T11:36:32.722000 has completed processing. You can review your data on your upload page: https://nomad-lab.eu/prod/v1/staging/gui/uploads
If you encounter any issues with your upload, please let us know and reply to this email.
The nomad team
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/991Random failing issues during the `gui tests`2022-09-13T08:18:04ZAdam FeketeRandom failing issues during the `gui tests`Quite regularly the gui test fails. As a workaround, we can restart the job and hope it will pass.
Here is the summary of a [failed test](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/1799899):
```
Summary of all failing tests
...Quite regularly the gui test fails. As a workaround, we can restart the job and hope it will pass.
Here is the summary of a [failed test](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/1799899):
```
Summary of all failing tests
FAIL src/components/uploads/UploadPage.spec.js (354.586 s)
● Delete selected entries from table
thrown: "Exceeded timeout of 120000 ms for a test.
Use jest.setTimeout(newTimeout) to increase the timeout value, if this is a long-running test."
252 | })
253 |
> 254 | test('Delete selected entries from table', async () => {
| ^
255 | await startAPI('tests.states.uploads.multiple_entries', 'tests/data/uploads/delete_entries_from_table', 'test', 'password')
256 | render(<UploadPage uploadId="dft_upload_1"/>)
257 |
at Object.<anonymous> (src/components/uploads/UploadPage.spec.js:254:1)
Test Suites: 1 failed, 25 passed, 26 total
Tests: 1 failed, 285 passed, 286 total
Snapshots: 0 total
Time: 2822.837 s
error Command failed with exit code 1.
```
There is another error message earlier in the log which might be the source of the actual problem:
```
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
```
Note: The tests' running time (54 minutes 1 second) is actually very close to the hard timeout limit (1h)Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/989Improvement to docker-compose to support dev work2023-12-21T15:38:43ZSherjeel ShabihImprovement to docker-compose to support dev workWe would like to have a docker-compose file that spins up needed docker containers and mounts our Nomad dev folder from the host machine into these containers. This means we run the local version of our Nomad source code rather than the ...We would like to have a docker-compose file that spins up needed docker containers and mounts our Nomad dev folder from the host machine into these containers. This means we run the local version of our Nomad source code rather than the one compiled into the container image.
This allows us to have almost the same dev environment as a production Nomad Oasis install.
Here are some recommendations or ideas we have already discussed in Area B:
- Make use of the docker-compose from ops/docker-compose/nomad-oasis
- Mount the dev folder into all containers that are not just infrastructure i.e. the app, hub, gui, etc.
- Run yarn in hot reload on the gui folder instead of using the production build for the gui container
- Maybe it is a good idea to check if we need to change/add some helper scripts to reload things like generate_gui_artifacts.sh. We would guess for scripts that just create new files on disk this wouldn't be an issue.
@dflor @sanbrock @kuehbachm @ecarohttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/969Testing data compatibility with metainfo schema2022-09-28T07:33:47ZLauri HimanenTesting data compatibility with metainfo schemaWe would like to add some form of testing how the production data complies with the current metainfo schema. This will give some important insights into how well the real data complies with our schema and lets us examine any compatibili...We would like to add some form of testing how the production data complies with the current metainfo schema. This will give some important insights into how well the real data complies with our schema and lets us examine any compatibility problems.
I would envision this as a manually triggered CI task. Would be written in python since we can easily load the data with python libs, and also perform the schema tests easily by loading the metainfo.Nathan DaelmanNathan Daelmanhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/935Improve on gitlab ci/cd script2022-09-13T08:21:03ZAdam FeketeImprove on gitlab ci/cd script- [x] define environments for `staging` (aka `develop`) similarly to `dev`
- [x] checking the rules because some of them does not make sense anymore (like: `$CI_COMMIT_REF_NAME =~ /^dev-.*$/`)
- [ ] checking the way how initialisation of...- [x] define environments for `staging` (aka `develop`) similarly to `dev`
- [x] checking the rules because some of them does not make sense anymore (like: `$CI_COMMIT_REF_NAME =~ /^dev-.*$/`)
- [ ] checking the way how initialisation of submodules works
- [x] get rid of docker login waning (`echo $CI_BUILD_TOKEN | docker login --username foo --password-stdin`)
- [x] cleanup (eg. using `before_script` more often)
- [ ] regularly rebuild the image from scratch to avoid issues like #937 (--no-cache)https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/934Inconsistent package versions during installation2022-09-22T08:23:48ZAdam FeketeInconsistent package versions during installationRunning `setup.sh` more precisely `dependencies.sh` changes the version of some of the python packages. In the end, you have an environment you have different packages that you "required" the first time. Note: as long as there is no brak...Running `setup.sh` more precisely `dependencies.sh` changes the version of some of the python packages. In the end, you have an environment you have different packages that you "required" the first time. Note: as long as there is no braking changes in the packages themself it is just an inconsistency...
Inconsistent versions in (and probably in more):
- dependencies/parsers/nexus/requirements.txt
- dependencies/parsers/workflow/requirements.txthttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/908Sporadically failing python-tests during CI2022-09-15T09:05:27ZMarkus ScheidgenSporadically failing python-tests during CI
This is a list of test cases, where we observed that they failed or errored, but the problem is gone when simply repeating the tests.
- `tests/app/v1/routers/test_uploads.py::test_post_upload[invalid-credentials]`
- `tests/app/v1/route...
This is a list of test cases, where we observed that they failed or errored, but the problem is gone when simply repeating the tests.
- `tests/app/v1/routers/test_uploads.py::test_post_upload[invalid-credentials]`
- `tests/app/v1/routers/test_uploads.py::test_put_upload_raw_path[zip-to-subfolder]`
- `tests/app/v1/routers/test_uploads.py::test_post_upload[stream-no-embargo]`
- `tests/app/v1/routers/test_uploads.py::test_editing_raw_file[conflict_in_concurrent_editing]`
These are probably race-conditions with a singular cause.David SikterDavid Sikterhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/906Improve on the build process of the docker image2022-11-29T16:38:33ZAdam FeketeImprove on the build process of the docker image1. [x] caching layers for multi-stage build
2. [x] The build process of the docker image can be accelerated by caching layers for multi-stage build:
https://testdriven.io/blog/faster-ci-builds-with-docker-cache/
TLDR: Each stage n...1. [x] caching layers for multi-stage build
2. [x] The build process of the docker image can be accelerated by caching layers for multi-stage build:
https://testdriven.io/blog/faster-ci-builds-with-docker-cache/
TLDR: Each stage needs to be built (using `--target ...`) and pushed separately.
2. [x] The building of "node" stage should be independent of the "python" stage.
3. [x] Many steps can be grouped together to avoid a lot of tiny layers
4. [x] installing python packages without versioning is useless because the setup.py script will uninstall most of them and replace them with a versioned package from the requirements.txt
5. [x] As far as I know there are more modern ways to handle requirements (citation needed)
6. [x] ~~mismatch in node versions (build:14.8 vs final 16.)~~
7. [x] ~~Consider (no hard feelings about this :smile: ) building separate images (still using the single Dockerfile but with multiple targets) for separate services (GUI, worker, north).~~
8. [x] ~~Inconsistent package versions during installation (closes: #934)~~
9. [x] Creating of a docker image for the development as mentioned in https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/989Adam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/905Kubernetes: switching to namespaces instead of using prefixes2022-12-10T00:44:47ZAdam FeketeKubernetes: switching to namespaces instead of using prefixesThe goal is to use namespaces to separate all the dev deployments from each other. Right now there is a single `nomad` namespace and all the pods/services are using a prefix to avoid any conflict.
Benefits:
- slightly more reasonable te...The goal is to use namespaces to separate all the dev deployments from each other. Right now there is a single `nomad` namespace and all the pods/services are using a prefix to avoid any conflict.
Benefits:
- slightly more reasonable testing environment
- different deployments cannot affect each other
- simplified pod/service names
- possibility to use the same release name
- in the case of a single namespace the explosion of the Pod Information to Containers Through Environment Variables could be dangerous
Disadvantage:
- the explosion of namespaces (cleanup is slightly more tricky...)
Known issues:
- `Error: secret “nomad-keycloak-password” not found`: Right now the secret has been defined manually for a specific namespaceAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/886Link between NOMAD OASIS and NOMAD Central2023-12-21T16:05:24ZFelix DietrichLink between NOMAD OASIS and NOMAD Central## General
Currently every OASIS is just a clone of the central repository.
They are not linked together, not even through simple pings.
In addition to simple links, it is necessary to define how OASES can differ from the central instal...## General
Currently every OASIS is just a clone of the central repository.
They are not linked together, not even through simple pings.
In addition to simple links, it is necessary to define how OASES can differ from the central installation (in their version, their data, their user rights, ...).
This may be communicated to central, too. In this issue, we outline these challenges and provide a road map for implementation and discussion.
## Current GIT Issues related to OASIS (2022-05-24)
* Improve oasis synchronisation functionality #585
* Integrating NORTH into OASIS #825
* Oasis installation and registration #820
* NOMAD Oasis (GUI) #784
* NOMAD Oasis UX Test: user roles #757
## Requirements: LINK
* Use case 1: Send telemetry data to central in regular intervals. This is discussed in {## Requirements: Telemetry}
* Use-case 2: send data buckets + metadata. This is discussed in section {## Requirements: Data LINK}
## Requirements: Telemetry
* Which data should be sent?
* Users registered
* Data uploaded, stored
* Number of searches
* OASIS location
* Do we need fine-grained info about site-access? This may be a security/leak risk that some may not like.
* See #820
## Requirements: Soft constraints / Installation
* Central: be careful with DDOS attacks from "too many / malicious OASIS links"
* LINK is opt-out, enabled by default
* OASIS owners can disable LINK in a config
* See roles, #757
* The fact that each OASIS sends telemetry data by default MUST be communicated clearly, also how to disable it
* We must be careful with "user data processing" requirements (GDPR). Aggregated data should be fine, but maybe some telemetry data can be used to identify users (link clicks, etc.).
## Requirements: Data LINK
* Also called "Synchronization"
* Send data buckets
* Send metaschema information, when they are stored in files
* How to send new metaschemas defined in python? Not send at all?
* Versioning of data: what if metaschema in the OASIS is newer/older than the one in central?
* See #784
* See #585Daniel LehmbergDaniel Lehmberghttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/856Revise version information2022-12-19T11:42:42ZMarkus ScheidgenRevise version information- The current NOMAD version is stated a many places throughout the source code. This is annoying to change.
- The documentation has no version statement
- The GUI could have a more prominent statement
- The API does not give any metadata...- The current NOMAD version is stated a many places throughout the source code. This is annoying to change.
- The documentation has no version statement
- The GUI could have a more prominent statement
- The API does not give any metadata with its responsesAdam FeketeAdam Feketehttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/845Updating the legacy Keycloak deployment2023-12-21T15:39:09ZAdam FeketeUpdating the legacy Keycloak deploymentAlthough the 16.1.1 of Keycloak was released about 5 months ago since there is a major update starting with version 17.0.0. The "backend" has been changed and some migration is needed for using the latest versions.
Useful documentations...Although the 16.1.1 of Keycloak was released about 5 months ago since there is a major update starting with version 17.0.0. The "backend" has been changed and some migration is needed for using the latest versions.
Useful documentations:
- [Main page about the migration](https://www.keycloak.org/migration/migrating-to-quarkus)
- [New base image is used for building a container](https://www.keycloak.org/server/containers)
- [Development (using localhost) versus production mode](https://www.keycloak.org/server/configuration)
- https://www.keycloak.org/guides#serverhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/757NOMAD Oasis UX Test: user roles2024-01-09T14:00:27ZFelix DietrichNOMAD Oasis UX Test: user rolesRelated to #756.
What people/roles are involved when a NOMAD Oasis is installed?
A person involved with the Oasis should be in one or multiple of the following roles:
- Oasis Owner
- Oasis Admin
- It Admin
- Data Producer
- Data C...Related to #756.
What people/roles are involved when a NOMAD Oasis is installed?
A person involved with the Oasis should be in one or multiple of the following roles:
- Oasis Owner
- Oasis Admin
- It Admin
- Data Producer
- Data Consumer
Oasis Owner:
- Responsible for the Oasis
- Usually professor, post-doc, lab owner...
- Decides to create the Oasis
- All general decisions are made by them
- Represents Oasis to the outside world
Oasis Admin:
- Can be the Owner or a delegated person (PhD candidate, post-doc, ...)
- Deals with more technical day-to-day issues
- Handles user management issues (from global keycloak repository? local users?)
- Plans and executes the setup of the IT infrastructure together with IT Admins
- Interacts with all other roles (Owner, IT, Producer, Consumer)
- Main contact person for users and for main NOMAD team regarding technical issues
IT Admin:
- Responsible for providing the technical infrastructure
- Setting up the server + providing hostname
- Setting up the installation of NOMAD (alternative: Oasis Admin?)
- Creates means for access restriction (VPN? HTTPS?)
Data Producer:
- Material Scientist who performs experiments and collects data
- Collects and prepares/sanitizes data
- Uploads data to the NOMA Oasis
- Decides on Access restrictions
- Potentially publishes the data to central NOMAD instance
Data Consumer:
- Scientist or Data Analyst or other end user of the materials science data
- Has read access to the data stored in the Oasis
- Downloads and analyzes data
- Compares different data items
- Visualizes dataFelix DietrichFelix Dietrichhttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/634Refactor config "api" keys2023-12-21T15:38:38ZMarkus ScheidgenRefactor config "api" keysThe configuration of API related keys is very confusing. The path is always the same, but host/port have three different meanings that need to be supported by three different keys:
- to use by nomad client
- to return for external use
- ...The configuration of API related keys is very confusing. The path is always the same, but host/port have three different meanings that need to be supported by three different keys:
- to use by nomad client
- to return for external use
- to access the API via the internal server side network between different app container
Another thing is GUI vs API (which makes a difference in dev mode).
Similar internal vs external distinction is necessary for files (inside container, outside container).https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/628Better user-magement2023-12-21T15:55:38ZMarkus ScheidgenBetter user-magement- upgrade keycloak (to version 8?), keeping our custom interface
- connect via OAuth (e.g. with github, google)
- connect via SAML (e.g. with DFN AAI)
- allow local keycloaks for Oasis (why?)
- common user id (e.g. ORCID)
- consolidate u...- upgrade keycloak (to version 8?), keeping our custom interface
- connect via OAuth (e.g. with github, google)
- connect via SAML (e.g. with DFN AAI)
- allow local keycloaks for Oasis (why?)
- common user id (e.g. ORCID)
- consolidate users (e.g. automatically based on ORCID, CLI functions to migrate users)
- properly use (or don't use at all) affiliations
- allow authors that are not usershttps://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/616Use pydantic settings management for nomad.config2023-12-21T15:40:00ZMarkus ScheidgenUse pydantic settings management for nomad.configPydantic has build in support for [settings management](https://pydantic-docs.helpmanual.io/usage/settings/). This supports everything that our `NomadConfig`-obj system provides (but better):
- nested config objects
- override from first...Pydantic has build in support for [settings management](https://pydantic-docs.helpmanual.io/usage/settings/). This supports everything that our `NomadConfig`-obj system provides (but better):
- nested config objects
- override from first env (limited), than config file, than default values
Here is an example of how it could work:
```python
from typing import Dict, Any
from pydantic import BaseModel, Field, BaseSettings, HttpUrl
import yaml
import os.path
import os
class Client(BaseModel):
user: str = Field('test', description='The username.')
password: str = '*'
url: HttpUrl = Field('http://nomad-lab.eu')
class NomadConfig(BaseSettings):
client: Client = Client()
not_nested: str = 'hello'
class Config:
env_prefix = 'nomad_'
case_sensitive = False
@classmethod
def customise_sources(
cls,
init_settings,
env_settings,
file_secret_settings):
return (
init_settings,
env_settings,
yaml_config_settings_source, file_secret_settings)
def yaml_config_settings_source(settings: BaseSettings) -> Dict[str, Any]:
if not os.path.exists('nomad.yaml'):
return {}
try:
with open('nomad.yaml') as f:
data = yaml.load(f, Loader=yaml.FullLoader)
if data is None:
return {}
return data
except yaml.YAMLError as e:
raise e
config = NomadConfig()
```
The support of env vars is limited as they only apply to the first level (e.g. NomadConfig). To set nested fields, the top level env var has to be set with a json value:
```
export NOMAD_CLIENT='{"password":"mypassword"}'
```
If necessary this could be replaced by a custom settings source.
```https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/issues/402Enforcing HTTPS across all services2023-12-21T15:40:02ZLauri HimanenEnforcing HTTPS across all servicesCurrently many of the resources that our production machine serves can be retrieved both with http or https protocols.
As far as I understand, the standard these days is to always use https, no matter what the resource is. This ensures ...Currently many of the resources that our production machine serves can be retrieved both with http or https protocols.
As far as I understand, the standard these days is to always use https, no matter what the resource is. This ensures that all outgoing data is always properly secured (we don't have to selectively enable https, as that is prone to mistakes), and that there will be no issue with resources interacting with different protocols.
Doing this should be relative easy: we will still accept incoming requests through http, but these requests will always be redirected to https. This can be done through our nginx server, [with something like this.](https://serversforhackers.com/c/redirect-http-to-https-nginx) You can see that most sites do something similar, e.g. if you try to load this issue page with http, Gitlab will automatically switch to https.
What do you think?