Commit 96345ea0 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Prepared everything for OASIS.

parent 9f76442b
Pipeline #69837 passed with stages
in 15 minutes and 27 seconds
......@@ -29,6 +29,10 @@ contributing, and API reference.
Omitted versions are plain bugfix releases with only minor changes and fixes.
### v0.7.9
- Everything to run a simple NOMAD OASIS based on the central user-management
- minor bugfixes
### v0.7.7
- Shows dataset contents with embargo data, but hides the entry details (raw-files, archive)
- minor bugfixes
......
Operating nomad
Operating NOMAD
===============
.. mdinclude:: ../ops/README.md
.. mdinclude:: ../ops/docker-compose/nomad/README.md
.. mdinclude:: ../ops/helm/nomad/README.md
.. mdinclude:: ../ops/containers/README.md
.. mdinclude:: ../ops/docker-compose/nomad-oasis/README.md
......@@ -11,18 +11,19 @@ The nomad infrastructure consists of a series of nomad and 3rd party services:
- rabbitmq: a task queue used to distribute work in a cluster
All 3rd party services should be run via *docker-compose* (see blow). The
nomad python services can also be run via *docker-compose* or manually started with python.
The gui can be run manually with a development server via yarn, or with
*docker-compose*
nomad python services can be run with python to develop them.
The gui can be run with a development server via yarn.
Below you will find information on how to install all python dependencies and code
manually. How to use *docker*/*docker-compose*. How run services with *docker-compose*
or manually.
manually. How to use *docker*/*docker-compose*. How run 3rd-party services with *docker-compose*.
Keep in mind the *docker-compose* configures all services in a way that mirror
the configuration of the python code in `nomad/config.py` and the gui config in
`gui/.env.development`.
To learn about how to run everything in docker, e.g. to operate a NOMAD OASIS in
production, go (here)(/app/docs/ops.html).
## Install python code and dependencies
### Cloning and development tools
......@@ -158,35 +159,12 @@ having to copy the git itself to the docker build context.
The images are build via *docker-compose* and don't have to be created manually.
### Build with docker-compose
We have multiple *docker-compose* files that must be used together.
- `docker-compose.yml` contains the base definitions for all services
- `docker-compose.override.yml` configures services for development (notably builds images for nomad services)
- `docker-compose.dev-elk.yml` will also provide the ELK service
- `docker-compose.prod.yml` configures services for production (notable uses a pre-build image for nomad services that was build during CI/CD)
It is sufficient to use the implicit `docker-compose.yml` only (like in the command below).
The `override` will be used automatically.
Now we can build the *docker-compose* that contains all external services (rabbitmq,
mongo, elastic, elk) and nomad services (worker, app, gui).
```
docker-compose build
```
Docker-compose tries to cache individual building steps. Sometimes this causes
troubles and not everything necessary is build when you changed something. In
this cases use:
```
docker-compose build --no-cache
```
### Run everything with docker-compose
### Run necessary 3-rd party services with docker-compose
You can run all containers with:
```
docker-compose up
cd ops/docker-compose/nomad
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d mongo elastic rabbitmq
```
To shut down everything, just `ctrl-c` the running output. If you started everything
......@@ -195,25 +173,6 @@ in *deamon* mode (`-d`) use:
docker-compose down
```
### Run containers selectively
The following services/containers are managed via our docker-compose:
- rabbitmq, mongo, elastic, (elk, only for production)
- worker, app
- gui
- proxy
The *proxy* container runs *nginx* based reverse proxies that put all services under
a single port and different paths.
You can also run services selectively, e.g.
```
docker-compose up -d rabbitmq, mongo, elastic
docker-compose up worker
docker-compose up app gui proxy
```
## Accessing 3'rd party services
Usually these services only used by the nomad containers, but sometimes you also
need to check something or do some manual steps.
......@@ -234,12 +193,7 @@ The index prefix for logs is `logstash-`. The ELK is only available with the
You can access mongodb and elastic search via your preferred tools. Just make sure
to use the right ports (see above).
## Run nomad services manually
You can run the worker, app, and gui as part of the docker infrastructure, like
seen above. But, of course there are always reasons to run them manually during
development, like running them in a debugger, profiler, etc.
## Run nomad services
### API and worker
......@@ -253,11 +207,6 @@ To run it directly with celery, do (from the root)
celery -A nomad.processing worker -l info
```
Run the app via docker, or (from the root):
```
nomad admin run app
```
You can also run worker and app together:
```
nomad admin run appworker
......
......@@ -37,7 +37,6 @@ import os
import os.path
import yaml
import warnings
import sys
from nomad import gitinfo
......@@ -100,7 +99,6 @@ fs = NomadConfig(
tmp='.volumes/fs/tmp',
staging='.volumes/fs/staging',
public='.volumes/fs/public',
migration_packages='.volumes/fs/migration_packages',
local_tmp='/tmp',
prefix_size=2,
working_directory=os.getcwd()
......@@ -309,7 +307,7 @@ def load_config(config_file: str = os.environ.get('NOMAD_CONFIG', 'nomad.yaml'))
NOMAD_CONFIG or ``nomad.yaml``.
"""
# load yaml and override defaults (only when not in test)
if os.path.exists(config_file) and 'pytest' not in sys.modules:
if os.path.exists(config_file):
with open(config_file, 'r') as stream:
try:
config_data = yaml.load(stream, Loader=getattr(yaml, 'FullLoader'))
......
......@@ -939,3 +939,8 @@ class PublicUploadFiles(UploadFiles):
for zip_file in self._zipfile_cache.values():
zip_file.close()
self._zipfile_cache = None
for directory in [config.fs.public, config.fs.staging, config.fs.tmp]:
if not os.path.exists(directory):
os.makedirs(directory)
......@@ -9,10 +9,3 @@ respective images.
The NOMAD specific images are provide by our
[gitlab container registry](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/container_registry).
To access these container you have to login without MPCDF gitlab account:
```
docker login gitlab-registry.mpcdf.mpg.de/nomad-lab
```
There are two basic options to run all these containers: docker-compose and kubernetes.
\ No newline at end of file
# Operating a NOMAD OASIS
The following describes the simplest way to run your own NOMAD.
## What is an OASIS
Originally NOMAD is a service run at Max-Planck's compute facility in Garching, Germany.
However, the NOMAD software is Open-Source and everybody can run it. We call any service that
uses NOMAD software independently a *NOMAD OASIS*.
While there are several use cases that require different setups, this documentations
describes the simples NOMAD OASIS setup possible. It will allow you to use NOMAD to
manage research data locally, while using NOMAD's central user-management and its users.
## Pre-requisites
NOMAD software is distributed as a set of docker containers. Further, other services
that can be run with docker are required. Further, we use docker-compose to setup
all necessary container in the simples possible manner.
You will need a single computer, with **docker** and **docker-compose** installed.
The following will run all necessary services with docker. These comprise: a **mongodb**
database, an **elasticsearch**, a **rabbitmq** distributed task queue, the NOMAD **app**,
NOMAD **worker**, and NOMAD **gui**. Refer to this [introduction](/app/docs/introduction.html#architecture)
to learn what each service does and why it is necessary.
There is also some information you need to configure your NOMAD OASIS:
- The hostname for the machine you run NOMAD on. This is important for redirects between
your OASIS and the central NOMAD user-management and to allow your users to upload files (via GUI or API).
Your machine needs to be accessible under this hostname from the public internet. The host
name needs to be registered with the central NOMAD in order to configure the central user-
management correctly.
- A NOMAD account that acts as an admin account for your OASIS. This account must be declared
to the central NOMAD as an OASIS admin in order to give it the necessary rights in the central user-
management.
## Configuration
All docker container are configured via docker-compose an the respective `docker-compose.yaml` file.
Further, we will need to mount some configuration files to configure the NOMAD services within
their respective containers.
Please [write us](mailto:webmaster@nomad-coe.eu) to register your NOMAD account as an OASIS
admin and to register your hostname. Please replace the indicated configuration items with
the right information.
There are three files to configure:
- `docker-compose.yaml`
- `nomad.yaml`
- `env.js`
- `nginx.conf`
In this example, we have all files in the same directory (the directory we also work from).
You can download examples files from
[here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/tree/master/ops/docker-compose/nomad-oasis/).
### Docker compose
The most basic `docker-compose.yaml` to run an OASIS looks like this:
```yaml
version: '3.4'
x-common-variables: &nomad_backend_env
NOMAD_RABBITMQ_HOST: rabbitmq
NOMAD_ELASTIC_HOST: elastic
NOMAD_MONGO_HOST: mongo
services:
# broker for celery
rabbitmq:
restart: always
image: rabbitmq:3.7.17
container_name: nomad_oasis_rabbitmq
environment:
- RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG
- RABBITMQ_DEFAULT_USER=rabbitmq
- RABBITMQ_DEFAULT_PASS=rabbitmq
- RABBITMQ_DEFAULT_VHOST=/
volumes:
- nomad_oasis_rabbitmq:/var/lib/rabbitmq
# the search engine
elastic:
restart: always
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: nomad_oasis_elastic
volumes:
- nomad_oasis_elastic:/usr/share/elasticsearch/data
# the user data db
mongo:
restart: always
image: mongo:4
container_name: nomad_oasis_mongo
environment:
- MONGO_DATA_DIR=/data/db
- MONGO_LOG_DIR=/dev/null
volumes:
- nomad_oasis_mongo:/data/db
command: mongod --logpath=/dev/null # --quiet
# nomad worker (processing)
worker:
restart: always
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:stable
container_name: nomad_oasis_worker
environment:
<<: *nomad_backend_env
NOMAD_SERVICE: nomad_oasis_worker
links:
- rabbitmq
- elastic
- mongo
volumes:
- nomad_oasis_files:/app/.volumes/fs
- ./nomad.yaml:/app/nomad.yaml
command: python -m celery worker -l info -A nomad.processing -Q celery,calcs,uploads
# nomad app (api)
app:
restart: always
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:stable
container_name: nomad_oasis_app
environment:
<<: *nomad_backend_env
NOMAD_SERVICE: nomad_oasis_app
links:
- rabbitmq
- elastic
- mongo
volumes:
- nomad_oasis_files:/app/.volumes/fs
- ./nomad.yaml:/app/nomad.yaml
command: python -m gunicorn.app.wsgiapp -w 4 --log-config ops/gunicorn.log.conf -b 0.0.0.0:8000 --timeout 300 nomad.app:app
# nomad gui (serving the js)
gui:
restart: always
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair/frontend:stable
container_name: nomad_oasis_gui
command: nginx -g 'daemon off;'
volumes:
- ./nginx.conf:/etc/nginx/conf.d/default.conf
- ./env.js:/app/nomad/env.js
links:
- app
ports:
- 80:80
command: ["./run.sh", "/example-nomad"]
volumes:
nomad_oasis_mongo:
nomad_oasis_elastic:
nomad_oasis_rabbitmq:
nomad_oasis_files:
```
There are no mandatory changes necessary.
A few things to notice:
- All services use docker volumes for storage. This could be changed to host mounts.
- It mounts three configuration files that need to be provided (see below): `nomad.yaml`, `nginx.conf`, `env.js`.
- The only exposed port is `80`. This could be changed to a desired port if necessary.
- The NOMAD images are pulled from our gitlab in Garching, the other services use images from a public registry (*dockerhub*).
- All container will be named `nomad_oasis_*`. These names can be used to later reference the container with the `docker` cmd.
- The NOMAD images we use are tagged `stable`. This could be replaced with concrete version tags.
- The services are setup to restart `always`, you might want to change this to `no` while debugging errors to prevent
indefinite restarts.
### nomad.yaml
NOMAD app and worker read a `nomad.yaml` for configuration.
```yaml
client:
url: 'http://<your-host>/nomad-oasis/api'
services:
api_base_path: '/nomad-oasis'
admin_user_id: '<your admin user id>'
keycloak:
realm_name: fairdi_nomad_prod
username: '<your admin username>'
password: '<your admin user password>'
oasis: true
```
You need to change:
- Replace `your-host` and admin credentials respectively.
- `api_base_path` defines the path under with the app is run. It needs to be changed, if you use a different base path.
A few things to notice:
- Be secretive about your admin credentials; make sure this file is not publicly readable.
### env.js
The GUI also has a config file, called `env.js` with a similar function than `nomad.yaml`.
```js
window.nomadEnv = {
'appBase': '/nomad-oasis/',
'keycloakBase': 'https://repository.nomad-coe.eu/fairdi/keycloak/auth/',
'keycloakRealm': 'fairdi_nomad_prod',
'keycloakClientId': 'nomad_public',
'debug': false,
};
```
You need to change:
- `appBase` defines the base path again. It needs to be changed, if you use a different base path.
### nginx.conf
The GUI container also serves as a proxy that forwards request to the app container. The
proxy is an nginx server and needs a configuration similar to this:
```
server {
listen 80;
server_name <your-host>;
location /nomad-oasis {
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_pass http://app:8000;
}
location /nomad-oasis/gui {
root /app/;
rewrite ^/nomad-oasis/gui/(.*)$ /nomad/$1 break;
try_files $uri /nomad-oasis/gui/index.html;
}
location /nomad-oasis/gui/service-worker.js {
add_header Last-Modified $date_gmt;
add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
if_modified_since off;
expires off;
etag off;
root /app/;
rewrite ^/nomad-oasis/gui/service-worker.js /nomad/service-worker.js break;
}
location /nomad-oasis/api/uploads {
client_max_body_size 35g;
proxy_request_buffering off;
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_pass http://app:8000;
}
location /nomad-oasis/api/raw {
proxy_buffering off;
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_pass http://app:8000;
}
}
```
You need to change:
- Replace `<your-host>`
A few things to notice:
- It configures the base path (`nomad-oasis`) at multiple places. It needs to be changed, if you use a different base path.
- You can use the server to server additional content if you like.
- `client_max_body_size` sets a limit to the possible upload size.
- If you operate the GUI container behind another proxy, keep in mind that your proxy should not buffer requests/responses to allow streaming of large requests/responses for `../api/uploads` and `../api/raw`.
## Starting and stopping
If you prepared the above files, simply use the usual `docker-compose` commands to start everything.
In the beginning and for debugging problems, it is recommended to start services separately:
```
docker-compose up -d mongodb elastic rabbitmq
docker-compose up app worker gui
```
The `-d` option runs container in the background as *daemons*. Later you can run all at once:
```
docker-compose up -d
```
You can also use docker to stop and remove faulty containers that run as *daemons*:
```
docker stop nomad_oasis_app
docker rm nomad_oasis_app
```
If everything works, the gui should be available under:
```
http://<your host>/nomad-oasis/gui/
```
If you run into troubles, use the dev-tools of you browser to check the javascript logs
or monitor the network traffic for HTTP 500/400/404/401 responses.
To see if at least the api works, check
```
http://<your host>/nomad-oasis/alive
http://<your host>/nomad-oasis/api/info
```
To see logs or 'go into' a running container, you can access the individual containers
with their names and the usual docker commands:
```
docker logs nomad_oasis_app
```
```
docker exec -ti nomad_oasis_app /bin/bash
```
If you want to report problems with your OASIS. Please provide the logs for
- nomad_oasis_app
- nomad_oasis_worker
- nomad_oasis_gui
\ No newline at end of file
# Copyright 2018 Markus Scheidgen
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an"AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
version: '3.4'
x-common-variables: &nomad_backend_env
NOMAD_RABBITMQ_HOST: rabbitmq
NOMAD_ELASTIC_HOST: elastic
NOMAD_MONGO_HOST: mongo
services:
# broker for celery
rabbitmq:
restart: always
image: rabbitmq:3.7.17
container_name: nomad_oasis_rabbitmq
environment:
- RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG
- RABBITMQ_DEFAULT_USER=rabbitmq
- RABBITMQ_DEFAULT_PASS=rabbitmq
- RABBITMQ_DEFAULT_VHOST=/
volumes:
- nomad_oasis_rabbitmq:/var/lib/rabbitmq
# the search engine
elastic:
restart: always
image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
container_name: nomad_oasis_elastic
volumes:
- nomad_oasis_elastic:/usr/share/elasticsearch/data
# the user data db
mongo:
restart: always
image: mongo:4
container_name: nomad_oasis_mongo
environment:
- MONGO_DATA_DIR=/data/db
- MONGO_LOG_DIR=/dev/null
volumes:
- nomad_oasis_mongo:/data/db
command: mongod --logpath=/dev/null # --quiet
# nomad worker (processing)
worker:
restart: always
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:latest
container_name: nomad_oasis_worker
environment:
<<: *nomad_backend_env
NOMAD_SERVICE: nomad_oasis_worker
links:
- rabbitmq
- elastic
- mongo
volumes:
- nomad_oasis_files:/app/.volumes/fs
- ./nomad.yaml:/app/nomad.yaml
command: python -m celery worker -l info -A nomad.processing -Q celery,calcs,uploads
# nomad app (api)
app:
restart: always
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:latest
container_name: nomad_oasis_app
environment:
<<: *nomad_backend_env
NOMAD_SERVICE: nomad_oasis_app
links:
- rabbitmq
- elastic
- mongo
volumes:
- nomad_oasis_files:/app/.volumes/fs
- ./nomad.yaml:/app/nomad.yaml
command: python -m gunicorn.app.wsgiapp -w 4 --log-config ops/gunicorn.log.conf -b 0.0.0.0:8000 --timeout 300 nomad.app:app
# nomad gui (serving the js)
gui:
restart: always
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair/frontend:latest
container_name: nomad_oasis_gui
command: nginx -g 'daemon off;'
volumes:
- ./nginx.conf:/etc/nginx/conf.d/default.conf
- ./env.js:/app/nomad/env.js
links:
- app
ports:
- 80:80
command: ["./run.sh", "/nomad-oasis"]
volumes:
nomad_oasis_mongo:
nomad_oasis_elastic:
nomad_oasis_rabbitmq:
nomad_oasis_files:
\ No newline at end of file
window.nomadEnv = {
'appBase': '/nomad-oasis/',
'keycloakBase': 'https://repository.nomad-coe.eu/fairdi/keycloak/auth/',
'keycloakRealm': 'fairdi_nomad_prod',
'keycloakClientId': 'nomad_public',
'debug': false,
};
\ No newline at end of file