Commit 86bd2e0c authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Updated docker-compose and ops documentation.

parent 3d57e4c9
Pipeline #54763 passed with stages
in 22 minutes and 35 seconds
......@@ -2,6 +2,6 @@ Operating nomad
===============
.. mdinclude:: ../ops/README.md
.. mdinclude:: ../ops/docker-compose/nomad/README.md
.. mdinclude:: ../ops/helm/nomad/README.md
.. mdinclude:: ../ops/containers/README.md
.. mdinclude:: ../ops/docker-compose/README.md
.. mdinclude:: ../ops/helm/README.md
\ No newline at end of file
## Overview
Read the [introduction](./introduction.html) and [setup](./setup.html) for input on
the different nomad services. This is about how to deploy and operate these services.
\ No newline at end of file
the different nomad services. This is about how to deploy and operate these services.
The databases and other external services can be run from
normal public dockerhub images or you have to setup your own private registry with
respective images.
The NOMAD specific images are provide by our
[gitlab container registry](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/container_registry).
To access these container you have to login without MPCDF gitlab account:
```
docker login gitlab-registry.mpcdf.mpg.de/nomad-lab
```
There are two basic options to run all these containers: docker-compose and kubernetes.
\ No newline at end of file
## Containers
### ELK
This image is based on the popular elk-stack docker image:
[github](https://github.com/spujadas/elk-docker),
[readthedocs](http://elk-docker.readthedocs.io/).
Changes
- disabled ssl for beats communication to logstash server
- added tcp input
- simplified elastic search output (don't now how to set metric and other vars yet :-()
- added kibana.yml::server.basePath="/nomad/kibana"
The file `elk/kibana_objects.json` contains an export of nomad specific searches,
visualizations, and dashboard.
\ No newline at end of file
There are some additional database services that NOMAD might need, which can not be
readily used by publicly available images. Everything necessary to build images for these
container can be found in `ops/containers`.
\ No newline at end of file
### ELK (optional)
This image is based on the popular elk-stack docker image:
[github](https://github.com/spujadas/elk-docker),
[readthedocs](http://elk-docker.readthedocs.io/).
Changes
- disabled ssl for beats communication to logstash server
- added tcp input
- simplified elastic search output (don't now how to set metric and other vars yet :-()
- added kibana.yml::server.basePath="/nomad/kibana"
The file `elk/kibana_objects.json` contains an export of nomad specific searches,
visualizations, and dashboard.
\ No newline at end of file
## Single Node Deployment, Using Docker Compose
### nomad
In `nomad` you find *docker-compose* files that can be used to run nomad in docker-compose,
either for developement or production purposes. See [setup](./setup.html) for details
on running things for development.
We use docker-compose overrides to modify config for development and production. Example:
```
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d api
```
The different overrides are:
- `*.prod.yml`, production (currently on enc-preprocessing-nomad.esc)
- `*.override.yml`, development (will be automatically used by docker-compose)
- `*.develk.yml`, like development but also runs ELK
The .env file contains some additional config and secrets. The development secrets do
not matter and are in the git (`.env_development`) and are replaced by real secret on
the production machine.
### Matomo (piwik)
This docker-compose can be used to run the user-data tracking server *Matomo* and its
database. This is currently not used by the official nomad production deployment.
This docker-compose can be used to run the user-data tracking server *Matomo* and its
database. This is currently not used by the official nomad production deployment.
\ No newline at end of file
# copy this to .env and modify
# in production the values must be replaced with real host, cookies, and secrets
# do not check this information into git
VOLUME_BINDS=../../.volumes
EXTERNAL_HOST=localhost
EXTERNAL_PORT=80
RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG
RABBITMQ_DEFAULT_USER=rabbitmq
RABBITMQ_DEFAULT_PASS=rabbitmq
RABBITMQ_DEFAULT_VHOST=/
API_SECRET=defaultApiSecret
Contains all files to run nomad with docker-compose on a single docker, both
for development and production.
## Single Node NOMAD Deployment, Using Docker Compose
You can run NOMAD with [docker-compose](https://docs.docker.com/compose/) on a single node
that supports docker and docker-compse. The docker-compose files are part of the
[NOMAD source code](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR)
and can be found under `ops/docker-compose/nomad`.
### How we use docker-compose
You can use docker-compose to run all necessary databases, the API, the worker, and
gui with one single docker-compose configuration. The `docker-compose.yml` defines
all the different container.
We use docker-compose overrides to extend a base configuration for different scenarios.
Example docker-compose usage:
We use docker-compose overrides to modify config for development and production. Example:
```
docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d api
```
The different overrides are:
- *.prod.yml, production (currently on enc-preprocessing-nomad.esc)
- *.override.yml, development (will be automatically used by docker-compose)
- *.prod.yml, production (to run the necessary databases for kubenetes deployments)
- *.override.yml, development (development configuration, will be automatically used by docker-compose)
- *.develk.yml, like development but also runs ELK
- *.example.yml, an example production configuration
To run your own NOMAD mirror or oasis, you should override the `example.yml` to fit your needs.
Within the overrides you can configure the NOMAD containers (api, worker, gui).
### Configuring the API and worker
The API and worker can be configured through a nomad.yaml file or with environment
variables. A nomad.yaml files needs to be mounted to `/app/nomad.yaml`. There are
[several options](https://docs.docker.com/compose/environment-variables/) to set
environment variables in docker-compose.
An example file can be found under `ops/docker-compose/nomad/example/nomad.yaml`.
### Configuring the GUI
This encompasses to parts a `env.js` file for the client side GUI code. And an
`nginx.conf` for the web server running the GUI (and reverse proxying the API).
Both need to be mounted to the respective containers. The `env.js` under `/app/nomad/env.js`
and the `nginx.conf` under `/etc/nginx/conf.d/default.conf`.
Example files can be found here `ops/docker-compose/nomad/example/`.
### Example *.prod.yml override:
Finally the full override for the example deployment based on the example config files
can be found here: `ops/docker-compose/nomad/docker-compose.example.yml`.
To simply run a fully fresh nomad:
```
git clone
cd nomad/ops/docker-compose/nomad/
docker login
docker-compose -f docker-compose.yml -f docker-compose.example.yml up
```
If everything goes well, NOMAD should be available at `http://your-host/my_example_nomad/gui/`.
The .env file contains some additional config and secrets. The development secrets do
not matter and are in the git (.env_development) and are replaced by real secret on
the production machine.
\ No newline at end of file
We recommend to either change the nginx.conf to use SSL or put this behind a reverse-proxy
that supports SSL. If you use a reverse-proxy, you should disable any buffering to support
large downloads/uploads. We recommend to change the volume `nomad_data` to a bind mount.
\ No newline at end of file
version: '3.4'
services:
api:
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:v0.5.3
volumes:
- "./example_nomad/nomad.yaml:/app/nomad.yaml"
worker:
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:v0.5.3
volumes:
- "./example_nomad/nomad.yaml:/app/nomad.yaml"
gui:
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair/frontend:v0.5.3
volumes:
- "./example_nomad/nginx.conf:/etc/nginx/conf.d/default.conf"
- "./example_nomad/env.js:/app/nomad/env.js"
ports:
- 80:80
command: ["./run.sh", "/example-nomad"]
\ No newline at end of file
......@@ -50,12 +50,6 @@ services:
ports:
- 8000:8000
# NOMAD-coe search api
coeapi:
restart: 'no'
ports:
- 8111:8111
# nomad gui
gui:
restart: 'no'
......
......@@ -66,6 +66,6 @@ services:
volumes:
- /nomad:/nomad
proxy:
gui:
ports:
- 8080:80
- 80:80
......@@ -19,6 +19,7 @@ x-common-variables: &nomad_backend_env
NOMAD_LOGSTASH_HOST: elk
NOMAD_ELASTIC_HOST: elastic
NOMAD_MONGO_HOST: mongo
NOMAD_REPOSITORY_DB_HOST: postgres
services:
# postgres for NOMAD-coe repository API and GUI
......@@ -38,9 +39,9 @@ services:
image: rabbitmq
container_name: nomad_rabbitmq
environment:
- RABBITMQ_ERLANG_COOKIE=${RABBITMQ_ERLANG_COOKIE}
- RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
- RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
- RABBITMQ_ERLANG_COOKIE=SWQOKODSQALRPCLNMEQG
- RABBITMQ_DEFAULT_USER=rabbitmq
- RABBITMQ_DEFAULT_PASS=rabbitmq
- RABBITMQ_DEFAULT_VHOST=/
volumes:
- nomad_rabbitmq:/var/lib/rabbitmq
......@@ -79,7 +80,7 @@ services:
- elastic
- mongo
volumes:
- ${VOLUME_BINDS}/fs:/app/.volumes/fs
- nomad_files:/app/.volumes/fs
command: python -m celery worker -l info -A nomad.processing -Q celery,cacls,uploads
# nomad api
......@@ -89,10 +90,6 @@ services:
container_name: nomad_api
environment:
<<: *nomad_backend_env
NOMAD_SERVICES_API_BASE_PATH: /fairdi/nomad/latest/api
NOMAD_SERVICES_API_HOST: ${EXTERNAL_HOST}
NOMAD_SERVICES_API_PORT: ${EXTERNAL_PORT}
NOMAD_SERVICES_API_SECRET: ${API_SECRET}
NOMAD_SERVICE: nomad_api
links:
- postgres
......@@ -100,45 +97,21 @@ services:
- elastic
- mongo
volumes:
- ${VOLUME_BINDS}/fs:/app/.volumes/fs
- nomad_files:/app/.volumes/fs
command: python -m gunicorn.app.wsgiapp -w 4 --log-config ops/gunicorn.log.conf -b 0.0.0.0:8000 --timeout 300 nomad.api:app
# NOMAD-coe search api
coeapi:
restart: always
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair/coe-repowebservice:latest
container_name: nomad_coeapi
environment:
REPO_DB_JDBC_URL: jdbc:postgresql://postgres:5432/nomad
REPO_ELASTIC_URL: elasticsearch://elastic:9200
links:
- postgres
- elastic
# nomad gui
gui:
restart: always
image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair/frontend:latest
container_name: nomad_gui
command: nginx -g 'daemon off;'
# reverse proxy everything together
proxy:
restart: always
image: nginx:1.13.9-alpine
container_name: nomad_proxy
links:
- gui
- api
ports:
- 80:80
volumes:
- ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
command: nginx -g 'daemon off;'
volumes:
nomad_postgres:
nomad_mongo:
nomad_elastic:
nomad_rabbitmq:
nomad_elk:
nomad_files:
window.nomadEnv = {
"apiBase": "/example-nomad/api",
"kibanaBase": "/fairdi/kibana",
"debug": false
};
\ No newline at end of file
server {
listen 80;
server_name www.example.com;
location /example-nomad {
return 301 /example-nomad/gui;
}
location /example-nomad/gui {
root /app/;
rewrite ^/example-nomad/gui/(.*)$ /nomad/$1 break;
try_files $uri /example-nomad/gui/index.html;
}
location /example-nomad/gui/service-worker.js {
add_header Last-Modified $date_gmt;
add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
if_modified_since off;
expires off;
etag off;
root /app/;
rewrite ^/example-nomad/gui/service-worker.js /nomad/service-worker.js break;
}
location /example-nomad/api {
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_pass http://api:8000;
}
location /example-nomad/api/uploads {
client_max_body_size 35g;
proxy_request_buffering off;
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_pass http://api:8000;
}
location /example-nomad/api/raw {
proxy_buffering off;
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_pass http://api:8000;
}
}
\ No newline at end of file
server {
listen 80;
server_name www.example.com;
location /fairdi/nomad/latest/gui {
proxy_pass http://gui:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location /fairdi/nomad/latest/api {
client_max_body_size 20g;
proxy_set_header Host $host;
proxy_pass_request_headers on;
proxy_pass http://api:8000;
}
location ~ ^/fairdi/kibana/(.*)$ {
resolver 127.0.0.11 ipv6=off; # docker embedded DNS
proxy_pass http://elk:5601/$1$is_args$args; # fix: https://github.com/elastic/kibana/issues/13533
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
## Cluster Deployment, Using Kubernetes and Helm
We use helm charts to describe the deployment of nomad services in a kubernetes cluster.
### nomad
The NOMAD chart is part of the
[NOMAD source code](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR)
and can be found under `ops/helm/nomad`.
This chart allows to run the nomad api, worker, gui, and proxy in a kubernetes cluster.
The `values.yaml` contains more documentation on the different values.
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment