Commit 1f8bb5ae authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merged v0.6.1.

parents e88cdbfe 5742383c
**/*.git
**/.mypy_cache
**/__pycache__
**/*.pyc
**/NOMAD.egg-info
.pyenv/
.vscode/
.volumes/
.git/
.mypy_cache/
dependencies/**/test
dependencies/**/tests
data/
docs/.build
docs/*.graffle
gui/
infrastructure/
__pycache__/
*.pyc
NOMAD.egg-info/
.coverage
examples/
local/
......
.DS_Store
.pyenv/
.env/
.ipynb_checkpoints/
__pycache__
.mypy_cache
*.pyc
......
......@@ -153,3 +153,6 @@
path = dependencies/parsers/onetep
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-onetep.git
branch = nomad-fair
[submodule "dependencies/optimade-python-tools"]
path = dependencies/optimade-python-tools
url = https://github.com/markus1978/optimade-python-tools.git
......@@ -78,10 +78,20 @@ your browser.
## Change log
Omitted versions are plain bugfix releases with only minor changes and fixes.
### v0.6.0
### v0.7.0
- Keycloak based user management
- no dependencies with the NOMAD CeE Repository
### v0.6.0
- GUI URL, and API endpoint that resolves NOMAD CoE legacy PIDs
- Support for datasets in the GUI
- more flexible search python module and repo API
- support for external_id
- support for code-based raw_id
- Optimade API 0.10.0
- GUI supports Optimade filter query and other quantities
- minor bugfixes
### v0.5.2
- allows to download large files over longer time period
- streamlined deployment without API+GUI proxy
......
......@@ -6,6 +6,7 @@ git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
while read path_key path
do
echo $path
[ -f $path/requirements.txt ] && pip install -r $path/requirements.txt
[ -f $path/setup.py ] && pip install $1 $path
(echo "$path" | grep -vEq ^dependencies/optimade-python-tools$) \
&& [ -f $path/requirements.txt ] && pip install -r $path/requirements.txt
[ -f $path/setup.py ] && pip install --ignore-requires-python $1 $path
done
Subproject commit 43d77d198acfc3137d1c4d08a4601248fbf8548d
Subproject commit c36949f0c8421fff340d314d16ee83f7da5974ac
Subproject commit b9619d6b34a8f8e66120fb02f5fd3dbc16d26517
Subproject commit ccbf641ab7a0930c5f18507147f6c5b51f4e7444
Subproject commit d73611bc1b16ea71daa3d0fd24ee275c78853557
Subproject commit 67e781f7288b7e87da0a25ec02bf33d12de436f5
Subproject commit 0d5765ded1e24c5dd22be98c21d6c9284d06ec51
Subproject commit 039ed6cb532b26926f8e0d7dc2027403e965e67c
Subproject commit dc59aff14e69a6818fbefeb7b1d504348438e26b
Subproject commit a1cba85370ad5923969ac9ceb3643a56b3c2e7d9
Subproject commit aee4be7407124f87b0ba99eb7b4af3646b8602e9
API Documentation
=================
API(s) Documentation
====================
Summary
-------
.. qrefflask:: nomad.api:app
.. qrefflask:: nomad.app:app
:undoc-static:
API Details
-----------
API(s) Details
--------------
.. autoflask:: nomad.api:app
.. autoflask:: nomad.app:app
:undoc-static:
......@@ -12,6 +12,7 @@ and infrastructure with a simplyfied architecture and consolidated code base.
dev_guidelines
api_tutorial
api
ops
metainfo
parser_tutorial
reference
ops
......@@ -122,7 +122,7 @@ The component library [Material-UI](https://material-ui.com/)
### docker
To run a **nomad@FAIRDI** instance, many services have to be orchestrated:
the nomad api, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
the nomad app, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together.
Further services might be needed (e.g. JypiterHUB), when nomad grows.
The container platform [Docker](https://docs.docker.com/) allows us to provide all services
......
Metainfo
========
.. automodule:: nomad.metainfo
Reference
=========
nomad.metainfo
--------------
.. automodule:: nomad.metainfo.metainfo
nomad.config
------------
.. automodule:: nomad.config
......@@ -38,9 +42,9 @@ nomad.search
.. automodule:: nomad.search
:members:
nomad.api
nomad.app
---------
.. automodule:: nomad.api
.. automodule:: nomad.app
nomad.cli
------------
......
......@@ -3,7 +3,7 @@
## Introduction
The nomad infrastructure consists of a series of nomad and 3rd party services:
- nomad worker (python): task worker that will do the processing
- nomad api (python): the nomad REST API
- nomad app (python): the nomad app and it's REST APIs
- nomad gui: a small server serving the web-based react gui
- proxy: an nginx server that reverse proxyies all services under one port
- elastic search: nomad's search and analytics engine
......@@ -34,11 +34,30 @@ git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-FAIR.git
cd nomad-FAIR
```
### C libs
Even though the NOMAD infrastructure is written in python, there is a C library
required by one of our pyhton dependencies.
#### libmagic
Libmagic allows to determine the MIME type of files. It should be installed on most
unix/linux systems. It can be installed on MacOS with homebrew:
```
brew install libmagic
```
### Virtual environment
#### pyenv
The nomad code currently targets python 3.6. If you host machine has 3.7 or later installed,
you can use [pyenv](https://github.com/pyenv/pyenv) to use python 3.6 in parallel.
While in principle everything should be compatable with 3.7 and later there have been
issues with some dependencies and requirements not being compatible with 3.7
To use 3.7 there is a slight issue about the `enum34` which fails the compilation of the
`mdtraj` and `mdanalysis` packages. A possible work arround is to uninstall and tham re-install
`enum34` once the other packages are installed.
#### virtualenv
We strongly recommend to use *virtualenv* to create a virtual environment. It will allow you
to keep nomad and its dependencies separate from your system's python installation.
Make sure to base the virtual environment on Python 3.
......@@ -49,6 +68,30 @@ virtualenv -p `which python3` .pyenv
source .pyenv/bin/activate
```
#### Conda
If you are a conda user, there is an equivalent, but you have to install pip and the
right python version while creating the environment.
```
conda create --name nomad_env pip python=3.6
conda activate nomad_env
```
To install libmagick for conda, you can use (other channels might also work):
```
conda -c conda-forge install --name nomad_env libmagic
```
The next steps can be done using the `setup.sh` script. If you prefere to understand all
the steps and run them manually, read on:
### Get all the submodules
We use git submodules to retrieve all the other NOMAD repositories, mainly parsers.
```
git submodules update --init
```
### Install python dependencies
We use *pip* to manage required python packages.
```
pip install -r requirements.txt
......@@ -56,19 +99,22 @@ pip install -r requirements.txt
### Install NOMAD-coe dependencies.
Nomad is based on python modules from the NOMAD-coe project.
This includes parsers, normalizers, python-common and the meta-info.
Those dependencies are managed and configured via python in
`nomad/dependencies.py`. This gives us more flexibility in interacting with
different parser, normalizer versions from within the running nomad infrastructure.
This includes parsers, python-common and the meta-info. These modules are maintained as
their own GITLab/git repositories. To clone and initialize them run:
To run the dependencies script and install all dependencies into your environment:
```
./dependencies.sh
git submodules update --init
```
This will checkout the proper version of the respective NOMAD-coe modules, install
further requirements, and install the modules themselves. The `-e` option will install
the NOMAD-coe dependencies with symbolic links allowing you to change the downloaded
dependency code without having to reinstall after.
All requirements for these submodules need to be installed and they need to be installed
themselves as python modules. Run the `dependencies.sh` script that will install
everything into your virtual environment:
```
./dependencies.sh -e
```
The `-e` option will install the NOMAD-coe dependencies with symbolic links allowing you
to change the downloaded dependency code without having to reinstall after.
### Install nomad
Finally, you can add nomad to the environment itself.
......@@ -94,7 +140,7 @@ There are currently two different images and respectively two different docker f
`Dockerfile`, and `gui/Dockerfile`.
Nomad comprises currently two services,
the *worker* (does the actual processing), and the *api*. Those services can be
the *worker* (does the actual processing), and the *app*. Those services can be
run from one image that have the nomad python code and all dependencies installed. This
is covered by the `Dockerfile`.
......@@ -117,21 +163,14 @@ The images are build via *docker-compose* and don't have to be created manually.
We have multiple *docker-compose* files that must be used together.
- `docker-compose.yml` contains the base definitions for all services
- `docker-compose.override.yml` configures services for development (notably builds images for nomad services)
- `docker-compose.dev-elk.yml` will also provide the ELK service
- `docker-compose.prod.yml` configures services for production (notable uses a pre-build image for nomad services that was build during CI/CD)
It is sufficient to use the implicit `docker-compose.yml` only (like in the command below).
The `override` will be used automatically.
There is also an `.env` file. For development you can use `.env_development`:
```
cd ./ops/docker-compose/nomad
ln -s .env_development .env
```
The production `.env` file is stored on our serves and not part of the source code.
Now we can build the *docker-compose* that contains all external services (rabbitmq,
mongo, elastic, elk) and nomad services (worker, api, gui).
mongo, elastic, elk) and nomad services (worker, app, gui).
```
docker-compose build
```
......@@ -159,7 +198,7 @@ docker-compose down
### Run containers selectively
The following services/containers are managed via our docker-compose:
- rabbitmq, mongo, elastic, (elk, only for production)
- worker, api
- worker, app
- gui
- proxy
......@@ -168,33 +207,27 @@ a single port and different paths.
You can also run services selectively, e.g.
```
docker-compose up -d rabbitmq, mongo, elastic, elk
docker-compose up -d rabbitmq, mongo, elastic
docker-compose up worker
docker-compose up api gui proxy
docker-compose up app gui proxy
```
### Configure the containers
The *docker-compose* takes some configuration from the environment. Environment
variables are set in `.env`, which is a link to either `.env_local` (intended to run
nomad for development on a local computer) and `.env_processing` (indented to run
nomad on nomad in *'production'*, currently on the enc pre-processing machine).
You can configure host ports, volume locations for host bindings, and these sort of things.
## Accessing 3'rd party services
Usually these services only used by the nomad containers, but sometimes you also
need to check something or do some manual steps.
The file `ops/docker-compose/nomad/.env` contains variables that control the ports
used to bind internal docker ports to your host machine. These are the ports you
have to use to connect to the respective services.
The *docker-compose* can be overriden with additional seetings. See documentation section on
operating NOMAD for more details. The override `docker-compose.override.yml` will
expose all database ports to the hostmachine and should be used in development. To use
it run docker-compose with `-f docker-compose.yml -f docker-compose.override.yml`.
### ELK (elastic stack)
If you run the ELK stack (and enable logstash in nomad/config.py),
you can reach the Kibana with [localhost:5601](http://localhost:5601).
The index prefix for logs is `logstash-`.
The index prefix for logs is `logstash-`. The ELK is only available with the
`docker-compose.dev-elk.yml` override.
### mongodb and elastic search
......@@ -204,48 +237,33 @@ to use the right ports (see above).
## Run nomad services manually
You can run the worker, api, and gui as part of the docker infrastructure, like
You can run the worker, app, and gui as part of the docker infrastructure, like
seen above. But, of course there are always reasons to run them manually during
development, like running them in a debugger, profiler, etc.
### Run the nomad worker manually
### API and worker
To simply run a worker with the installed nomad cli, do (from the root)
```
nomad run worker
nomad admin run worker
```
To run it manually with celery, do (from the root)
To run it directly with celery, do (from the root)
```
celery -A nomad.processing worker -l info
```
You can use different debug level (e.g. switch `info` to `debug`)
Use watchdog during development to reload the worker on code changes.
Watchdog is part of the requirements-dev.txt. For MacOS (there is currently a bug in watchdog)
uninstall and install this [fixed](https://github.com/gorakhargosh/watchdog/issues/330) version
Run the app via docker, or (from the root):
```
pip uninstall watchdog
pip install git+https://github.com/gorakhargosh/watchdog.git
nomad admin run app
```
Now use this to auto relead worker:
You can also run worker and app together:
```
watchmedo auto-restart -d ./nomad -p '*.py' -- celery worker -l info -A nomad.processing
nomad admin run appworker
```
### Run the api
Either with docker, or:
```
nomad run api
```
Or manually:
```
python nomad/api.py
```
### Run the gui
### GUI
When you run the gui on its own (e.g. with react dev server below), you have to have
the API running manually also. This *inside docker* API is configured for ngingx paths
and proxies, which are run by the gui container. But you can run the *production* gui
......@@ -270,12 +288,131 @@ pytest -svx tests
We use pylint, pycodestyle, and mypy to ensure code quality. To run those:
```
nomad qa --skip-test
nomad dev qa --skip-test
```
To run all tests and code qa:
```
nomad qa
nomad dev qa
```
This mimiques the tests and checks that the GitLab CI/CD will perform.
## Setup your (I)DE
The documentation section on development guidelines details how the code is organized,
tested, formatted, and documented. To help you meet these guidelines, we recomment to
use a proper IDE for development and ditch any VIM/Emacs (mal-)practices.
### Visual Studio Code
Here are some VSCode settings that will enable features for linting, some auto formating,
line size ruler, etc.
```json
{
"python.venvPath": "${workspaceFolder}/.pyenv",
"python.pythonPath": "${workspaceFolder}/.pyenv/bin/python",
"git.ignoreLimitWarning": true,
"editor.rulers": [90],
"editor.renderWhitespace": "all",
"editor.tabSize": 4,
"[javascript]": {
"editor.tabSize": 2
},
"files.trimTrailingWhitespace": true,
"git.enableSmartCommit": true,
"eslint.autoFixOnSave": true,
"python.linting.pylintArgs": [
"--load-plugins=pylint_mongoengine",
],
"python.linting.pep8Path": "pycodestyle",
"python.linting.pep8Enabled": true,
"python.linting.pep8Args": ["--ignore=E501,E701"],
"python.linting.mypyEnabled": true,
"python.linting.mypyArgs": [
"--ignore-missing-imports",
"--follow-imports=silent",
"--no-strict-optional"
],
"workbench.colorCustomizations": {
"editorError.foreground": "#FF2222",
"editorOverviewRuler.errorForeground": "#FF2222",
"editorWarning.foreground": "#FF5500",
"editorOverviewRuler.warningForeground": "#FF5500",
"activityBar.background": "#4D2111",
"titleBar.activeBackground": "#6B2E18",
"titleBar.activeForeground": "#FDF9F7"
},
"files.watcherExclude": {
"**/.git/objects/**": true,
"**/.git/subtree-cache/**": true,
"**/node_modules/*/**": true,
"**/.pyenv/*/**": true,
"**/__pycache__/*/**": true,
"**/.mypy_cache/*/**": true,
"**/.volumes/*/**": true,
"**/docs/.build/*/**": true
}
}
```
Here are some example launch configs for VSCode:
```json
{
"version": "0.2.0",
"configurations": [
{
"type": "chrome",
"request": "launch",
"name": "Launch Chrome against localhost",
"url": "http://localhost:3000",
"webRoot": "${workspaceFolder}/gui"
},
{
"name": "Python: API Flask (0.11.x or later)",
"type": "python",
"request": "launch",
"module": "flask",
"env": {
"FLASK_APP": "nomad/app/__init__.py"
},
"args": [
"run",
"--port",
"8000",
"--no-debugger",
"--no-reload"
]
},
{
"name": "Python: some test",
"type": "python",
"request": "launch",
"cwd": "${workspaceFolder}",
"program": "${workspaceFolder}/.pyenv/bin/pytest",
"args": [
"-sv",
"tests/test_cli.py::TestClient::test_mirror"
]
},
{
"name": "Python: Current File",
"type": "python",
"request": "launch",
"program": "${file}"
},
{
"name": "Python: Attach",
"type": "python",
"request": "attach",
"localRoot": "${workspaceFolder}",
"remoteRoot": "${workspaceFolder}",
"port": 3000,
"secret": "my_secret",
"host": "localhost"
}
]
}
```
\ No newline at end of file
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment