Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
Menu
Open sidebar
nomad-lab
nomad-FAIR
Commits
1f8bb5ae
Commit
1f8bb5ae
authored
Oct 16, 2019
by
Markus Scheidgen
Browse files
Merged v0.6.1.
parents
e88cdbfe
5742383c
Changes
141
Hide whitespace changes
Inline
Side-by-side
.dockerignore
View file @
1f8bb5ae
**/*.git
**/.mypy_cache
**/__pycache__
**/*.pyc
**/NOMAD.egg-info
.pyenv/
.vscode/
.volumes/
.git/
.mypy_cache/
dependencies/**/test
dependencies/**/tests
data/
docs/.build
docs/*.graffle
gui/
infrastructure/
__pycache__/
*.pyc
NOMAD.egg-info/
.coverage
examples/
local/
...
...
.gitignore
View file @
1f8bb5ae
.DS_Store
.pyenv/
.env/
.ipynb_checkpoints/
__pycache__
.mypy_cache
*.pyc
...
...
.gitmodules
View file @
1f8bb5ae
...
...
@@ -153,3 +153,6 @@
path = dependencies/parsers/onetep
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-onetep.git
branch = nomad-fair
[submodule "dependencies/optimade-python-tools"]
path = dependencies/optimade-python-tools
url = https://github.com/markus1978/optimade-python-tools.git
.python-version
0 → 100644
View file @
1f8bb5ae
3.6.9
README.md
View file @
1f8bb5ae
...
...
@@ -78,10 +78,20 @@ your browser.
## Change log
Omitted versions are plain bugfix releases with only minor changes and fixes.
### v0.
6
.0
### v0.
7
.0
-
Keycloak based user management
-
no dependencies with the NOMAD CeE Repository
### v0.6.0
-
GUI URL, and API endpoint that resolves NOMAD CoE legacy PIDs
-
Support for datasets in the GUI
-
more flexible search python module and repo API
-
support for external_id
-
support for code-based raw_id
-
Optimade API 0.10.0
-
GUI supports Optimade filter query and other quantities
-
minor bugfixes
### v0.5.2
-
allows to download large files over longer time period
-
streamlined deployment without API+GUI proxy
...
...
dependencies.sh
View file @
1f8bb5ae
...
...
@@ -6,6 +6,7 @@ git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
while
read
path_key path
do
echo
$path
[
-f
$path
/requirements.txt
]
&&
pip
install
-r
$path
/requirements.txt
[
-f
$path
/setup.py
]
&&
pip
install
$1
$path
(
echo
"
$path
"
|
grep
-vEq
^dependencies/optimade-python-tools
$)
\
&&
[
-f
$path
/requirements.txt
]
&&
pip
install
-r
$path
/requirements.txt
[
-f
$path
/setup.py
]
&&
pip
install
--ignore-requires-python
$1
$path
done
nomad-meta-info
@
c36949f0
Compare
43d77d19
...
c36949f0
Subproject commit
43d77d198acfc3137d1c4d08a4601248fbf8548d
Subproject commit
c36949f0c8421fff340d314d16ee83f7da5974ac
optimade-python-tools
@
b9619d6b
Subproject commit b9619d6b34a8f8e66120fb02f5fd3dbc16d26517
dependencies/parsers/all_test_files.zip
0 → 100644
View file @
1f8bb5ae
File added
exciting
@
d73611bc
Compare
ccbf641a
...
d73611bc
Subproject commit
ccbf641ab7a0930c5f18507147f6c5b51f4e7444
Subproject commit
d73611bc1b16ea71daa3d0fd24ee275c78853557
fhi-aims
@
0d5765de
Compare
67e781f7
...
0d5765de
Subproject commit
67e781f7288b7e87da0a25ec02bf33d12de436f5
Subproject commit
0d5765ded1e24c5dd22be98c21d6c9284d06ec51
vasp
@
dc59aff1
Compare
039ed6cb
...
dc59aff1
Subproject commit
039ed6cb532b26926f8e0d7dc2027403e965e67c
Subproject commit
dc59aff14e69a6818fbefeb7b1d504348438e26b
python_common
@
aee4be74
Compare
a1cba853
...
aee4be74
Subproject commit a
1cba85370ad5923969ac9ceb3643a56b3c2e7d
9
Subproject commit a
ee4be7407124f87b0ba99eb7b4af3646b8602e
9
docs/api.rst
View file @
1f8bb5ae
API Documentation
=================
API
(s)
Documentation
=================
===
Summary
-------
.. qrefflask:: nomad.ap
i
:app
.. qrefflask:: nomad.ap
p
:app
:undoc-static:
API Details
-----------
API
(s)
Details
-----------
---
.. autoflask:: nomad.ap
i
:app
.. autoflask:: nomad.ap
p
:app
:undoc-static:
docs/index.rst
View file @
1f8bb5ae
...
...
@@ -12,6 +12,7 @@ and infrastructure with a simplyfied architecture and consolidated code base.
dev_guidelines
api_tutorial
api
ops
metainfo
parser_tutorial
reference
ops
docs/introduction.md
View file @
1f8bb5ae
...
...
@@ -122,7 +122,7 @@ The component library [Material-UI](https://material-ui.com/)
### docker
To run a
**nomad@FAIRDI**
instance, many services have to be orchestrated:
the nomad ap
i
, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
the nomad ap
p
, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together.
Further services might be needed (e.g. JypiterHUB), when nomad grows.
The container platform
[
Docker
](
https://docs.docker.com/
)
allows us to provide all services
...
...
docs/metainfo.rst
0 → 100644
View file @
1f8bb5ae
Metainfo
========
.. automodule:: nomad.metainfo
docs/reference.rst
View file @
1f8bb5ae
Reference
=========
nomad.metainfo
--------------
.. automodule:: nomad.metainfo.metainfo
nomad.config
------------
.. automodule:: nomad.config
...
...
@@ -38,9 +42,9 @@ nomad.search
.. automodule:: nomad.search
:members:
nomad.ap
i
nomad.ap
p
---------
.. automodule:: nomad.ap
i
.. automodule:: nomad.ap
p
nomad.cli
------------
...
...
docs/setup.md
View file @
1f8bb5ae
...
...
@@ -3,7 +3,7 @@
## Introduction
The nomad infrastructure consists of a series of nomad and 3rd party services:
-
nomad worker (python): task worker that will do the processing
-
nomad ap
i
(python): the nomad REST API
-
nomad ap
p
(python): the nomad
app and it's
REST API
s
-
nomad gui: a small server serving the web-based react gui
-
proxy: an nginx server that reverse proxyies all services under one port
-
elastic search: nomad's search and analytics engine
...
...
@@ -34,11 +34,30 @@ git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-FAIR.git
cd nomad-FAIR
```
### C libs
Even though the NOMAD infrastructure is written in python, there is a C library
required by one of our pyhton dependencies.
#### libmagic
Libmagic allows to determine the MIME type of files. It should be installed on most
unix/linux systems. It can be installed on MacOS with homebrew:
```
brew install libmagic
```
### Virtual environment
#### pyenv
The nomad code currently targets python 3.6. If you host machine has 3.7 or later installed,
you can use
[
pyenv
](
https://github.com/pyenv/pyenv
)
to use python 3.6 in parallel.
While in principle everything should be compatable with 3.7 and later there have been
issues with some dependencies and requirements not being compatible with 3.7
To use 3.7 there is a slight issue about the
`enum34`
which fails the compilation of the
`mdtraj`
and
`mdanalysis`
packages. A possible work arround is to uninstall and tham re-install
`enum34`
once the other packages are installed.
#### virtualenv
We strongly recommend to use
*virtualenv*
to create a virtual environment. It will allow you
to keep nomad and its dependencies separate from your system's python installation.
Make sure to base the virtual environment on Python 3.
...
...
@@ -49,6 +68,30 @@ virtualenv -p `which python3` .pyenv
source .pyenv/bin/activate
```
#### Conda
If you are a conda user, there is an equivalent, but you have to install pip and the
right python version while creating the environment.
```
conda create --name nomad_env pip python=3.6
conda activate nomad_env
```
To install libmagick for conda, you can use (other channels might also work):
```
conda -c conda-forge install --name nomad_env libmagic
```
The next steps can be done using the
`setup.sh`
script. If you prefere to understand all
the steps and run them manually, read on:
### Get all the submodules
We use git submodules to retrieve all the other NOMAD repositories, mainly parsers.
```
git submodules update --init
```
### Install python dependencies
We use
*pip*
to manage required python packages.
```
pip install -r requirements.txt
...
...
@@ -56,19 +99,22 @@ pip install -r requirements.txt
### Install NOMAD-coe dependencies.
Nomad is based on python modules from the NOMAD-coe project.
This includes parsers, normalizers, python-common and the meta-info.
Those dependencies are managed and configured via python in
`nomad/dependencies.py`
. This gives us more flexibility in interacting with
different parser, normalizer versions from within the running nomad infrastructure.
This includes parsers, python-common and the meta-info. These modules are maintained as
their own GITLab/git repositories. To clone and initialize them run:
To run the dependencies script and install all dependencies into your environment:
```
./dependencies.sh
git submodules update --init
```
This will checkout the proper version of the respective NOMAD-coe modules, install
further requirements, and install the modules themselves. The
`-e`
option will install
the NOMAD-coe dependencies with symbolic links allowing you to change the downloaded
dependency code without having to reinstall after.
All requirements for these submodules need to be installed and they need to be installed
themselves as python modules. Run the
`dependencies.sh`
script that will install
everything into your virtual environment:
```
./dependencies.sh -e
```
The
`-e`
option will install the NOMAD-coe dependencies with symbolic links allowing you
to change the downloaded dependency code without having to reinstall after.
### Install nomad
Finally, you can add nomad to the environment itself.
...
...
@@ -94,7 +140,7 @@ There are currently two different images and respectively two different docker f
`Dockerfile`
, and
`gui/Dockerfile`
.
Nomad comprises currently two services,
the
*worker*
(does the actual processing), and the
*ap
i
*
. Those services can be
the
*worker*
(does the actual processing), and the
*ap
p
*
. Those services can be
run from one image that have the nomad python code and all dependencies installed. This
is covered by the
`Dockerfile`
.
...
...
@@ -117,21 +163,14 @@ The images are build via *docker-compose* and don't have to be created manually.
We have multiple
*docker-compose*
files that must be used together.
-
`docker-compose.yml`
contains the base definitions for all services
-
`docker-compose.override.yml`
configures services for development (notably builds images for nomad services)
-
`docker-compose.dev-elk.yml`
will also provide the ELK service
-
`docker-compose.prod.yml`
configures services for production (notable uses a pre-build image for nomad services that was build during CI/CD)
It is sufficient to use the implicit
`docker-compose.yml`
only (like in the command below).
The
`override`
will be used automatically.
There is also an
`.env`
file. For development you can use
`.env_development`
:
```
cd ./ops/docker-compose/nomad
ln -s .env_development .env
```
The production
`.env`
file is stored on our serves and not part of the source code.
Now we can build the
*docker-compose*
that contains all external services (rabbitmq,
mongo, elastic, elk) and nomad services (worker, ap
i
, gui).
mongo, elastic, elk) and nomad services (worker, ap
p
, gui).
```
docker-compose build
```
...
...
@@ -159,7 +198,7 @@ docker-compose down
### Run containers selectively
The following services/containers are managed via our docker-compose:
-
rabbitmq, mongo, elastic, (elk, only for production)
-
worker, ap
i
-
worker, ap
p
-
gui
-
proxy
...
...
@@ -168,33 +207,27 @@ a single port and different paths.
You can also run services selectively, e.g.
```
docker-compose up -d rabbitmq, mongo, elastic
, elk
docker-compose up -d rabbitmq, mongo, elastic
docker-compose up worker
docker-compose up ap
i
gui proxy
docker-compose up ap
p
gui proxy
```
### Configure the containers
The
*docker-compose*
takes some configuration from the environment. Environment
variables are set in
`.env`
, which is a link to either
`.env_local`
(intended to run
nomad for development on a local computer) and
`.env_processing`
(indented to run
nomad on nomad in
*'production'*
, currently on the enc pre-processing machine).
You can configure host ports, volume locations for host bindings, and these sort of things.
## Accessing 3'rd party services
Usually these services only used by the nomad containers, but sometimes you also
need to check something or do some manual steps.
The file
`ops/docker-compose/nomad/.env`
contains variables that control the ports
used to bind internal docker ports to your host machine. These are the ports you
have to use to connect to the respective services.
The
*docker-compose*
can be overriden with additional seetings. See documentation section on
operating NOMAD for more details. The override
`docker-compose.override.yml`
will
expose all database ports to the hostmachine and should be used in development. To use
it run docker-compose with
`-f docker-compose.yml -f docker-compose.override.yml`
.
### ELK (elastic stack)
If you run the ELK stack (and enable logstash in nomad/config.py),
you can reach the Kibana with
[
localhost:5601
](
http://localhost:5601
)
.
The index prefix for logs is
`logstash-`
.
The index prefix for logs is
`logstash-`
. The ELK is only available with the
`docker-compose.dev-elk.yml`
override.
### mongodb and elastic search
...
...
@@ -204,48 +237,33 @@ to use the right ports (see above).
## Run nomad services manually
You can run the worker, ap
i
, and gui as part of the docker infrastructure, like
You can run the worker, ap
p
, and gui as part of the docker infrastructure, like
seen above. But, of course there are always reasons to run them manually during
development, like running them in a debugger, profiler, etc.
###
Run the nomad worker manually
###
API and worker
To simply run a worker with the installed nomad cli, do (from the root)
```
nomad run worker
nomad
admin
run worker
```
To run it
manual
ly with celery, do (from the root)
To run it
direct
ly with celery, do (from the root)
```
celery -A nomad.processing worker -l info
```
You can use different debug level (e.g. switch
`info`
to
`debug`
)
Use watchdog during development to reload the worker on code changes.
Watchdog is part of the requirements-dev.txt. For MacOS (there is currently a bug in watchdog)
uninstall and install this
[
fixed
](
https://github.com/gorakhargosh/watchdog/issues/330
)
version
Run the app via docker, or (from the root):
```
pip uninstall watchdog
pip install git+https://github.com/gorakhargosh/watchdog.git
nomad admin run app
```
Now use this to auto relead work
er:
You can also run worker and app togeth
er:
```
watchmedo auto-restart -d ./nomad -p '*.py' -- celery worker -l info -A nomad.processing
nomad admin run appworker
```
### Run the api
Either with docker, or:
```
nomad run api
```
Or manually:
```
python nomad/api.py
```
### Run the gui
### GUI
When you run the gui on its own (e.g. with react dev server below), you have to have
the API running manually also. This
*inside docker*
API is configured for ngingx paths
and proxies, which are run by the gui container. But you can run the
*production*
gui
...
...
@@ -270,12 +288,131 @@ pytest -svx tests
We use pylint, pycodestyle, and mypy to ensure code quality. To run those:
```
nomad qa --skip-test
nomad
dev
qa --skip-test
```
To run all tests and code qa:
```
nomad qa
nomad
dev
qa
```
This mimiques the tests and checks that the GitLab CI/CD will perform.
## Setup your (I)DE
The documentation section on development guidelines details how the code is organized,
tested, formatted, and documented. To help you meet these guidelines, we recomment to
use a proper IDE for development and ditch any VIM/Emacs (mal-)practices.
### Visual Studio Code
Here are some VSCode settings that will enable features for linting, some auto formating,
line size ruler, etc.
```
json
{
"python.venvPath"
:
"${workspaceFolder}/.pyenv"
,
"python.pythonPath"
:
"${workspaceFolder}/.pyenv/bin/python"
,
"git.ignoreLimitWarning"
:
true
,
"editor.rulers"
:
[
90
],
"editor.renderWhitespace"
:
"all"
,
"editor.tabSize"
:
4
,
"[javascript]"
:
{
"editor.tabSize"
:
2
},
"files.trimTrailingWhitespace"
:
true
,
"git.enableSmartCommit"
:
true
,
"eslint.autoFixOnSave"
:
true
,
"python.linting.pylintArgs"
:
[
"--load-plugins=pylint_mongoengine"
,
],
"python.linting.pep8Path"
:
"pycodestyle"
,
"python.linting.pep8Enabled"
:
true
,
"python.linting.pep8Args"
:
[
"--ignore=E501,E701"
],
"python.linting.mypyEnabled"
:
true
,
"python.linting.mypyArgs"
:
[
"--ignore-missing-imports"
,
"--follow-imports=silent"
,
"--no-strict-optional"
],
"workbench.colorCustomizations"
:
{
"editorError.foreground"
:
"#FF2222"
,
"editorOverviewRuler.errorForeground"
:
"#FF2222"
,
"editorWarning.foreground"
:
"#FF5500"
,
"editorOverviewRuler.warningForeground"
:
"#FF5500"
,
"activityBar.background"
:
"#4D2111"
,
"titleBar.activeBackground"
:
"#6B2E18"
,
"titleBar.activeForeground"
:
"#FDF9F7"
},
"files.watcherExclude"
:
{
"**/.git/objects/**"
:
true
,
"**/.git/subtree-cache/**"
:
true
,
"**/node_modules/*/**"
:
true
,
"**/.pyenv/*/**"
:
true
,
"**/__pycache__/*/**"
:
true
,
"**/.mypy_cache/*/**"
:
true
,
"**/.volumes/*/**"
:
true
,
"**/docs/.build/*/**"
:
true
}
}
```
Here are some example launch configs for VSCode:
```
json
{
"version"
:
"0.2.0"
,
"configurations"
:
[
{
"type"
:
"chrome"
,
"request"
:
"launch"
,
"name"
:
"Launch Chrome against localhost"
,
"url"
:
"http://localhost:3000"
,
"webRoot"
:
"${workspaceFolder}/gui"
},
{
"name"
:
"Python: API Flask (0.11.x or later)"
,
"type"
:
"python"
,
"request"
:
"launch"
,
"module"
:
"flask"
,
"env"
:
{
"FLASK_APP"
:
"nomad/app/__init__.py"
},
"args"
:
[
"run"
,
"--port"
,
"8000"
,
"--no-debugger"
,
"--no-reload"
]
},
{
"name"
:
"Python: some test"
,
"type"
:
"python"
,
"request"
:
"launch"
,
"cwd"
:
"${workspaceFolder}"
,
"program"
:
"${workspaceFolder}/.pyenv/bin/pytest"
,
"args"
:
[
"-sv"
,
"tests/test_cli.py::TestClient::test_mirror"
]
},
{
"name"
:
"Python: Current File"
,
"type"
:
"python"
,
"request"
:
"launch"
,
"program"
:
"${file}"
},
{
"name"
:
"Python: Attach"
,
"type"
:
"python"
,
"request"
:
"attach"
,
"localRoot"
:
"${workspaceFolder}"
,
"remoteRoot"
:
"${workspaceFolder}"
,
"port"
:
3000
,
"secret"
:
"my_secret"
,
"host"
:
"localhost"
}
]
}
```
\ No newline at end of file
examples/external_project_parallel_upload/example-1.tar.gz
0 → 100644
View file @
1f8bb5ae
File added
Prev
1
2
3
4
5
…
8
Next
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment