Merged v0.6.1.

.dockerignore

+**/*.git
+**/.mypy_cache
+**/__pycache__
+**/*.pyc
+**/NOMAD.egg-info
 .pyenv/
 .vscode/
 .volumes/
-.git/
-.mypy_cache/
+dependencies/**/test
+dependencies/**/tests
 data/
 docs/.build
 docs/*.graffle
-gui/
-infrastructure/
-__pycache__/
-*.pyc
-NOMAD.egg-info/
 .coverage
 examples/
 local/

.gitignore

 .DS_Store
 .pyenv/
+.env/
+.ipynb_checkpoints/
 __pycache__
 .mypy_cache
 *.pyc

.gitmodules

@@ -153,3 +153,6 @@
 	path = dependencies/parsers/onetep
 	url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-onetep.git
 	branch = nomad-fair
+[submodule "dependencies/optimade-python-tools"]
+	path = dependencies/optimade-python-tools
+	url = https://github.com/markus1978/optimade-python-tools.git

.python-version

+3.6.9

README.md

@@ -78,10 +78,20 @@ your browser.
 ## Change log
 Omitted versions are plain bugfix releases with only minor changes and fixes.
 
-### v0.6.0
+### v0.7.0
 - Keycloak based user management
 - no dependencies with the NOMAD CeE Repository
 
+### v0.6.0
+- GUI URL, and API endpoint that resolves NOMAD CoE legacy PIDs
+- Support for datasets in the GUI
+- more flexible search python module and repo API
+- support for external_id
+- support for code-based raw_id
+- Optimade API 0.10.0
+- GUI supports Optimade filter query and other quantities
+- minor bugfixes
+
 ### v0.5.2
 - allows to download large files over longer time period
 - streamlined deployment without API+GUI proxy

dependencies.sh

@@ -6,6 +6,7 @@ git config -f .gitmodules --get-regexp '^submodule\..*\.path$' |
     while read path_key path
     do
         echo $path
-        [ -f $path/requirements.txt ] && pip install -r $path/requirements.txt
-        [ -f $path/setup.py ] && pip install $1 $path
+        (echo "$path" | grep -vEq  ^dependencies/optimade-python-tools$) \
+            && [ -f $path/requirements.txt ] && pip install -r $path/requirements.txt
+        [ -f $path/setup.py ] && pip install --ignore-requires-python $1 $path
     done

nomad-meta-info @ c36949f0

-Subproject commit 43d77d198acfc3137d1c4d08a4601248fbf8548d
+Subproject commit c36949f0c8421fff340d314d16ee83f7da5974ac

optimade-python-tools @ b9619d6b

+Subproject commit b9619d6b34a8f8e66120fb02f5fd3dbc16d26517

exciting @ d73611bc

-Subproject commit ccbf641ab7a0930c5f18507147f6c5b51f4e7444
+Subproject commit d73611bc1b16ea71daa3d0fd24ee275c78853557

fhi-aims @ 0d5765de

-Subproject commit 67e781f7288b7e87da0a25ec02bf33d12de436f5
+Subproject commit 0d5765ded1e24c5dd22be98c21d6c9284d06ec51

vasp @ dc59aff1

-Subproject commit 039ed6cb532b26926f8e0d7dc2027403e965e67c
+Subproject commit dc59aff14e69a6818fbefeb7b1d504348438e26b

python_common @ aee4be74

-Subproject commit a1cba85370ad5923969ac9ceb3643a56b3c2e7d9
+Subproject commit aee4be7407124f87b0ba99eb7b4af3646b8602e9

docs/api.rst

-API Documentation
-=================
+API(s) Documentation
+====================
 
 
 Summary
 -------
 
-.. qrefflask:: nomad.api:app
+.. qrefflask:: nomad.app:app
   :undoc-static:
 
 
 
-API Details
------------
+API(s) Details
+--------------
 
-.. autoflask:: nomad.api:app
+.. autoflask:: nomad.app:app
   :undoc-static:

docs/index.rst

@@ -12,6 +12,7 @@ and infrastructure with a simplyfied architecture and consolidated code base.
    dev_guidelines
    api_tutorial
    api
-   ops
+   metainfo
    parser_tutorial
    reference
+   ops

docs/introduction.md

@@ -122,7 +122,7 @@ The component library [Material-UI](https://material-ui.com/)
 ### docker
 
 To run a **nomad@FAIRDI** instance, many services have to be orchestrated:
-the nomad api, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
+the nomad app, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
 Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together.
 Further services might be needed (e.g. JypiterHUB), when nomad grows.
 The container platform [Docker](https://docs.docker.com/) allows us to provide all services

docs/metainfo.rst

+Metainfo
+========
+
+.. automodule:: nomad.metainfo

docs/reference.rst

 Reference
 =========
 
+nomad.metainfo
+--------------
+.. automodule:: nomad.metainfo.metainfo
+
 nomad.config
 ------------
 .. automodule:: nomad.config
@@ -38,9 +42,9 @@ nomad.search
 .. automodule:: nomad.search
     :members:
 
-nomad.api
+nomad.app
 ---------
-.. automodule:: nomad.api
+.. automodule:: nomad.app
 
 nomad.cli
 ------------

docs/setup.md

@@ -3,7 +3,7 @@
 ## Introduction
 The nomad infrastructure consists of a series of nomad and 3rd party services:
 - nomad worker (python): task worker that will do the processing
-- nomad api (python): the nomad REST API
+- nomad app (python): the nomad app and it's REST APIs
 - nomad gui: a small server serving the web-based react gui
 - proxy: an nginx server that reverse proxyies all services under one port
 - elastic search: nomad's search and analytics engine
@@ -34,11 +34,30 @@ git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-FAIR.git
 cd nomad-FAIR
 ```
 
+### C libs
+
+Even though the NOMAD infrastructure is written in python, there is a C library
+required by one of our pyhton dependencies.
+
+#### libmagic
+
+Libmagic allows to determine the MIME type of files. It should be installed on most
+unix/linux systems. It can be installed on MacOS with homebrew:
+
+```
+brew install libmagic
+```
+
+### Virtual environment
+
+#### pyenv
 The nomad code currently targets python 3.6. If you host machine has 3.7 or later installed,
 you can use [pyenv](https://github.com/pyenv/pyenv) to use python 3.6 in parallel.
-While in principle everything should be compatable with 3.7 and later there have been
-issues with some dependencies and requirements not being compatible with 3.7
+To use 3.7 there is a slight issue about the `enum34` which fails the compilation of the
+`mdtraj` and `mdanalysis` packages. A possible work arround is to uninstall and tham re-install
+`enum34` once the other packages are installed.
 
+#### virtualenv
 We strongly recommend to use *virtualenv* to create a virtual environment. It will allow you
 to keep nomad and its dependencies separate from your system's python installation.
 Make sure to base the virtual environment on Python 3.
@@ -49,6 +68,30 @@ virtualenv -p `which python3` .pyenv
 source .pyenv/bin/activate
 ```
 
+#### Conda
+If you are a conda user, there is an equivalent, but you have to install pip and the
+right python version while creating the environment.
+```
+conda create --name nomad_env pip python=3.6
+conda activate nomad_env
+```
+
+To install libmagick for conda, you can use (other channels might also work):
+```
+conda -c conda-forge install --name nomad_env libmagic
+```
+
+The next steps can be done using the `setup.sh` script. If you prefere to understand all
+the steps and run them manually, read on:
+
+### Get all the submodules
+We use git submodules to retrieve all the other NOMAD repositories, mainly parsers.
+
+```
+git submodules update --init
+```
+
+### Install python dependencies
 We use *pip* to manage required python packages.
 ```
 pip install -r requirements.txt
@@ -56,19 +99,22 @@ pip install -r requirements.txt
 
 ### Install NOMAD-coe dependencies.
 Nomad is based on python modules from the NOMAD-coe project.
-This includes parsers, normalizers, python-common and the meta-info.
-Those dependencies are managed and configured via python in
-`nomad/dependencies.py`. This gives us more flexibility in interacting with
-different parser, normalizer versions from within the running nomad infrastructure.
+This includes parsers, python-common and the meta-info. These modules are maintained as
+their own GITLab/git repositories. To clone and initialize them run:
 
-To run the dependencies script and install all dependencies into your environment:
 ```
-./dependencies.sh
+git submodules update --init
 ```
-This will checkout the proper version of the respective NOMAD-coe modules, install
-further requirements, and install the modules themselves. The `-e` option will install
-the NOMAD-coe dependencies with symbolic links allowing you to change the downloaded
-dependency code without having to reinstall after.
+
+All requirements for these submodules need to be installed and they need to be installed
+themselves as python modules. Run the `dependencies.sh` script that will install
+everything into your virtual environment:
+```
+./dependencies.sh -e
+```
+
+The `-e` option will install the NOMAD-coe dependencies with symbolic links allowing you
+to change the downloaded dependency code without having to reinstall after.
 
 ### Install nomad
 Finally, you can add nomad to the environment itself.
@@ -94,7 +140,7 @@ There are currently two different images and respectively two different docker f
 `Dockerfile`, and `gui/Dockerfile`.
 
 Nomad comprises currently two services,
-the *worker* (does the actual processing), and the *api*. Those services can be
+the *worker* (does the actual processing), and the *app*. Those services can be
 run from one image that have the nomad python code and all dependencies installed. This
 is covered by the `Dockerfile`.
 
@@ -117,21 +163,14 @@ The images are build via *docker-compose* and don't have to be created manually.
 We have multiple *docker-compose* files that must be used together.
 - `docker-compose.yml` contains the base definitions for all services
 - `docker-compose.override.yml` configures services for development (notably builds images for nomad services)
+- `docker-compose.dev-elk.yml` will also provide the ELK service
 - `docker-compose.prod.yml` configures services for production (notable uses a pre-build image for nomad services that was build during CI/CD)
 
 It is sufficient to use the implicit `docker-compose.yml` only (like in the command below).
 The `override` will be used automatically.
 
-There is also an `.env` file. For development you can use `.env_development`:
-```
-cd ./ops/docker-compose/nomad
-ln -s .env_development .env
-```
-
-The production `.env` file is stored on our serves and not part of the source code.
-
 Now we can build the *docker-compose* that contains all external services (rabbitmq,
-mongo, elastic, elk) and nomad services (worker, api, gui).
+mongo, elastic, elk) and nomad services (worker, app, gui).
 ```
 docker-compose build
 ```
@@ -159,7 +198,7 @@ docker-compose down
 ### Run containers selectively
 The following services/containers are managed via our docker-compose:
 - rabbitmq, mongo, elastic, (elk, only for production)
-- worker, api
+- worker, app
 - gui
 - proxy
 
@@ -168,33 +207,27 @@ a single port and different paths.
 
 You can also run services selectively, e.g.
 ```
-docker-compose up -d rabbitmq, mongo, elastic, elk
+docker-compose up -d rabbitmq, mongo, elastic
 docker-compose up worker
-docker-compose up api gui proxy
+docker-compose up app gui proxy
 ```
 
-### Configure the containers
-The *docker-compose* takes some configuration from the environment. Environment
-variables are set in `.env`, which is a link to either `.env_local` (intended to run
-nomad for development on a local computer) and `.env_processing` (indented to run
-nomad on nomad in *'production'*, currently on the enc pre-processing machine).
-
-You can configure host ports, volume locations for host bindings, and these sort of things.
-
 ## Accessing 3'rd party services
 
 Usually these services only used by the nomad containers, but sometimes you also
 need to check something or do some manual steps.
 
-The file `ops/docker-compose/nomad/.env` contains variables that control the ports
-used to bind internal docker ports to your host machine. These are the ports you
-have to use to connect to the respective services.
+The *docker-compose* can be overriden with additional seetings. See documentation section on
+operating NOMAD for more details. The override `docker-compose.override.yml` will
+expose all database ports to the hostmachine and should be used in development. To use
+it run docker-compose with `-f docker-compose.yml -f docker-compose.override.yml`.
 
 ### ELK (elastic stack)
 
 If you run the ELK stack (and enable logstash in nomad/config.py),
 you can reach the Kibana with [localhost:5601](http://localhost:5601).
-The index prefix for logs is `logstash-`.
+The index prefix for logs is `logstash-`. The ELK is only available with the
+`docker-compose.dev-elk.yml` override.
 
 ### mongodb and elastic search
 
@@ -204,48 +237,33 @@ to use the right ports (see above).
 
 ## Run nomad services manually
 
-You can run the worker, api, and gui as part of the docker infrastructure, like
+You can run the worker, app, and gui as part of the docker infrastructure, like
 seen above. But, of course there are always reasons to run them manually during
 development, like running them in a debugger, profiler, etc.
 
-### Run the nomad worker manually
+### API and worker
 
 To simply run a worker with the installed nomad cli, do (from the root)
 ```
-nomad run worker
+nomad admin run worker
 ```
 
-To run it manually with celery, do (from the root)
+To run it directly with celery, do (from the root)
 ```
 celery -A nomad.processing worker -l info
 ```
-You can use different debug level (e.g. switch `info` to `debug`)
 
-Use watchdog during development to reload the worker on code changes.
-Watchdog is part of the requirements-dev.txt. For MacOS (there is currently a bug in watchdog)
-uninstall and install this [fixed](https://github.com/gorakhargosh/watchdog/issues/330) version
+Run the app via docker, or (from the root):
 ```
-pip uninstall watchdog
-pip install git+https://github.com/gorakhargosh/watchdog.git
+nomad admin run app
 ```
 
-Now use this to auto relead worker:
+You can also run worker and app together:
 ```
-watchmedo auto-restart -d ./nomad -p '*.py' -- celery worker -l info -A nomad.processing
+nomad admin run appworker
 ```
 
-### Run the api
-Either with docker, or:
-```
-nomad run api
-```
-
-Or manually:
-```
-python nomad/api.py
-```
-
-### Run the gui
+### GUI
 When you run the gui on its own (e.g. with react dev server below), you have to have
 the API running manually also. This *inside docker* API is configured for ngingx paths
 and proxies, which are run by the gui container. But you can run the *production* gui
@@ -270,12 +288,131 @@ pytest -svx tests
 
 We use pylint, pycodestyle, and mypy to ensure code quality. To run those:
 ```
-nomad qa --skip-test
+nomad dev qa --skip-test
 ```
 
 To run all tests and code qa:
 ```
-nomad qa
+nomad dev qa
 ```
 
 This mimiques the tests and checks that the GitLab CI/CD will perform.
+
+
+## Setup your (I)DE
+
+The documentation section on development guidelines details how the code is organized,
+tested, formatted, and documented. To help you meet these guidelines, we recomment to
+use a proper IDE for development and ditch any VIM/Emacs (mal-)practices.
+
+### Visual Studio Code
+
+Here are some VSCode settings that will enable features for linting, some auto formating,
+line size ruler, etc.
+```json
+{
+    "python.venvPath": "${workspaceFolder}/.pyenv",
+    "python.pythonPath": "${workspaceFolder}/.pyenv/bin/python",
+    "git.ignoreLimitWarning": true,
+    "editor.rulers": [90],
+    "editor.renderWhitespace": "all",
+    "editor.tabSize": 4,
+    "[javascript]": {
+        "editor.tabSize": 2
+    },
+    "files.trimTrailingWhitespace": true,
+    "git.enableSmartCommit": true,
+    "eslint.autoFixOnSave": true,
+    "python.linting.pylintArgs": [
+        "--load-plugins=pylint_mongoengine",
+    ],
+    "python.linting.pep8Path": "pycodestyle",
+    "python.linting.pep8Enabled": true,
+    "python.linting.pep8Args": ["--ignore=E501,E701"],
+    "python.linting.mypyEnabled": true,
+    "python.linting.mypyArgs": [
+        "--ignore-missing-imports",
+        "--follow-imports=silent",
+        "--no-strict-optional"
+    ],
+    "workbench.colorCustomizations": {
+        "editorError.foreground": "#FF2222",
+        "editorOverviewRuler.errorForeground": "#FF2222",
+        "editorWarning.foreground": "#FF5500",
+        "editorOverviewRuler.warningForeground": "#FF5500",
+        "activityBar.background": "#4D2111",
+        "titleBar.activeBackground": "#6B2E18",
+        "titleBar.activeForeground": "#FDF9F7"
+    },
+    "files.watcherExclude": {
+        "**/.git/objects/**": true,
+        "**/.git/subtree-cache/**": true,
+        "**/node_modules/*/**": true,
+        "**/.pyenv/*/**": true,
+        "**/__pycache__/*/**": true,
+        "**/.mypy_cache/*/**": true,
+        "**/.volumes/*/**": true,
+        "**/docs/.build/*/**": true
+    }
+}
+```
+
+Here are some example launch configs for VSCode:
+
+```json
+{
+  "version": "0.2.0",
+  "configurations": [
+    {
+      "type": "chrome",
+      "request": "launch",
+      "name": "Launch Chrome against localhost",
+      "url": "http://localhost:3000",
+      "webRoot": "${workspaceFolder}/gui"
+    },
+    {
+      "name": "Python: API Flask (0.11.x or later)",
+      "type": "python",
+      "request": "launch",
+      "module": "flask",
+      "env": {
+        "FLASK_APP": "nomad/app/__init__.py"
+      },
+      "args": [
+        "run",
+        "--port",
+        "8000",
+        "--no-debugger",
+        "--no-reload"
+      ]
+    },
+    {
+      "name": "Python: some test",
+      "type": "python",
+      "request": "launch",
+      "cwd": "${workspaceFolder}",
+      "program": "${workspaceFolder}/.pyenv/bin/pytest",
+      "args": [
+        "-sv",
+        "tests/test_cli.py::TestClient::test_mirror"
+      ]
+    },
+    {
+      "name": "Python: Current File",
+      "type": "python",
+      "request": "launch",
+      "program": "${file}"
+    },
+    {
+      "name": "Python: Attach",
+      "type": "python",
+      "request": "attach",
+      "localRoot": "${workspaceFolder}",
+      "remoteRoot": "${workspaceFolder}",
+      "port": 3000,
+      "secret": "my_secret",
+      "host": "localhost"
+    }
+  ]
+}
+```
\ No newline at end of file