Merge branch 'v0.6.0' into dev-alvin0

.gitignore

 .DS_Store
 .pyenv/
 .env/
+.ipynb_checkpoints/
 __pycache__
 .mypy_cache
 *.pyc

nomad-meta-info @ c36949f0

-Subproject commit 43d77d198acfc3137d1c4d08a4601248fbf8548d
+Subproject commit c36949f0c8421fff340d314d16ee83f7da5974ac

fhi-aims @ c14a0bc5

-Subproject commit 67e781f7288b7e87da0a25ec02bf33d12de436f5
+Subproject commit c14a0bc535aa8a7bbae64b565d6f7f61d3b5515d

python_common @ aee4be74

-Subproject commit a1cba85370ad5923969ac9ceb3643a56b3c2e7d9
+Subproject commit aee4be7407124f87b0ba99eb7b4af3646b8602e9

docs/index.rst

@@ -12,6 +12,7 @@ and infrastructure with a simplyfied architecture and consolidated code base.
    dev_guidelines
    api_tutorial
    api
-   ops
+   metainfo
    parser_tutorial
    reference
+   ops

docs/introduction.md

@@ -124,7 +124,7 @@ The component library [Material-UI](https://material-ui.com/)
 ### docker
 
 To run a **nomad@FAIRDI** instance, many services have to be orchestrated:
-the nomad api, nomad worker, mongodb, Elasticsearch, PostgreSQL, RabbitMQ,
+the nomad app, nomad worker, mongodb, Elasticsearch, PostgreSQL, RabbitMQ,
 Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together.
 Further services might be needed (e.g. JypiterHUB), when nomad grows.
 The container platform [Docker](https://docs.docker.com/) allows us to provide all services

docs/metainfo.rst

+Metainfo
+========
+
+.. automodule:: nomad.metainfo

docs/setup.md

@@ -3,7 +3,7 @@
 ## Introduction
 The nomad infrastructure consists of a series of nomad and 3rd party services:
 - nomad worker (python): task worker that will do the processing
-- nomad api (python): the nomad REST API
+- nomad app (python): the nomad app and it's REST APIs
 - nomad gui: a small server serving the web-based react gui
 - proxy: an nginx server that reverse proxyies all services under one port
 - elastic search: nomad's search and analytics engine
@@ -140,7 +140,7 @@ There are currently two different images and respectively two different docker f
 `Dockerfile`, and `gui/Dockerfile`.
 
 Nomad comprises currently two services,
-the *worker* (does the actual processing), and the *api*. Those services can be
+the *worker* (does the actual processing), and the *app*. Those services can be
 run from one image that have the nomad python code and all dependencies installed. This
 is covered by the `Dockerfile`.
 
@@ -170,7 +170,7 @@ It is sufficient to use the implicit `docker-compose.yml` only (like in the comm
 The `override` will be used automatically.
 
 Now we can build the *docker-compose* that contains all external services (rabbitmq,
-mongo, elastic, elk) and nomad services (worker, api, gui).
+mongo, elastic, elk) and nomad services (worker, app, gui).
 ```
 docker-compose build
 ```
@@ -198,7 +198,7 @@ docker-compose down
 ### Run containers selectively
 The following services/containers are managed via our docker-compose:
 - rabbitmq, mongo, elastic, (elk, only for production)
-- worker, api
+- worker, app
 - gui
 - proxy
 
@@ -209,7 +209,7 @@ You can also run services selectively, e.g.
 ```
 docker-compose up -d rabbitmq, mongo, elastic
 docker-compose up worker
-docker-compose up api gui proxy
+docker-compose up app gui proxy
 ```
 
 ## Accessing 3'rd party services
@@ -237,7 +237,7 @@ to use the right ports (see above).
 
 ## Run nomad services manually
 
-You can run the worker, api, and gui as part of the docker infrastructure, like
+You can run the worker, app, and gui as part of the docker infrastructure, like
 seen above. But, of course there are always reasons to run them manually during
 development, like running them in a debugger, profiler, etc.
 
@@ -253,14 +253,14 @@ To run it directly with celery, do (from the root)
 celery -A nomad.processing worker -l info
 ```
 
-Run the api via docker, or (from the root):
+Run the app via docker, or (from the root):
 ```
-nomad admin run api
+nomad admin run app
 ```
 
-You can also run worker and api together:
+You can also run worker and app together:
 ```
-nomad admin run apiworker
+nomad admin run appworker
 ```
 
 ### GUI
@@ -376,7 +376,7 @@ Here are some example launch configs for VSCode:
       "request": "launch",
       "module": "flask",
       "env": {
-        "FLASK_APP": "nomad/api/__init__.py"
+        "FLASK_APP": "nomad/app/__init__.py"
       },
       "args": [
         "run",

examples/metainfo.ipynb

+%% Cell type:markdown id: tags:
+
+# NOMAD Metainfo 2.0 demonstration
+
+You can find more complete documentation [here](https://labdev-nomad.esc.rzg.mpg.de/fairdi/nomad/testing/docs/metainfo.html)
+
+%% Cell type:code id: tags:
+
+``` python
+from nomad.metainfo import MSection, SubSection, Quantity, Datetime, units
+import numpy as np
+import datetime
+```
+
+%% Cell type:markdown id: tags:
+
+## Sections and quantities
+
+To define sections and their quantities, we use Python classes and attributes. Quantities have *type*, *shape*, and *unit*.
+
+%% Cell type:code id: tags:
+
+``` python
+class System(MSection):
+    """ The simulated system """
+    number_of_atoms = Quantity(type=int, derived=lambda system: len(system.atom_labels))
+    atom_labels = Quantity(type=str, shape=['number_of_atoms'])
+    atom_positions = Quantity(type=np.dtype(np.float64), shape=['number_of_atoms', 3], unit=units.m)
+```
+
+%% Cell type:markdown id: tags:
+
+Such *section classes* can then be instantiated like regular Python classes. Respectively, *section instances* are just regular Python object and section quantities can be get and set like regular Python object attributes.
+
+%% Cell type:code id: tags:
+
+``` python
+system = System()
+system.atom_labels = ['H', 'H', '0']
+system.atom_positions = np.array([[6, 0, 0], [0, 0, 0], [3, 2, 0]]) * units.angstrom
+```
+
+%% Cell type:markdown id: tags:
+
+Of course the metainfo is not just about dealing with physics data in Python. Its also about storing and managing data in various fileformats and databases. Therefore, the created data can be serialized, e.g. to JSON. All *section
+instances* have a set of additional `m_`-methods that provide addtional functions. Note the unit conversion.
+
+%% Cell type:code id: tags:
+
+``` python
+system.m_to_json()
+```
+
+%%%% Output: execute_result
+
+    '{"atom_labels": ["H", "H", "0"], "atom_positions": [[6e-10, 0.0, 0.0], [0.0, 0.0, 0.0], [3e-10, 2e-10, 0.0]]}'
+
+%% Cell type:markdown id: tags:
+
+## Sub-sections to form hiearchies of data
+
+*Section instances* can be nested to form data hierarchies. To achive this, we first have to create *section
+definitions* that have sub-sections.
+
+%% Cell type:code id: tags:
+
+``` python
+class Run(MSection):
+    timestamp = Quantity(type=Datetime, description='The time that this run was conducted.')
+    systems = SubSection(sub_section=System, repeats=True)
+```
+
+%% Cell type:markdown id: tags:
+
+Now we can add *section instances* for `System` to *instances* of `Run`.
+
+%% Cell type:code id: tags:
+
+``` python
+run = Run()
+run.timestamp = datetime.datetime.now()
+
+system = run.m_create(System)
+system.atom_labels = ['H', 'H', '0']
+system.atom_positions = np.array([[6, 0, 0], [0, 0, 0], [3, 2, 0]]) * units.angstrom
+
+system = run.m_create(System)
+system.atom_labels = ['H', 'H', '0']
+system.atom_positions = np.array([[5, 0, 0], [0, 0, 0], [2.5, 2, 0]]) * units.angstrom
+run.m_to_json()
+```
+
+%%%% Output: execute_result
+
+    '{"timestamp": "2019-10-09T14:48:43.663363", "systems": [{"atom_labels": ["H", "H", "0"], "atom_positions": [[6e-10, 0.0, 0.0], [0.0, 0.0, 0.0], [3e-10, 2e-10, 0.0]]}, {"atom_labels": ["H", "H", "0"], "atom_positions": [[5e-10, 0.0, 0.0], [0.0, 0.0, 0.0], [2.5e-10, 2e-10, 0.0]]}]}'
+
+%% Cell type:markdown id: tags:
+
+The whole data hiearchy can be navigated with regular Python object/attribute style programming and values can be
+used for calculations as usual.
+
+%% Cell type:code id: tags:
+
+``` python
+(run.systems[1].atom_positions - run.systems[0].atom_positions).to(units.angstrom)
+```
+
+%%%% Output: execute_result
+
+    $[[-1.   0.   0. ] [ 0.   0.   0. ] [-0.5  0.   0. ]] angstrom$
+    <Quantity([[-1.   0.   0. ]
+     [ 0.   0.   0. ]
+     [-0.5  0.   0. ]], 'angstrom')>
+
+%% Cell type:markdown id: tags:
+
+## Reflection, inspection, and code-completion
+
+Since all definitions are available as *section classes*, Python already knows about all possible quantities. We can
+use this in Python notebooks, via *tab* or the `?`-operator. Furthermore, you can access the *section definition* of all *section instances* with `m_def`. Since a *section defintion* itself is just a piece of metainfo data, you can use it to programatically explore the definition itselve.
+
+%% Cell type:code id: tags:
+
+``` python
+run.systems[0].m_def.quantities
+```
+
+%%%% Output: execute_result
+
+    [number_of_atoms:Quantity, atom_labels:Quantity, atom_positions:Quantity]
+
+%% Cell type:code id: tags:
+
+``` python
+run.m_def.all_quantities['timestamp'].description
+```
+
+%%%% Output: execute_result
+
+    'The time that this run was conducted.'
+
+%% Cell type:code id: tags:
+
+``` python
+System.atom_labels.shape
+```
+
+%%%% Output: execute_result
+
+    ['number_of_atoms']
+
+%% Cell type:code id: tags:
+
+``` python
+t = np.dtype(np.i64)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+t.type
+```
+
+%%%% Output: execute_result
+
+    numpy.int64
+
+%% Cell type:code id: tags:
+
+``` python
+```

gui/public/env.js

 window.nomadEnv = {
-  'apiBase': 'http://localhost:8000/fairdi/nomad/latest/api',
+  'appBase': 'http://localhost:8000/fairdi/nomad/latest',
   'kibanaBase': '/fairdi/kibana',
   'debug': false
 }

gui/src/components/About.js

@@ -2,7 +2,7 @@ import React from 'react'
 import PropTypes from 'prop-types'
 import { withStyles } from '@material-ui/core/styles'
 import Markdown from './Markdown'
-import { kibanaBase, apiBase, debug } from '../config'
+import { kibanaBase, appBase, optimadeBase, apiBase, debug } from '../config'
 import { compose } from 'recompose'
 import { withApi } from './api'
 import { withDomain } from './domains'
@@ -39,18 +39,25 @@ class About extends React.Component {
           system](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/issues).
 
           ### Developer Documentation
-          You find in depth developer documentation [here](${apiBase}/docs/index.html).
-          It contains a general introduction to NOMAD, the underlying architecture,
+          The [in-depth developer documentation](${appBase}/docs/index.html)
+          contains a general introduction to NOMAD, the underlying architecture,
           is (meta)data, and processing. You will also find some information on how to use
           the NOMAD ReST API. It contains information about how to develop NOMAD, how to
-          operate it, how to contribute parser, and much more.
+          operate it, how to contribute parsers, and much more.
 
-          ### ReST API
-          NOMAD services can also be accessed programmatically via NOMAD's
-          ReST API. The API is described via [swagger](https://swagger.io/), therefore
-          you can use your favorite swagger client library (e.g.
-          [bravado](https://github.com/Yelp/bravado) for Python).
-          Here is [our API's swagger UI](${apiBase}/) as reference documentation.
+          ### ReST APIs
+          NOMAD services can also be accessed programmatically via ReST APIs.
+          There is the proprietary NOMAD API and an implementation of the
+          [OPTiMaDe API (0.10.0)](https://github.com/Materials-Consortia/OPTiMaDe/tree/master)
+          standardized by the [OPTiMaDe consortium](https://www.optimade.org/)
+
+          Both APIs are described via [swagger](https://swagger.io/) (also known as OpenAPI spec.),
+          therefore you can use your favorite swagger client library
+          (e.g. [bravado](https://github.com/Yelp/bravado) for Python).
+
+          There are also web-based GUIs that allow to explore the APIs and their documentation:
+          - [NOMAD API](${apiBase}/)
+          - [OPTiMaDe API](${optimadeBase}/)
 
           ### Source code
           The source-code for this new version of NOMAD (dubbed *nomad@FAIRDI*) is maintained

gui/src/components/App.js

@@ -21,7 +21,7 @@ import { ErrorSnacks, withErrors } from './errors'
 import EntryPage from './entry/EntryPage'
 import About from './About'
 import LoginLogout from './LoginLogout'
-import { genTheme, repoTheme, archiveTheme, appBase } from '../config'
+import { genTheme, repoTheme, archiveTheme, guiBase } from '../config'
 import { DomainProvider, withDomain } from './domains'
 import {help as metainfoHelp, default as MetaInfoBrowser} from './metaInfoBrowser/MetaInfoBrowser'
 import packageJson from '../../package.json'
@@ -41,8 +41,6 @@ export class VersionMismatch extends Error {
 
 const drawerWidth = 200
 
-
-
 class NavigationUnstyled extends React.Component {
   static propTypes = {
     classes: PropTypes.object.isRequired,
@@ -185,7 +183,7 @@ class NavigationUnstyled extends React.Component {
   }
 
   componentDidMount() {
-    fetch(`${appBase}/meta.json`)
+    fetch(`${guiBase}/meta.json`)
       .then((response) => response.json())
       .then((meta) => {
         if (meta.version !== packageJson.version) {

gui/src/components/search/SearchPage.js

@@ -5,7 +5,7 @@ import { FormControl, FormControlLabel, Checkbox, FormGroup, FormLabel, Tooltip
 import { compose } from 'recompose'
 import { withErrors } from '../errors'
 import { withApi, DisableOnLoading } from '../api'
-import { appBase } from '../../config'
+import { guiBase } from '../../config'
 import Search from './Search'
 import qs from 'qs'
 
@@ -44,7 +44,7 @@ The results table gives you a quick overview of all entries that fit your search
 You can click entries to see more details, download data, see the archive, etc.
 The *raw files* tab, will show you all files that belong to the entry and offers a download
 on individual, or all files. The *archive* tab, shows you the parsed data as a tree
-data structure. This view is connected to NOMAD's [meta-info](${appBase}/metainfo), which acts a schema for
+data structure. This view is connected to NOMAD's [meta-info](${guiBase}/metainfo), which acts a schema for
 all parsed data. The *log* tab, will show you a log of the entry's processing.
 `
 

gui/src/config.js

@@ -6,8 +6,10 @@ import secondary from '@material-ui/core/colors/blueGrey'
 import { createMuiTheme } from '@material-ui/core'
 
 window.nomadEnv = window.nomadEnv || {}
-export const apiBase = window.nomadEnv.apiBase
-export const appBase = process.env.PUBLIC_URL
+export const appBase = window.nomadEnv.appBase.replace(/\/$/, '')
+export const apiBase = `${appBase}/api`
+export const optimadeBase = `${appBase}/optimade`
+export const guiBase = process.env.PUBLIC_URL
 export const kibanaBase = window.nomadEnv.kibanaBase
 export const matomoUrl = window.nomadEnv.matomoUrl
 export const matomoSiteId = window.nomadEnv.matomoSiteId

nomad/app/optimade/api.py

@@ -40,7 +40,7 @@ def url(endpoint: str = None, **kwargs):
 
 api = Api(
     blueprint,
-    version='1.0', title='NOMAD optimade PI',
-    description='The NOMAD optimade API',
+    version='1.0', title='NOMAD\'s OPTiMaDe API implementation',
+    description='NOMAD\'s OPTiMaDe API implementation, version 0.10.0.',
     validate=True)
 """ Provides the flask restplust api instance for the optimade api"""

nomad/app/optimade/endpoints.py

@@ -14,16 +14,21 @@
 
 from flask_restplus import Resource, abort
 from flask import request
+from elasticsearch_dsl import Q
 
 from nomad import search
-from nomad.metainfo.optimade import OptimadeStructureEntry
+from nomad.metainfo.optimade import OptimadeEntry
 
 from .api import api, url
 from .models import json_api_single_response_model, entry_listing_endpoint_parser, Meta, \
-    Links, CalculationDataObject, single_entry_endpoint_parser, base_endpoint_parser
+    Links, CalculationDataObject, single_entry_endpoint_parser, base_endpoint_parser,\
+    json_api_info_response_model
 from .filterparser import parse_filter, FilterException
 
 
+ns = api.namespace('', description='The (only) API namespace with all OPTiMaDe endpoints.')
+
+
 # TODO replace with decorator that filters response_fields
 def base_request_args():
     if request.args.get('response_format', 'json') != 'json':
@@ -35,7 +40,13 @@ def base_request_args():
     return None
 
 
-@api.route('/calculations')
+def base_search_request():
+    """ Creates a search request for all public and optimade enabled data. """
+    return search.SearchRequest().owner('all', None).query(
+        Q('exists', field='optimade.nelements'))  # TODO use the elastic annotations when done
+
+
+@ns.route('/calculations')
 class CalculationList(Resource):
     @api.doc('list_calculations')
     @api.response(400, 'Invalid requests, e.g. bad parameter.')
@@ -54,7 +65,8 @@ class CalculationList(Resource):
         except Exception:
             abort(400, message='bad parameter types')  # TODO Specific json API error handling
 
-        search_request = search.SearchRequest().owner('all', None)
+        search_request = base_search_request()
+
         if filter is not None:
             try:
                 search_request.query(parse_filter(filter))
@@ -67,27 +79,26 @@ class CalculationList(Resource):
         # order_by='optimade.%s' % sort)  # TODO map the Optimade property
 
         available = result['pagination']['total']
-       
-        print(result['results'][0]['optimade'].keys())
-        raise 
+        results = search.to_calc_with_metadata(result['results'])
+        assert len(results) == len(result['results']), 'Mongodb and elasticsearch are not consistent'
 
         return dict(
             meta=Meta(
                 query=request.url,
-                returned=len(result['results']),
+                returned=len(results),
                 available=available,
-                last_id=result['results'][-1]['calc_id'] if available > 0 else None),
+                last_id=results[-1].calc_id if available > 0 else None),
             links=Links(
                 'calculations',
                 available=available,
                 page_number=page_number,
                 page_limit=page_limit,
                 sort=sort, filter=filter),
-            data=[CalculationDataObject(d, request_fields=request_fields) for d in result['results']]
+            data=[CalculationDataObject(d, request_fields=request_fields) for d in results]
         ), 200
 
 
-@api.route('/calculations/<string:id>')
+@ns.route('/calculations/<string:id>')
 class Calculation(Resource):
     @api.doc('retrieve_calculation')
     @api.response(400, 'Invalid requests, e.g. bad parameter.')
@@ -97,49 +108,43 @@ class Calculation(Resource):
     def get(self, id: str):
         """ Retrieve a single calculation for the given id. """
         request_fields = base_request_args()
-        search_request = search.SearchRequest().owner('all', None).search_parameters(calc_id=id)
+        search_request = base_search_request().search_parameters(calc_id=id)
 
         result = search_request.execute_paginated(
             page=1,
             per_page=1)
 
         available = result['pagination']['total']
+        results = search.to_calc_with_metadata(result['results'])
+        assert len(results) == len(result['results']), 'Mongodb and elasticsearch are not consistent'
+
         if available == 0:
             abort(404, 'The calculation with id %s does not exist' % id)
 
-        print('================', result['results'][0])
-        raise
         return dict(
             meta=Meta(query=request.url, returned=1),
-            data=CalculationDataObject(result['results'][0], request_fields=request_fields)
+            data=CalculationDataObject(results[0], request_fields=request_fields)
         ), 200
 
 
-@api.route('/info/calculation')
+@ns.route('/info/calculation')
 class CalculationInfo(Resource):
     @api.doc('calculations_info')
     @api.response(400, 'Invalid requests, e.g. bad parameter.')
     @api.expect(base_endpoint_parser, validate=True)
-    @api.marshal_with(json_api_single_response_model, skip_none=True, code=200)
+    @api.marshal_with(json_api_info_response_model, skip_none=True, code=200)
     def get(self):
         """ Returns information relating to the API implementation- """
         base_request_args()
 
         result = {
-            'type': 'info',
-            'id': 'calculation',
-            'attributes': {
-                'description': 'A calculations entry.',
-                # TODO non optimade, nomad specific properties
-                'properties': {
-                    attr.name: dict(description=attr.description)
-                    for attr in OptimadeStructureEntry.m_def.attributes.values()
-                },
-                'formats': ['json'],
-                'output_fields_by_format': {
-                    'json': OptimadeStructureEntry.m_def.attributes.keys()
-                }
-            }
+            'description': 'a calculation entry',
+            'properties': {
+                attr.name: dict(description=attr.description)
+                for attr in OptimadeEntry.m_def.all_properties.values()},
+            'formats': ['json'],
+            'output_fields_by_format': {
+                'json': OptimadeEntry.m_def.all_properties.keys()}
         }
 
         return dict(
@@ -148,7 +153,7 @@ class CalculationInfo(Resource):
         ), 200
 
 
-@api.route('/info')
+@ns.route('/info')
 class Info(Resource):
     @api.doc('info')
     @api.response(400, 'Invalid requests, e.g. bad parameter.')

nomad/app/optimade/filterparser.py

@@ -16,7 +16,7 @@ from typing import Dict
 from optimade.filterparser import LarkParser
 from optimade.filtertransformers.elasticsearch import Transformer, Quantity
 from elasticsearch_dsl import Q
-from nomad.metainfo.optimade import OptimadeStructureEntry
+from nomad.metainfo.optimade import OptimadeEntry
 
 
 class FilterException(Exception):
@@ -29,13 +29,9 @@ quantities: Dict[str, Quantity] = {
         q.name, es_field='optimade.%s' % q.name,
         elastic_mapping_type=q.m_annotations['elastic']['type'])
 
-    for q in OptimadeStructureEntry.m_def.quantities.values()
+    for q in OptimadeEntry.m_def.all_quantities.values()
     if 'elastic' in q.m_annotations}
 
-#for q in OptimadeStructureEntry.m_def.quantities.values():
-#    print(q.name, '-------------',q.m_annotations.keys(),'optimade' in q.m_annotations)
-#print('IIIIIIIIIIIIIIIIIIIIIIIIIIII',type(quantities['elements']))
-#raise
 quantities['elements'].length_quantity = quantities['nelements']
 quantities['dimension_types'].length_quantity = quantities['dimension_types']
 quantities['elements'].has_only_quantity = Quantity(name='only_atoms')

nomad/app/optimade/models.py

@@ -16,21 +16,22 @@
 All the API flask restplus models.
 """
 
-from typing import Dict, Any, Set
+from typing import Set
 from flask_restplus import fields
 import datetime
 import math
 
 from nomad import config
 from nomad.app.utils import RFC3339DateTime
+from nomad.datamodel import CalcWithMetadata
 
 from .api import api, base_url, url
 
 
 # TODO error/warning objects
 
-json_api_meta_object_model = api.model('JsonApiMetaObject', {
-    'query': fields.Nested(model=api.model('JsonApiQuery', {