Commit ff80718e authored by Alvin Noe Ladines's avatar Alvin Noe Ladines
Browse files

Merge branch 'v0.6.0' into dev-alvin0

parents 9301b014 ce6c2ea6
Pipeline #61741 passed with stages
in 20 minutes and 50 seconds
.DS_Store
.pyenv/
.env/
.ipynb_checkpoints/
__pycache__
.mypy_cache
*.pyc
......
Subproject commit 43d77d198acfc3137d1c4d08a4601248fbf8548d
Subproject commit c36949f0c8421fff340d314d16ee83f7da5974ac
Subproject commit 67e781f7288b7e87da0a25ec02bf33d12de436f5
Subproject commit c14a0bc535aa8a7bbae64b565d6f7f61d3b5515d
Subproject commit a1cba85370ad5923969ac9ceb3643a56b3c2e7d9
Subproject commit aee4be7407124f87b0ba99eb7b4af3646b8602e9
......@@ -12,6 +12,7 @@ and infrastructure with a simplyfied architecture and consolidated code base.
dev_guidelines
api_tutorial
api
ops
metainfo
parser_tutorial
reference
ops
......@@ -124,7 +124,7 @@ The component library [Material-UI](https://material-ui.com/)
### docker
To run a **nomad@FAIRDI** instance, many services have to be orchestrated:
the nomad api, nomad worker, mongodb, Elasticsearch, PostgreSQL, RabbitMQ,
the nomad app, nomad worker, mongodb, Elasticsearch, PostgreSQL, RabbitMQ,
Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together.
Further services might be needed (e.g. JypiterHUB), when nomad grows.
The container platform [Docker](https://docs.docker.com/) allows us to provide all services
......
Metainfo
========
.. automodule:: nomad.metainfo
......@@ -3,7 +3,7 @@
## Introduction
The nomad infrastructure consists of a series of nomad and 3rd party services:
- nomad worker (python): task worker that will do the processing
- nomad api (python): the nomad REST API
- nomad app (python): the nomad app and it's REST APIs
- nomad gui: a small server serving the web-based react gui
- proxy: an nginx server that reverse proxyies all services under one port
- elastic search: nomad's search and analytics engine
......@@ -140,7 +140,7 @@ There are currently two different images and respectively two different docker f
`Dockerfile`, and `gui/Dockerfile`.
Nomad comprises currently two services,
the *worker* (does the actual processing), and the *api*. Those services can be
the *worker* (does the actual processing), and the *app*. Those services can be
run from one image that have the nomad python code and all dependencies installed. This
is covered by the `Dockerfile`.
......@@ -170,7 +170,7 @@ It is sufficient to use the implicit `docker-compose.yml` only (like in the comm
The `override` will be used automatically.
Now we can build the *docker-compose* that contains all external services (rabbitmq,
mongo, elastic, elk) and nomad services (worker, api, gui).
mongo, elastic, elk) and nomad services (worker, app, gui).
```
docker-compose build
```
......@@ -198,7 +198,7 @@ docker-compose down
### Run containers selectively
The following services/containers are managed via our docker-compose:
- rabbitmq, mongo, elastic, (elk, only for production)
- worker, api
- worker, app
- gui
- proxy
......@@ -209,7 +209,7 @@ You can also run services selectively, e.g.
```
docker-compose up -d rabbitmq, mongo, elastic
docker-compose up worker
docker-compose up api gui proxy
docker-compose up app gui proxy
```
## Accessing 3'rd party services
......@@ -237,7 +237,7 @@ to use the right ports (see above).
## Run nomad services manually
You can run the worker, api, and gui as part of the docker infrastructure, like
You can run the worker, app, and gui as part of the docker infrastructure, like
seen above. But, of course there are always reasons to run them manually during
development, like running them in a debugger, profiler, etc.
......@@ -253,14 +253,14 @@ To run it directly with celery, do (from the root)
celery -A nomad.processing worker -l info
```
Run the api via docker, or (from the root):
Run the app via docker, or (from the root):
```
nomad admin run api
nomad admin run app
```
You can also run worker and api together:
You can also run worker and app together:
```
nomad admin run apiworker
nomad admin run appworker
```
### GUI
......@@ -376,7 +376,7 @@ Here are some example launch configs for VSCode:
"request": "launch",
"module": "flask",
"env": {
"FLASK_APP": "nomad/api/__init__.py"
"FLASK_APP": "nomad/app/__init__.py"
},
"args": [
"run",
......
%% Cell type:markdown id: tags:
# NOMAD Metainfo 2.0 demonstration
You can find more complete documentation [here](https://labdev-nomad.esc.rzg.mpg.de/fairdi/nomad/testing/docs/metainfo.html)
%% Cell type:code id: tags:
``` python
from nomad.metainfo import MSection, SubSection, Quantity, Datetime, units
import numpy as np
import datetime
```
%% Cell type:markdown id: tags:
## Sections and quantities
To define sections and their quantities, we use Python classes and attributes. Quantities have *type*, *shape*, and *unit*.
%% Cell type:code id: tags:
``` python
class System(MSection):
""" The simulated system """
number_of_atoms = Quantity(type=int, derived=lambda system: len(system.atom_labels))
atom_labels = Quantity(type=str, shape=['number_of_atoms'])
atom_positions = Quantity(type=np.dtype(np.float64), shape=['number_of_atoms', 3], unit=units.m)
```
%% Cell type:markdown id: tags:
Such *section classes* can then be instantiated like regular Python classes. Respectively, *section instances* are just regular Python object and section quantities can be get and set like regular Python object attributes.
%% Cell type:code id: tags:
``` python
system = System()
system.atom_labels = ['H', 'H', '0']
system.atom_positions = np.array([[6, 0, 0], [0, 0, 0], [3, 2, 0]]) * units.angstrom
```
%% Cell type:markdown id: tags:
Of course the metainfo is not just about dealing with physics data in Python. Its also about storing and managing data in various fileformats and databases. Therefore, the created data can be serialized, e.g. to JSON. All *section
instances* have a set of additional `m_`-methods that provide addtional functions. Note the unit conversion.
%% Cell type:code id: tags:
``` python
system.m_to_json()
```
%%%% Output: execute_result
'{"atom_labels": ["H", "H", "0"], "atom_positions": [[6e-10, 0.0, 0.0], [0.0, 0.0, 0.0], [3e-10, 2e-10, 0.0]]}'
%% Cell type:markdown id: tags:
## Sub-sections to form hiearchies of data
*Section instances* can be nested to form data hierarchies. To achive this, we first have to create *section
definitions* that have sub-sections.
%% Cell type:code id: tags:
``` python
class Run(MSection):
timestamp = Quantity(type=Datetime, description='The time that this run was conducted.')
systems = SubSection(sub_section=System, repeats=True)
```
%% Cell type:markdown id: tags:
Now we can add *section instances* for `System` to *instances* of `Run`.
%% Cell type:code id: tags:
``` python
run = Run()
run.timestamp = datetime.datetime.now()
system = run.m_create(System)
system.atom_labels = ['H', 'H', '0']
system.atom_positions = np.array([[6, 0, 0], [0, 0, 0], [3, 2, 0]]) * units.angstrom
system = run.m_create(System)
system.atom_labels = ['H', 'H', '0']
system.atom_positions = np.array([[5, 0, 0], [0, 0, 0], [2.5, 2, 0]]) * units.angstrom
run.m_to_json()
```
%%%% Output: execute_result
'{"timestamp": "2019-10-09T14:48:43.663363", "systems": [{"atom_labels": ["H", "H", "0"], "atom_positions": [[6e-10, 0.0, 0.0], [0.0, 0.0, 0.0], [3e-10, 2e-10, 0.0]]}, {"atom_labels": ["H", "H", "0"], "atom_positions": [[5e-10, 0.0, 0.0], [0.0, 0.0, 0.0], [2.5e-10, 2e-10, 0.0]]}]}'
%% Cell type:markdown id: tags:
The whole data hiearchy can be navigated with regular Python object/attribute style programming and values can be
used for calculations as usual.
%% Cell type:code id: tags:
``` python
(run.systems[1].atom_positions - run.systems[0].atom_positions).to(units.angstrom)
```
%%%% Output: execute_result
$[[-1. 0. 0. ] [ 0. 0. 0. ] [-0.5 0. 0. ]] angstrom$
<Quantity([[-1. 0. 0. ]
[ 0. 0. 0. ]
[-0.5 0. 0. ]], 'angstrom')>
%% Cell type:markdown id: tags:
## Reflection, inspection, and code-completion
Since all definitions are available as *section classes*, Python already knows about all possible quantities. We can
use this in Python notebooks, via *tab* or the `?`-operator. Furthermore, you can access the *section definition* of all *section instances* with `m_def`. Since a *section defintion* itself is just a piece of metainfo data, you can use it to programatically explore the definition itselve.
%% Cell type:code id: tags:
``` python
run.systems[0].m_def.quantities
```
%%%% Output: execute_result
[number_of_atoms:Quantity, atom_labels:Quantity, atom_positions:Quantity]
%% Cell type:code id: tags:
``` python
run.m_def.all_quantities['timestamp'].description
```
%%%% Output: execute_result
'The time that this run was conducted.'
%% Cell type:code id: tags:
``` python
System.atom_labels.shape
```
%%%% Output: execute_result
['number_of_atoms']
%% Cell type:code id: tags:
``` python
t = np.dtype(np.i64)
```
%% Cell type:code id: tags:
``` python
t.type
```
%%%% Output: execute_result
numpy.int64
%% Cell type:code id: tags:
``` python
```
window.nomadEnv = {
'apiBase': 'http://localhost:8000/fairdi/nomad/latest/api',
'appBase': 'http://localhost:8000/fairdi/nomad/latest',
'kibanaBase': '/fairdi/kibana',
'debug': false
}
......@@ -2,7 +2,7 @@ import React from 'react'
import PropTypes from 'prop-types'
import { withStyles } from '@material-ui/core/styles'
import Markdown from './Markdown'
import { kibanaBase, apiBase, debug } from '../config'
import { kibanaBase, appBase, optimadeBase, apiBase, debug } from '../config'
import { compose } from 'recompose'
import { withApi } from './api'
import { withDomain } from './domains'
......@@ -39,18 +39,25 @@ class About extends React.Component {
system](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/issues).
### Developer Documentation
You find in depth developer documentation [here](${apiBase}/docs/index.html).
It contains a general introduction to NOMAD, the underlying architecture,
The [in-depth developer documentation](${appBase}/docs/index.html)
contains a general introduction to NOMAD, the underlying architecture,
is (meta)data, and processing. You will also find some information on how to use
the NOMAD ReST API. It contains information about how to develop NOMAD, how to
operate it, how to contribute parser, and much more.
operate it, how to contribute parsers, and much more.
### ReST API
NOMAD services can also be accessed programmatically via NOMAD's
ReST API. The API is described via [swagger](https://swagger.io/), therefore
you can use your favorite swagger client library (e.g.
[bravado](https://github.com/Yelp/bravado) for Python).
Here is [our API's swagger UI](${apiBase}/) as reference documentation.
### ReST APIs
NOMAD services can also be accessed programmatically via ReST APIs.
There is the proprietary NOMAD API and an implementation of the
[OPTiMaDe API (0.10.0)](https://github.com/Materials-Consortia/OPTiMaDe/tree/master)
standardized by the [OPTiMaDe consortium](https://www.optimade.org/)
Both APIs are described via [swagger](https://swagger.io/) (also known as OpenAPI spec.),
therefore you can use your favorite swagger client library
(e.g. [bravado](https://github.com/Yelp/bravado) for Python).
There are also web-based GUIs that allow to explore the APIs and their documentation:
- [NOMAD API](${apiBase}/)
- [OPTiMaDe API](${optimadeBase}/)
### Source code
The source-code for this new version of NOMAD (dubbed *nomad@FAIRDI*) is maintained
......
......@@ -21,7 +21,7 @@ import { ErrorSnacks, withErrors } from './errors'
import EntryPage from './entry/EntryPage'
import About from './About'
import LoginLogout from './LoginLogout'
import { genTheme, repoTheme, archiveTheme, appBase } from '../config'
import { genTheme, repoTheme, archiveTheme, guiBase } from '../config'
import { DomainProvider, withDomain } from './domains'
import {help as metainfoHelp, default as MetaInfoBrowser} from './metaInfoBrowser/MetaInfoBrowser'
import packageJson from '../../package.json'
......@@ -41,8 +41,6 @@ export class VersionMismatch extends Error {
const drawerWidth = 200
class NavigationUnstyled extends React.Component {
static propTypes = {
classes: PropTypes.object.isRequired,
......@@ -185,7 +183,7 @@ class NavigationUnstyled extends React.Component {
}
componentDidMount() {
fetch(`${appBase}/meta.json`)
fetch(`${guiBase}/meta.json`)
.then((response) => response.json())
.then((meta) => {
if (meta.version !== packageJson.version) {
......
......@@ -5,7 +5,7 @@ import { FormControl, FormControlLabel, Checkbox, FormGroup, FormLabel, Tooltip
import { compose } from 'recompose'
import { withErrors } from '../errors'
import { withApi, DisableOnLoading } from '../api'
import { appBase } from '../../config'
import { guiBase } from '../../config'
import Search from './Search'
import qs from 'qs'
......@@ -44,7 +44,7 @@ The results table gives you a quick overview of all entries that fit your search
You can click entries to see more details, download data, see the archive, etc.
The *raw files* tab, will show you all files that belong to the entry and offers a download
on individual, or all files. The *archive* tab, shows you the parsed data as a tree
data structure. This view is connected to NOMAD's [meta-info](${appBase}/metainfo), which acts a schema for
data structure. This view is connected to NOMAD's [meta-info](${guiBase}/metainfo), which acts a schema for
all parsed data. The *log* tab, will show you a log of the entry's processing.
`
......
......@@ -6,8 +6,10 @@ import secondary from '@material-ui/core/colors/blueGrey'
import { createMuiTheme } from '@material-ui/core'
window.nomadEnv = window.nomadEnv || {}
export const apiBase = window.nomadEnv.apiBase
export const appBase = process.env.PUBLIC_URL
export const appBase = window.nomadEnv.appBase.replace(/\/$/, '')
export const apiBase = `${appBase}/api`
export const optimadeBase = `${appBase}/optimade`
export const guiBase = process.env.PUBLIC_URL
export const kibanaBase = window.nomadEnv.kibanaBase
export const matomoUrl = window.nomadEnv.matomoUrl
export const matomoSiteId = window.nomadEnv.matomoSiteId
......
......@@ -40,7 +40,7 @@ def url(endpoint: str = None, **kwargs):
api = Api(
blueprint,
version='1.0', title='NOMAD optimade PI',
description='The NOMAD optimade API',
version='1.0', title='NOMAD\'s OPTiMaDe API implementation',
description='NOMAD\'s OPTiMaDe API implementation, version 0.10.0.',
validate=True)
""" Provides the flask restplust api instance for the optimade api"""
......@@ -14,16 +14,21 @@
from flask_restplus import Resource, abort
from flask import request
from elasticsearch_dsl import Q
from nomad import search
from nomad.metainfo.optimade import OptimadeStructureEntry
from nomad.metainfo.optimade import OptimadeEntry
from .api import api, url
from .models import json_api_single_response_model, entry_listing_endpoint_parser, Meta, \
Links, CalculationDataObject, single_entry_endpoint_parser, base_endpoint_parser
Links, CalculationDataObject, single_entry_endpoint_parser, base_endpoint_parser,\
json_api_info_response_model
from .filterparser import parse_filter, FilterException
ns = api.namespace('', description='The (only) API namespace with all OPTiMaDe endpoints.')
# TODO replace with decorator that filters response_fields
def base_request_args():
if request.args.get('response_format', 'json') != 'json':
......@@ -35,7 +40,13 @@ def base_request_args():
return None
@api.route('/calculations')
def base_search_request():
""" Creates a search request for all public and optimade enabled data. """
return search.SearchRequest().owner('all', None).query(
Q('exists', field='optimade.nelements')) # TODO use the elastic annotations when done
@ns.route('/calculations')
class CalculationList(Resource):
@api.doc('list_calculations')
@api.response(400, 'Invalid requests, e.g. bad parameter.')
......@@ -54,7 +65,8 @@ class CalculationList(Resource):
except Exception:
abort(400, message='bad parameter types') # TODO Specific json API error handling
search_request = search.SearchRequest().owner('all', None)
search_request = base_search_request()
if filter is not None:
try:
search_request.query(parse_filter(filter))
......@@ -67,27 +79,26 @@ class CalculationList(Resource):
# order_by='optimade.%s' % sort) # TODO map the Optimade property
available = result['pagination']['total']
print(result['results'][0]['optimade'].keys())
raise
results = search.to_calc_with_metadata(result['results'])
assert len(results) == len(result['results']), 'Mongodb and elasticsearch are not consistent'
return dict(
meta=Meta(
query=request.url,
returned=len(result['results']),
returned=len(results),
available=available,
last_id=result['results'][-1]['calc_id'] if available > 0 else None),
last_id=results[-1].calc_id if available > 0 else None),
links=Links(
'calculations',
available=available,
page_number=page_number,
page_limit=page_limit,
sort=sort, filter=filter),
data=[CalculationDataObject(d, request_fields=request_fields) for d in result['results']]
data=[CalculationDataObject(d, request_fields=request_fields) for d in results]
), 200
@api.route('/calculations/<string:id>')
@ns.route('/calculations/<string:id>')
class Calculation(Resource):
@api.doc('retrieve_calculation')
@api.response(400, 'Invalid requests, e.g. bad parameter.')
......@@ -97,49 +108,43 @@ class Calculation(Resource):
def get(self, id: str):
""" Retrieve a single calculation for the given id. """
request_fields = base_request_args()
search_request = search.SearchRequest().owner('all', None).search_parameters(calc_id=id)
search_request = base_search_request().search_parameters(calc_id=id)
result = search_request.execute_paginated(
page=1,
per_page=1)
available = result['pagination']['total']
results = search.to_calc_with_metadata(result['results'])
assert len(results) == len(result['results']), 'Mongodb and elasticsearch are not consistent'
if available == 0:
abort(404, 'The calculation with id %s does not exist' % id)
print('================', result['results'][0])
raise
return dict(
meta=Meta(query=request.url, returned=1),
data=CalculationDataObject(result['results'][0], request_fields=request_fields)
data=CalculationDataObject(results[0], request_fields=request_fields)
), 200
@api.route('/info/calculation')
@ns.route('/info/calculation')
class CalculationInfo(Resource):
@api.doc('calculations_info')
@api.response(400, 'Invalid requests, e.g. bad parameter.')
@api.expect(base_endpoint_parser, validate=True)
@api.marshal_with(json_api_single_response_model, skip_none=True, code=200)
@api.marshal_with(json_api_info_response_model, skip_none=True, code=200)
def get(self):
""" Returns information relating to the API implementation- """
base_request_args()
result = {
'type': 'info',
'id': 'calculation',
'attributes': {
'description': 'A calculations entry.',
# TODO non optimade, nomad specific properties
'properties': {
attr.name: dict(description=attr.description)
for attr in OptimadeStructureEntry.m_def.attributes.values()
},
'formats': ['json'],
'output_fields_by_format': {
'json': OptimadeStructureEntry.m_def.attributes.keys()
}
}
'description': 'a calculation entry',
'properties': {
attr.name: dict(description=attr.description)
for attr in OptimadeEntry.m_def.all_properties.values()},
'formats': ['json'],
'output_fields_by_format': {
'json': OptimadeEntry.m_def.all_properties.keys()}
}
return dict(
......@@ -148,7 +153,7 @@ class CalculationInfo(Resource):
), 200
@api.route('/info')
@ns.route('/info')
class Info(Resource):
@api.doc('info')
@api.response(400, 'Invalid requests, e.g. bad parameter.')
......
......@@ -16,7 +16,7 @@ from typing import Dict
from optimade.filterparser import LarkParser
from optimade.filtertransformers.elasticsearch import Transformer, Quantity
from elasticsearch_dsl import Q
from nomad.metainfo.optimade import OptimadeStructureEntry
from nomad.metainfo.optimade import OptimadeEntry
class FilterException(Exception):
......@@ -29,13 +29,9 @@ quantities: Dict[str, Quantity] = {
q.name, es_field='optimade.%s' % q.name,
elastic_mapping_type=q.m_annotations['elastic']['type'])
for q in OptimadeStructureEntry.m_def.quantities.values()
for q in OptimadeEntry.m_def.all_quantities.values()
if 'elastic' in q.m_annotations}
#for q in OptimadeStructureEntry.m_def.quantities.values():
# print(q.name, '-------------',q.m_annotations.keys(),'optimade' in q.m_annotations)
#print('IIIIIIIIIIIIIIIIIIIIIIIIIIII',type(quantities['elements']))
#raise
quantities['elements'].length_quantity = quantities['nelements']
quantities['dimension_types'].length_quantity = quantities['dimension_types']
quantities['elements'].has_only_quantity = Quantity(name='only_atoms')
......
......@@ -16,21 +16,22 @@
All the API flask restplus models.
"""
from typing import Dict, Any, Set
from typing import Set
from flask_restplus import fields
import datetime
import math
from nomad import config
from nomad.app.utils import RFC3339DateTime
from nomad.datamodel import CalcWithMetadata
from .api import api, base_url, url
# TODO error/warning objects
json_api_meta_object_model = api.model('JsonApiMetaObject', {
'query': fields.Nested(model=api.model('JsonApiQuery', {