diff --git a/docs/api.md b/docs/api.md
index a2d7a542c2920de5f565c154a100f49071d3f705..6b55af62ae75dd0bcee56ffec58376ec7b7a9797 100644
--- a/docs/api.md
+++ b/docs/api.md
@@ -1,133 +1,422 @@
-# How to use the APIs
+# Using the APIs
-The NOMAD Repository and Archive offers all its functionality through an application
-programming interface (API). More specifically a [RESTful HTTP API](https://en.wikipedia.org/wiki/Representational_state_transfer) that allows you
-to use NOMAD as a set of resources (think data) that can be uploaded, accessed, downloaded,
-searched for, etc. via [HTTP requests](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol).
-
-There are different tools and libraries to use the NOMAD API that come with different
-trade-offs between expressiveness, learning curve, and convinience:
+## Getting started
-- use an HTTP program like *curl* or *wget* to directly use NOMAD from within a shell
-- use a generic Python HTTP library like [requests](https://requests.readthedocs.io/en/master/)
-- directly in the browser via our generated [OpenAPI dashboard](../api/v1)
-- use the NOMAD Python client library, which offers custom and more powerful
- implementations for certain tasks (currently only for accessing the NOMAD Archive)
+If you are comfortable with REST APIs and using Pythons `requests` library, this example
+demonstrates the basic concepts of NOMAD's main API. You can get more documentation and
+details on all functions from the [API dashboard]({{ nomad_url() }}/v1/).
-This set of tutorials provides a few examples for common NOMAD tasks using the various
-options.
+The following issues a search query for all entries that have both *Ti* and *O*
+among the elements of their respective materials. It restricts the results to
+one entry and only returns the `entry_id`.
-## Using *curl* (or *wget*)
+```py
+import requests
+import json
-Terminal programs like *curl* act as an HTTP client and allow you to send requests and
-display or store the respective responses. HTTP basically allows you to GET, POST, PUT,
-and DELETE "resources" on a remote server. These resources are identified via URLs (=uniform
-resource locator). URLs usually consists of a protocol (e.g. HTTP), a domain (our servers),
-a path (a place on our servers), and query parameters (additional options).
+base_url = 'http://nomad-lab.eu/prod/v1/api/v1'
+
+response = requests.post(
+ f'{base_url}/entries/query',
+ json={
+ 'query': {
+ 'results.material.elements': {
+ 'all': ['Ti', 'O']
+ }
+ },
+ 'pagination': {
+ 'page_size': 1
+ },
+ 'required': {
+ 'include': ['entry_id']
+ }
+ })
+response_json = response.json()
+print(json.dumps(response.json(), indent=2))
+```
-NOMAD provides three main set of resources: **repo** (i.e. the NOMAD Repository), **raw**
-(raw uploaded files), **archive** (i.e. the NOMAD Archive). Within all these resource sets
-you have endpoints that either allow you to directly locate a NOMAD entry (i.e. an
-uploaded code run) or to ask a query to locate many NOMAD entries at the same time. Here,
-the **repo** will return the repository metadata for said entries, **archive** the archive
-data, ...
+This will give you something like this:
+```json
+{
+ "owner": "public",
+ "query": {
+ "name": "results.material.elements",
+ "value": {
+ "all": [
+ "Ti",
+ "O"
+ ]
+ }
+ },
+ "pagination": {
+ "page_size": 1,
+ "order_by": "entry_id",
+ "order": "asc",
+ "total": 17957,
+ "next_page_after_value": "--SZVYOxA2jTu_L-mSxefSQFmeyF"
+ },
+ "required": {
+ "include": [
+ "entry_id"
+ ]
+ },
+ "data": [
+ {
+ "entry_id": "--SZVYOxA2jTu_L-mSxefSQFmeyF"
+ }
+ ]
+}
+```
-Let's say you want to see the repository metadata (i.e. the information that you see in
-our gui) for entries that fit search criteria, like compounds having atoms *Si* and *O* in
-it:
+The `entry_id` is a unique identifier for, well, entries. You can use it to access
+other entry data. For example, you want to access the entry's archive. More
+precisely, you want to gather the formula and energies from the main workflow result.
+The following requests the archive based on the `entry_id` and only requires
+the some archive sections.
+
+```py
+first_entry_id = response_json['data'][0]['entry_id']
+response = requests.post(
+ f'{base_url}/entries/{first_entry_id}/archive/query',
+ json={
+ 'required': {
+ 'workflow': {
+ 'calculation_result_ref': {
+ 'energy': '*',
+ 'system_ref': {
+ 'chemical_composition': '*'
+ }
+ }
+ }
+ }
+ })
+response_json = response.json()
+print(json.dumps(response_json, indent=2))
+```
-```sh
-curl -X GET "http://nomad-lab.eu/prod/rae/api/repo/?atoms=Si&atoms=O"
+The result will look like this:
+```json
+{
+ "required": {
+ "workflow": {
+ "calculation_result_ref": {
+ "energy": "*",
+ "system_ref": {
+ "chemical_composition": "*"
+ }
+ }
+ }
+ },
+ "entry_id": "--SZVYOxA2jTu_L-mSxefSQFmeyF",
+ "data": {
+ "entry_id": "--SZVYOxA2jTu_L-mSxefSQFmeyF",
+ "upload_id": "YXUIZpw5RJyV3LAsFI2MmQ",
+ "parser_name": "parsers/fhi-aims",
+ "archive": {
+ "run": [
+ {
+ "system": [
+ {
+ "chemical_composition": "OOSrTiOOOSrTiOOOSrTiOFF"
+ }
+ ],
+ "calculation": [
+ {
+ "energy": {
+ "fermi": -1.1363378335891879e-18,
+ "total": {
+ "value": -5.697771591896252e-14
+ },
+ "correlation": {
+ "value": -5.070133798617076e-17
+ },
+ "exchange": {
+ "value": -2.3099755059272454e-15
+ },
+ "xc": {
+ "value": -2.360676843913416e-15
+ },
+ "xc_potential": {
+ "value": 3.063766944960246e-15
+ },
+ "free": {
+ "value": -5.697771595558439e-14
+ },
+ "sum_eigenvalues": {
+ "value": -3.3841806795825544e-14
+ },
+ "total_t0": {
+ "value": -5.697771593727346e-14
+ },
+ "correction_entropy": {
+ "value": -1.8310927833270112e-23
+ },
+ "correction_hartree": {
+ "value": -4.363790430157292e-17
+ },
+ "correction_xc": {
+ "value": -2.3606768439090564e-15
+ }
+ },
+ "system_ref": "/run/0/system/0"
+ }
+ ]
+ }
+ ],
+ "workflow": [
+ {
+ "calculation_result_ref": "/run/0/calculation/0"
+ }
+ ]
+ }
+ }
+}
```
-Here we used curl to send an HTTP GET request to return the resource located by the given URL.
-In practice you can omit the `-X GET` (which is the default) and you might want to format
-the output:
+You can work with the results in the given JSON (or respective Python dict/list) data already.
+If you have [NOMAD's Python library](pythonlib) installed ,
+you can take the archive data and use the Python interface.
+The [Python interface](metainfo) will help with code-completion (e.g. in notebook environments),
+resolve archive references (e.g. from workflow to calculation to system), and allow unit conversion:
+```py
+from nomad.datamodel import EntryArchive
+from nomad.metainfo import units
+
+archive = EntryArchive.m_from_dict(response_json['data']['archive'])
+result = archive.workflow[0].calculation_result_ref
+print(result.system_ref.chemical_composition)
+print(result.energy.total.value.to(units('eV')))
+```
-```sh
-curl "http://nomad-lab.eu/prod/rae/api/repo/?atoms=Si&atoms=O" | python -m json.tool
+This will give you an output like this:
```
+OOSrTiOOOSrTiOOOSrTiOFF
+-355626.93095025205 electron_volt
+```
+## Overview
+
+NOMAD offers all its functionality through application
+programming interfaces (APIs). More specifically [RESTful HTTP APIs](https://en.wikipedia.org/wiki/Representational_state_transfer) that allows you
+to use NOMAD as a set of resources (think data) that can be uploaded, accessed, downloaded,
+searched for, etc. via [HTTP requests](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol).
+
+You can get an overview on all NOMAD APIs on the [API page](https://nomad-lab.eu/prod/v1/gui/analyze/apis).
+We will focus here on NOMAD's main API (v1). In fact, this API is also used by
+the web interface and should provide everything you need.
-You'll see the the metadata of the first 10 entries that match your criteria. There
-are various other query parameters. You find a full list in the generated [swagger dashboard
-of our API](https://nomad-lab.eu/prod/rae/api/).
+There are different tools and libraries to use the NOMAD API that come with different
+trade-offs between expressiveness, learning curve, and convinience.
-Besides search criteria you can determine how many results (`per_page`) and what page of
-results should be returned (`page`). If you want to go beyond the first 10.000 results
-you can use our *scroll* API (`scroll=true`, `scroll_after`). You can limit what properties
-should be returned (`include`, `exclude`). See the the generated [swagger dashboard
-of our API](https://nomad-lab.eu/prod/rae/api/) for more parameters.
+#### You can use your browser
-If you use the [NOMAD Repository and Archive search interface](https://nomad-lab.eu/prod/rae/gui/search)
-and create a query, you can click th a **<>**-button (right and on top of the result list).
-This will give you some code examples with URLs for your search query.
+For example to see the metadata for all entries with elements *Ti* and *O* go here:
+[{{ nomad_url() }}/v1/entries?elements=Ti&elements=O]({{ nomad_url() }}/v1/entries?elements=Ti&elements=O)
-Similar functionality is offered to download archive or raw data. Let's say you have
-identified an entry (given via a `upload_id`/`calc_id`, see the query output), and
-you want to download it:
+#### Use *curl* or *wget*
+REST API's use resources located via URLs. You access URLs with *curl* or *wget*. Same
+*Ti*, *O* example as before:
```sh
-curl "http://nomad-lab.eu/prod/rae/api/raw/calc/JvdvikbhQp673R4ucwQgiA/k-ckeQ73sflE6GDA80L132VCWp1z/*" -o download.zip
+curl "{{ nomad_url() }}/v1/entries?results.material.elements=Ti&results.material.elements=O" | python -m json.tool
```
-With `*` you basically requests all the files under an entry or path..
-If you need a specific file (that you already know) of that calculation:
+#### Use Python and *requests*
-```sh
-curl "http://nomad-lab.eu/prod/rae/api/raw/calc/JvdvikbhQp673R4ucwQgiA/k-ckeQ73sflE6GDA80L132VCWp1z/INFO.OUT"
-```
+Requests is a popular Python library to use the internets HTTP protocol that is used to
+communicate with REST APIs. Install with `pip install requests`.
+See [the initial example](#getting-started).
-You can also download a specific file from the upload (given a `upload_id`), if you know
-the path of that file:
+#### Use our *dashboard*
-```sh
-curl "http://nomad-lab.eu/prod/rae/api/raw/JvdvikbhQp673R4ucwQgiA/exciting_basis_set_error_study/monomers_expanded_k8_rgkmax_080_PBE/72_Hf/INFO.OUT"
-```
+The NOMAD API has an [OpenAPI dashboard](../api/v1). This is an interactive documentation of all
+API functions that allows you to try these functions in the browser.
-If you have a query
-that is more selective, you can also download all results. Here all compounds that only
-consist of Si, O, bulk material simulations of cubic systems (currently ~100 entries):
+#### Use NOMAD's Python package
+Install the [NOMAD Python client library](pythonlib) and use it's `ArchiveQuery`
+functionality for a more convenient query based access of archive data.
-```sh
-curl "http://nomad-lab.eu/prod/rae/api/raw/query?only_atoms=Si&only_atoms=O&system=bulk&crystal_system=cubic" -o download.zip
+## Different kinds of data
+
+We distinguish between different kinds of NOMAD data and there are different functions in
+the API:
+
+- Entry metadata, a summary of extracted data for an entry.
+- Raw files, the files as they were uploaded to NOMAD.
+- Archive data, all of the extracted data for an entry.
+
+There are also different entities (see also [Datamodel](index#datamodel)) with different functions in the API:
+
+- Entries
+- Uploads
+- Datasets
+- Users
+
+The API URLs typically start with the entity, followed by the kind of data. Examples
+are:
+
+- `entries/query` - Query entries for metadata
+- `entries/archive/query` - Query entries for archive data
+- `entries/{entry-id}/raw/download` - Download raw data for a specific entry
+- `uploads/{upload-id}/raw/path/to/file` - Download a specific file of an upload
+
+## Common concepts
+
+The [initial example](#getting-started) above, showed how to execute a basic search.
+This includes some fundamental concepts that can be applied to many parts of the API.
+Let's discuss some of the common concepts.
+
+### Response layout
+
+Functions that have a JSON response, will have a common layout. First, the response will contain all keys and values of the request. The request is not repeated verbatim, but
+in a normalized form. Abbreviations in search queries might be expanded, default values for optional parameters are added, or additional response specific information
+is included. Second, the response will contain the results under the key `data`.
+
+### Owner
+
+All functions that allow a query will also allow to specify the `owner`. Depending on
+the API function, its default value will be mostly `visible`. Some values are only
+available if you are [logged in](#authentication).
+
+{{ doc_snippet('owner')}}
+### Queries
+
+{{ doc_snippet('query') }}
+
+### Pagination
+
+Typically when you issue a query, not all results can be returned. Instead an API will
+typically only return one *page*. This behavior is controlled through pagination parameters,
+like `page_site`, `page`, `page_offset`, `page_after_value`.
+
+Let's consider a search for entries as an example.
+```py
+response = requests.post(
+ f'{base_url}/entries/query',
+ json={
+ 'query': {
+ 'results.material.elements': {
+ 'all': ['Ti', 'O']
+ }
+ },
+ 'pagination': {
+ 'page_size': 10
+ }
+ }
+)
```
-Here are a few more examples for downloading the raw data of based on DOI or dataset.
-You will have to encode non URL safe characters in potential dataset names (e.g. with a service like [www.urlencoder.org](https://www.urlencoder.org/)):
+This will only produce a response with max 10 entries in it. The response will contain a
+`pagination` object like this:
+```json
+{
+ "page_size": 10,
+ "order_by": "entry_id",
+ "order": "asc",
+ "total": 17957,
+ "next_page_after_value": "--SZVYOxA2jTu_L-mSxefSQFmeyF"
+}
+```
-```sh
-curl "http://nomad-lab.eu/prod/rae/api/raw/query?datasets.doi=10.17172/NOMAD/2020.03.18-1" -o download.zip
-curl "http://nomad-lab.eu/prod/rae/api/raw/query?dataset=Full%20ahnarmonic%20stAViC%20approach%3A%20Silicon%20and%20SrTiO3" -o download.zip
+In this case, the pagination is based on *after values*. This means that the search can
+be continued with a follow up request at a certain point characterized by the
+`next_page_after_value`. If you follow up with:
+
+```py
+response = requests.post(
+ f'{base_url}/entries/query',
+ json={
+ 'query': {
+ 'results.material.elements': {
+ 'all': ['Ti', 'O']
+ }
+ },
+ 'pagination': {
+ 'page_size': 10,
+ 'page_after_value': '--SZVYOxA2jTu_L-mSxefSQFmeyF'
+ }
+ }
+)
```
+You will get the next 10 results.
-In a similar way you can see the archive of an entry:
+### Authentication
-```sh
-curl "http://nomad-lab.eu/prod/rae/api/archive/f0KQE2aiSz2KRE47QtoZtw/6xe9fZ9xoxBYZOq5lTt8JMgPa3gX" | python -m json.tool
+Most of the API operations do not require any authorization and can be freely used
+without a user or credentials. However, to upload data, edit data, or view your own and potentially unpublished data, the API needs to authenticate you.
+
+The NOMAD API uses OAuth and tokens to authenticate users. We provide simple operations
+that allow you to acquire an *access token* via username and password based:
+
+```py
+import requests
+
+response = requests.get(
+ '{{ nomad_url() }}/v1/auth/token', params=dict(username='myname', password='mypassword'))
+token = response.json()['access_token']
+
+response = requests.get(
+ '{{ nomad_url() }}/v1/uploads',
+ headers={'Authorization': f'Bearer {token}'})
+uploads = response.json()['data']
```
-Or query and display the first page of 10 archives:
+If you have the [NOMAD Python package](pythonlib) installed. You can use its `Auth`
+implementation:
-```sh
-curl "http://nomad-lab.eu/prod/rae/api/archive/query?only_atoms=Si&only_atoms=O" | python -m json.tool
+```py
+import requests
+from nomad.client import Auth
+
+response = requests.get(
+ '{{ nomad_url() }}/v1/uploads',
+ auth=Auth(user='myname or email', password='mypassword'))
+uploads = response.json()['data']
```
-## Using Python's *request* library
+To use authentication in the dashboard, simply use the Authorize button. The
+dashboard GUI will manage the access token and use it while you try out the various
+operations.
-Similar to *curl* in the shell, you can use *requests* in Python. Its a generic HTTP
-client library that allows you to send requests:
+## Search for entries
-```python
-import requests
-import json
+See [getting started](#getting-started) for a typical search example. Combine the [different
+concepts](#common concepts) above to create the queries that you need.
+
+Searching for entries is typically just an initial step. Once you know what entries exist
+you'll probably want to do one of the following things.
-response = requests.get("http://nomad-lab.eu/prod/rae/api/archive/query?only_atoms=Si&only_atoms=O")
-data = response.json()
-print(json.dumps(data), indent=2)
+## Download raw files
+You can use [queries](#queries) to download raw files. But typically you don't want to
+download file-by-file or entry-by-entry. Therefore, we allow to download a large set of
+files in one big zip-file. Here, you might want to use a program like *curl* to download
+directly from the shell:
+
+```
+curl "{{ nomad_url() }}/v1/entries/raw/download?results.material.elements=Ti&results.material.elements=O" -o download.zip
```
-## NOMAD's Python client library
+## Access archives
+Above under [getting started](#getting started), you'll already learned how to access
+archive data. A speciality of archive API functions is that you can define is `required`
+from the archives.
+
+```
+response = requests.post(
+ f'{base_url}/entries/archive/query',
+ json={
+ 'query': ...,
+ 'pagination': ...,
+ 'required': {
+ 'workflow': {
+ 'calculation_result_ref': {
+ 'energy': '*',
+ 'system_ref': {
+ 'chemical_composition': '*'
+ }
+ }
+ }
+ }
+ })
+```
-This library is part devevloped by NOMAD. It is supposed to provide more powerful
-access to common yet complex tasks. It currently only support access to the NOMAD
-Archive. It has its separate documentation [here](pythonlib).
\ No newline at end of file
+{{ doc_snippet('archive-required') }}
\ No newline at end of file
diff --git a/docs/gui/archive.md b/docs/gui/archive.md
deleted file mode 100644
index 2b17a9716386ee025103d130e7c613a096c0df30..0000000000000000000000000000000000000000
--- a/docs/gui/archive.md
+++ /dev/null
@@ -1,3 +0,0 @@
-# About the Archive and Metainfo
-
-Comming soon ...
\ No newline at end of file
diff --git a/docs/gui/explore.md b/docs/gui/explore.md
deleted file mode 100644
index a578dff00669baf686f736fe11c2c3c1f746313a..0000000000000000000000000000000000000000
--- a/docs/gui/explore.md
+++ /dev/null
@@ -1,3 +0,0 @@
-# Explore data
-
-Comming soon ...
\ No newline at end of file
diff --git a/docs/gui/upload.md b/docs/gui/upload.md
deleted file mode 100644
index d5aa73acdcfc02095a4e5df3091697dc96763440..0000000000000000000000000000000000000000
--- a/docs/gui/upload.md
+++ /dev/null
@@ -1,166 +0,0 @@
-# Uploading and publishing data
-
-To contribute your data to the repository, please, login to our [upload page](https://nomad-lab.eu/prod/rae/gui/uploads)
-(you need to register first, if you do not have a NOMAD account yet).
-
-*A note for returning NOMAD users!* We revised the upload process with browser based upload
-alongside new shell commands. The new Upload page allows you to monitor upload processing
-and verify processing results before publishing your data to the Repository.
-
-The [upload page](https://nomad-lab.eu/prod/rae/gui/uploads) acts as a staging area for your data. It allows you to
-upload data, to supervise the processing of your data, and to examine all metadata that
-NOMAD extracts from your uploads. The data on the upload page will be private and can be
-deleted again. If you are satisfied with our processing, you can publish the data.
-Only then, data will become publicly available and cannot be deleted anymore.
-You will always be able to access, share, and download your data. You may curate your data
-and create datasets to give them a hierarchical structure. These functions are available
-from the Your data page by selecting and editing data.
-
-You should upload many files at the same time by creating .zip or .tar files of your folder structures.
-Ideally, input and output files are accompanied by relevant auxiliary files. NOMAD will
-consider everything within a single directory as related.
-
-Once published, data cannot be erased. Linking a corrected version to a corresponding older
-one ("erratum") will be possible soon. Files from an improved calculation, even for the
-same material, will be handled as a new entry.
-
-You can publish data as being open access or restricted for up to three years (with embargo).
-For the latter you may choose with whom you want to share your data. We strongly support the
-idea of open access and thus suggest to impose as few restrictions as possible from the very
-beginning. In case of open access data, all uploaded files are downloadable by any user.
-Additional information, e.g. pointing to publications or how your data should be cited,
-can be provided after the upload. Also DOIs can be requested. The restriction on data
-can be lifted at any time. You cannot restrict data that was published as open access.
-
-Unless published without an embargo, all your information will be private and only visible
-to you (or NOMAD users you explicitly shared your data with). Viewing private data will
-always require a login.
-
-By uploading you confirm authorship of the uploaded calculations. Co-authors must be specified
-after the upload process. This procedure is very much analogous to the submission of a
-publication to a scientific journal.
-
-Upload of data is free of charge.
-
-## Limits
-
-The following limitations apply to uploading:
-
-- One upload cannot exceed 32 GB in size
-- Only 10 non published uploads are allowed per user
-
-## On the supported codes
-
-NOMAD is interpreting your files. It will check each file and recognize if it is the
-main output file of one of the supported codes. NOMAD will create a entry for this *mainfile*
-that represents the respective data of this code run, experiment, etc. NOMAD only
-shows that for such recognized entries. If you uploads do not contain any files that
-NOMAD recognizes, you upload will be shown as empty and no data can be published.
-
-However, all files that are associated to a recognized *mainfile* by residing in the
-same directory, will be presented as *auxiliary* files along side the entry represented
-by the *mainfile*.
-
-### A note for VASP users
-
-On the handling of **POTCAR** files: NOMAD takes care of it; you don't
-need to worry about it. We understand that according to your VASP license, POTCAR files are
-not supposed to be visible to the public. Thus, in agreement with Georg Kresse, NOMAD will
-extract the most important information of POTCAR files and store it in the files named
-`POTCAR.stripped`. These files can be assessed and downloaded by anyone, while the original
-POTCAR files are removed when the upload is published. This is done automatically; you don't
-need to do anything.
-
-## Preparing an upload file
-
-You can upload .zip and .tar.gz files to NOMAD. The directory structure within can
-be arbitrary. Keep in mind that files in a single directory are all associated (see above).
-Ideally you only keep the files of a single (or closely related) code runs, experiments, etc.
-in one directory.
-
-You should not place files in additional archives within the upload file. NOMAD will not
-extract any zips in zips and similar entrapments.
-
-## Uploading large amounts of data
-
-This problem is many fold. In the remainder the following topics are discussed.
-
-- NOMAD restrictions about upload size and number of unpublished simultaneous uploads
-- Managing metadata (comments, references, co-authors, datasets) for a large number of entries
-- Safely transferring the data to NOMAD
-
-### General strategy
-
-Before you attempt to upload large amounts of data, do some experiments with a representative
-and small subset of your data. Use this to simulate a larger upload,
-checking and editing it the normal way. You do not have to publish this test upload;
-simply delete it before publish, once you are satisfied with the results.
-
-Ask for assistance. [Contact us](https://nomad-lab.eu/about/contact) in advance. This will
-allow us to react to your specific situation and eventually prepare additional measures.
-
-Keep enough time before you need your data to be published. Adding multiple hundreds of
-GBs to NOMAD isn't a trivial feat and will take some time and effort from all sides.
-
-### Upload restrictions
-
-The upload restrictions are necessary to keep NOMAD data in manageable chunks and we cannot
-simply grant exceptions to these rules.
-
-This means you have to split your data into 32 GB uploads. Uploading these files, observing
-the processing, and publishing the data can be automatized through NOMAD APIs.
-
-When splitting your data, it is important to not split sub-directories if they represent
-all files of a single entry. NOMAD can only bundle those related files to an entry if
-they are part of the same upload (and directory). Therefore, there is no single recipe to
-follow and a script to split your data will depend on how your data is organized.
-
-### Avoid additional operations on your data
-
-Changing the metadata of a large amounts of entries can be expensive and will also mean
-more work with our APIs. A simpler solution is to add the metadata directly to your uploads.
-This way NOMAD can pick it up automatically, no further actions required.
-
-Each NOMAD upload can contain a `nomad.json` file at the root. This file can contain
-metadata that you want to apply to all your entries. Here is an example:
-
-```
-{
- "comment": "Data from a cool research project",
- "references": ['http://archivex.org/mypaper'],
- "coauthors": [
- '<co-author-ids>',
- '<co-author-ids>'
- ]
- "datasets": [
- '<dataset-id>'
- ],
- "entries": {
- "path/to/calcs/vasp.xml": {
- "commit": "An entry specific comment."
- }
- }
-}
-```
-
-Another measure is to directly publish your data upon upload. After performing some
-smaller test upload, you should consider to skip our staging and publish the upload
-right away. This can save you some time and additional API calls. The upload endpoint
-has a parameter `publish_directly`. You can modify the upload command
-that you get from the upload page like this:
-
-```
-curl "http://nomad-lab.eu/prod/rae/api/uploads/?token=<your-token>&publish_directly=true" -T <local_file>
-```
-
-### Save transfer of files
-
-HTTP makes it easy for you to upload files via browser and curl, but it is not an
-ideal protocol for the stable transfer of large and many files. Alternatively, we can organize
-a separate manual file transfer to our servers. We will put your prepared upload
-files (.zip or .tag.gz) on a predefined path on the NOMAD servers. NOMAD allows to *"upload"*
-files directly from its servers via an additional `local_path` parameter:
-
-```
-curl -X PUT "http://nomad-lab.eu/prod/rae/api/uploads/?token=<your-token>&local_path=<path-to-upload-file>"
-```
diff --git a/docs/metainfo.md b/docs/metainfo.md
index c98908be8ca0f3b0d69a657a3ffb5dc6808af517..f40592913994c5c06dd88f63c990f4045e0e84c7 100644
--- a/docs/metainfo.md
+++ b/docs/metainfo.md
@@ -15,7 +15,7 @@ and molecular-dynamics (force-field) codes. This ensures a complete coverage of
material and molecule properties, even though some properties might not be as important as
others, or are missing in some input/output files of electronic-structure programs.
-
+
NOMAD Metainfo is kept independent of the actual storage format and is not bound to any
specific storage method. In our practical implementation, we use a binary form of JSON,
diff --git a/docs/web.md b/docs/web.md
new file mode 100644
index 0000000000000000000000000000000000000000..5caa2473c9c661a349c5fa4c94555cb2a592c372
--- /dev/null
+++ b/docs/web.md
@@ -0,0 +1,130 @@
+# Using the web interface
+
+For our upcoming tutorial on metadata management with NOMAD, we will produce a serious
+of videos that demonstrate the web interface of NOMAD v1. The videos will show how to
+
+- upload and publish data
+- search and download data
+- use the archive
+
+This will come in February 2022.
+
+
+## Uploading and publishing data
+
+
+### Preparing files
+
+You can upload files one by one, but you can also provider larger `.zip` or `.tar.gz`
+archive files, if this is easier to you. Also the file upload form the command line with
+curl or wget, is based an archive files. The specific layout of these files is up to you.
+NOMAD will simply extract them and consider the whole directory structure within.
+
+
+### Supported codes
+
+NOMAD is interpreting your files. It will check each file and recognize if it is the
+main output file of one of the supported codes. NOMAD will create a entry for this *mainfile*
+that represents the respective data of this code run, experiment, etc.
+
+While you can browse all files of an upload from its *upload page*, NOMAD only
+allows to search for such recognized entries. As long as your upload does not contain any
+files that NOMAD recognizes, you cannot be publish the upload.
+
+However, all files that are associated to a recognized *mainfile* by residing in the
+same directory, will be presented as *auxiliary* files along side the entry represented
+by the *mainfile*.
+
+**A note for VASP users**
+
+On the handling of **POTCAR** files: NOMAD takes care of it; you don't
+need to worry about it. We understand that according to your VASP license, POTCAR files are
+not supposed to be visible to the public. Thus, in agreement with Georg Kresse, NOMAD will
+extract the most important information of POTCAR files and store it in the files named
+`POTCAR.stripped`. These files can be assessed and downloaded by anyone, while the original
+POTCAR files are automatically removed.
+
+
+### User metadata
+
+NOMAD will automatically extract as much information as possible from your files. But you
+can still provide additional metadata. This is what we call *user metadata*. This includes
+you and your co-authors (use the *edit members* function on the *upload page*) as well
+as comments, additional web-references, and datasets (use the *edit metadata* function on
+the *upload page*).
+
+User metadata can also be provided in a file that is uploaded. This can be a `.json` or
+`.yaml` file. It has to be named `nomad.json` or `nomad.yaml`. Here is a JSON example:
+
+```json
+{
+ "comment": "Data from a cool research project",
+ "references": ["http://archivex.org/mypaper"],
+ "coauthors": [
+ "<email-or-username>",
+ "<email-or-username>"
+ ],
+ "datasets": [
+ "<dataset-name>"
+ ],
+ "entries": {
+ "path/to/calcs/vasp.xml": {
+ "commit": "An entry specific comment."
+ }
+ }
+}
+```
+
+This file is only applied on initial processing of an entry. So make sure you either
+upload it first, or upload everything as part of one archive file.
+
+
+### Limits
+
+- One upload cannot exceed **32 GB** in size.
+- Only **10 non published uploads** are allowed per user.
+- Only uploads at least one recognized entry can be published. See also [supported codes](#supported-codes) below.
+
+
+### Strategies for large amounts of data
+
+Before you attempt to upload large amounts of data, do some experiments with a representative
+and small subset of your data. Use this to simulate a larger upload,
+checking and editing it the normal way. You do not have to publish this test upload;
+simply delete it before publish, once you are satisfied with the results.
+
+Ask for assistance. [Contact us](https://nomad-lab.eu/about/contact) in advance. This will
+allow us to react to your specific situation and eventually prepare additional measures.
+Keep enough time before you need your data to be published. Adding multiple hundreds of
+GBs to NOMAD isn't a trivial feat and will take some time and effort from all sides.
+
+The [upload limits](#limits) above are necessary to keep NOMAD data manageable and we cannot easily
+grant exceptions to these rules. This means you have to split your data into 32 GB uploads. Uploading these files, observing the processing, and publishing the data can be automatized through NOMAD APIs.
+
+When splitting your data, it is important to not split sub-directories if they represent
+all files of a single entry. NOMAD can only bundle those related files to an entry if
+they are part of the same upload (and directory). Therefore, there is no single recipe to
+follow and a script to split your data will depend on how your data is organized.
+
+If you provide data for a potentially large amount of entries, it might be advisable
+to provide *user metadata* via file. See [user metadata](#user-metadata) above for details.
+
+To further automate, you can also upload and directly publish data. After performing some
+smaller test upload, you should consider to skip our staging and publish the upload
+right away. This can save you some time and additional API calls. The upload endpoint
+has a parameter `publish_directly`. You can modify the upload command
+that you get from the upload page like this:
+
+```
+curl "http://nomad-lab.eu/prod/rae/api/uploads/?token=<your-token>&publish_directly=true" -T <local_file>
+```
+
+HTTP makes it easy for you to upload files via browser and curl, but it is not an
+ideal protocol for the stable transfer of large and many files. Alternatively, we can organize
+a separate manual file transfer to our servers. We will put your prepared upload
+files (.zip or .tag.gz) on a predefined path on the NOMAD servers. NOMAD allows to *"upload"*
+files directly from its servers via an additional `local_path` parameter:
+
+```
+curl -X PUT "http://nomad-lab.eu/prod/rae/api/uploads/?token=<your-token>&local_path=<path-to-upload-file>"
+```
\ No newline at end of file
diff --git a/examples/api_use_authenticated.py b/examples/api/authenticated.py
similarity index 100%
rename from examples/api_use_authenticated.py
rename to examples/api/authenticated.py
diff --git a/examples/api/getting_started.py b/examples/api/getting_started.py
new file mode 100644
index 0000000000000000000000000000000000000000..781ccb957a6c41faa50c4a77a83b62fdad670e35
--- /dev/null
+++ b/examples/api/getting_started.py
@@ -0,0 +1,54 @@
+'''
+Demonstrates how to use requests for a simple query and archive access.
+'''
+
+import requests
+import json
+
+base_url = 'http://nomad-lab.eu/prod/v1/api/v1'
+
+response = requests.post(
+ f'{base_url}/entries/query',
+ json={
+ 'query': {
+ 'results.material.elements': {
+ 'all': ['Ti', 'O']
+ }
+ },
+ 'pagination': {
+ 'page_size': 1
+ },
+ 'required': {
+ 'include': ['entry_id']
+ }
+ })
+response_json = response.json()
+print(json.dumps(response.json(), indent=2))
+
+
+first_entry_id = response_json['data'][0]['entry_id']
+response = requests.post(
+ f'{base_url}/entries/{first_entry_id}/archive/query',
+ json={
+ 'required': {
+ 'workflow': {
+ 'calculation_result_ref': {
+ 'energy': '*',
+ 'system_ref': {
+ 'chemical_composition': '*'
+ }
+ }
+ }
+ }
+ })
+response_json = response.json()
+print(json.dumps(response_json, indent=2))
+
+
+from nomad.datamodel import EntryArchive
+from nomad.metainfo import units
+
+archive = EntryArchive.m_from_dict(response_json['data']['archive'])
+result = archive.workflow[0].calculation_result_ref
+print(result.system_ref.chemical_composition)
+print(result.energy.total.value.to(units('eV')))
diff --git a/examples/api_use.py b/examples/api_use.py
deleted file mode 100644
index abf168b3a1931e923f0e524c346e6d2a7abdeb69..0000000000000000000000000000000000000000
--- a/examples/api_use.py
+++ /dev/null
@@ -1,25 +0,0 @@
-'''
-This is a brief example on how to use the public API.
-'''
-import requests
-
-from nomad import config
-
-
-nomad_url = config.client.url
-
-# perform the search request to print number of public entries
-response = requests.post(
- f'{nomad_url}/v1/entries',
- json={
- 'query': {
- 'results.material.elements:any': ['Si', 'O']
- }
- })
-response_data = response.json()
-
-# print the total ammount of search results
-print(response_data['pagination']['total'])
-
-# print the data of the first result
-print(response_data['data'][0])
diff --git a/examples/tutorials/api_with_archive_query.py b/examples/archive/archive_query.py
similarity index 100%
rename from examples/tutorials/api_with_archive_query.py
rename to examples/archive/archive_query.py
diff --git a/examples/tutorials/api_with_requests.py b/examples/tutorials/api_with_requests.py
deleted file mode 100644
index aad036fb72f8755f65de5f3445b9be0b8a02c1b8..0000000000000000000000000000000000000000
--- a/examples/tutorials/api_with_requests.py
+++ /dev/null
@@ -1,26 +0,0 @@
-# type: ignore
-
-'''
-A simple example used in the NOMAD webinar API tutorial
-'''
-
-import requests
-import json
-
-base_url = 'http://nomad-lab.eu/prod/rae/api'
-
-# response = requests.get(base_url + '/repo?datasets.dataset_name=NOMAD%20webinar')
-response = requests.get(
- base_url + '/repo',
- params={'datasets.dataset_name': 'NOMAD webinar', 'per_page': 1})
-
-data = response.json()
-upload_id = data['results'][0]['upload_id']
-calc_id = data['results'][0]['calc_id']
-
-response = requests.get(
- base_url + '/archive/%s/%s' % (upload_id, calc_id))
-
-print(json.dumps(response.json(), indent=2))
-print(
- response.json()['section_run'][0]['section_single_configuration_calculation'][-1]['section_dos'][0]['dos_energies'])
diff --git a/examples/external_project_upload/example.zip b/examples/uploading/external_project_upload/example.zip
similarity index 100%
rename from examples/external_project_upload/example.zip
rename to examples/uploading/external_project_upload/example.zip
diff --git a/examples/external_project_upload/upload.py b/examples/uploading/external_project_upload/upload.py
similarity index 100%
rename from examples/external_project_upload/upload.py
rename to examples/uploading/external_project_upload/upload.py
diff --git a/mkdocs.yml b/mkdocs.yml
index ee346764a7427e4548533824c698fa88b2f3a55b..0b6f72f5922e8d61bd48e7198322abcfbf2fdb51 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -4,11 +4,8 @@ site_description: |
site_author: The NOMAD Authors
nav:
- Introduction: index.md
- - Using the web interface:
- - gui/upload.md
- - gui/explore.md
- - gui/archive.md
- - Using the API: api.md
+ - web.md
+ - api.md
- Using the AI Toolkit and other remote tools: aitoolkit.md
- Using the Python library: pythonlib.md
- Extending and Developing NOMAD:
@@ -33,6 +30,13 @@ markdown_extensions:
- pymdownx.inlinehilite
- pymdownx.snippets
- pymdownx.superfences
+ - toc:
+ permalink: True
+ toc_depth: 3
extra:
generator: false
-use_directory_urls: false
\ No newline at end of file
+use_directory_urls: false
+plugins:
+ - search
+ - macros:
+ module_name: nomad/mkdocs
\ No newline at end of file
diff --git a/nomad/app/v1/main.py b/nomad/app/v1/main.py
index b785cdb51266f5f24ef7fb0165044f3866808b3e..32b98c8cf0297ceeb5bb45fbe1164b7a9899e3ce 100644
--- a/nomad/app/v1/main.py
+++ b/nomad/app/v1/main.py
@@ -29,6 +29,7 @@ from .routers import users, entries, materials, auth, info, datasets, uploads, s
logger = utils.get_logger(__name__)
+
app = FastAPI(
root_path=root_path,
openapi_url='/openapi.json',
@@ -38,112 +39,8 @@ app = FastAPI(
title='NOMAD API',
version='v1, NOMAD %s@%s' % (config.meta.version, config.meta.commit),
description=utils.strip(f'''
- ## Getting started
-
- Here is a simple example on how to execute a search with Python and requests:
- ```python
- import requests
-
- response = requests.post(
- '{config.client.url}/v1/entry/query',
- json={{
- 'query': {{
- 'results.material.elements:any': ['Si', 'O']
- }}
- }})
- response_data = response.json()
- n_search_results = response_data['pagination']['total']
- search_results = response_data['data']
- ```
-
- ## Conventions
-
- ### Paths
-
- The various API operations are organized with the following path scheme. The first
- part of the path, describes the data entity that is covered by
- the operations below (e.g. `entries`, `users`, `datasets`, `uploads`). For example
- everything below `entries` will be about searching entries, getting
- an entry, editing entries, etc.
-
- The second (optional and variable) path segment allows to denote a specific entity instance,
- e.g. a specific entry or dataset, usually by id. With out such a variable second
- path segment, its about all instances, e.g. searching entries or listing all datasets.
-
- Optional (if available) further path segments will determine the variety and format
- of data. This is mostly for entries to distinguish the metadata, raw, and archive
- data or distinguish between listing (i.e. paginated json) and downloading
- (i.e. streaming a zip-file)
-
- Further, we try to adhere to the paradim of getting and posting resources. Therefore,
- when you post a complex query, you will not post it to `/entries` (a query is not an entry),
- but `/entries/query`. Here *query* being a kind of virtual resource.
-
- ### Parameters and bodies for GET and POST operations
-
- We offer **GET** and **POST** versions for many complex operations. The idea is that
- **GET** is easy to use, e.g. via curl or simply in the browser, while **POST**
- allows to provide more complex parameters (i.e. a JSON body). For example to
- search for entries, you can use the **GET** operation `/entries` to specify simple
- queries via URL, e.g. `/entries?code_name=VASP&atoms=Ti`, but you would use
- **POST** `/entries/query` to provide a complex nested queries, e.g. with logical
- operators.
-
- Typicall the **POST** version is a super-set of the functionality of the **GET**
- version. But, most top-level parameters in the **POST** body, will be available
- in the **GET** version as URL parameters with the same name and meaning. This
- is especially true for reoccuring parameters for general API concepts like pagination
- or specifying required result fields.
-
- ### Response layout
-
- Typically a response will mirror all input parameters in the normalized form that
- was used to perform the operation.
-
- Some of these will be augmented with result values. For example the pagination
- section of a request will be augmented with the total available number.
-
- The actual requested data, will be placed under the key `data`.
-
- ## About Authentication
-
- NOMAD is an open datasharing platform, and most of the API operations do not require
- any authorization and can be freely used without a user or credentials. However,
- to upload data, edit data, or view your own and potentially unpublished data,
- the API needs to authenticate you.
-
- The NOMAD API uses OAuth and tokens to authenticate users. We provide simple operations
- that allow you to acquire an *access token* via username and password based:
-
- ```
- import requests
-
- response = requests.get(
- '{config.client.url}/v1/auth/token', params=dict(username='myname', password='mypassword'))
- token = response.json()['access_token']
-
- response = requests.get(
- '{config.client.url}/v1/uploads',
- headers={{'Authorization': f'Bearer {{token}}'}})
- uploads = response.json()['data']
- ```
-
- If you have the `nomad-lab` Python module installed. You can use its `Auth`
- implementation:
-
- ```
- import requests
- from nomad.client import Auth
-
- response = requests.get(
- '{config.client.url}/v1/uploads',
- auth=Auth(user='myname or email', password='mypassword'))
- uploads = response.json()['data']
- ```
-
- To use authentication in the dashboard, simply use the Authorize button. The
- dashboard GUI will manage the access token and use it while you try out the various
- operations.
+ Please visit the [API section of the NOMAD documentation]({config.api_url(True, 'docs/api.html')})
+ for a introduction and examples.
'''))
app.add_middleware(
diff --git a/nomad/app/v1/models.py b/nomad/app/v1/models.py
index a9f5ed467256dacbfbb99b26cbdde7155e6287a2..d7530d6464ff1d8c6f498b14007952e47c63c0f3 100644
--- a/nomad/app/v1/models.py
+++ b/nomad/app/v1/models.py
@@ -52,23 +52,26 @@ Value = Union[StrictInt, StrictFloat, StrictBool, str, datetime.datetime]
ComparableValue = Union[StrictInt, StrictFloat, str, datetime.datetime]
+owner_documentation = strip('''
+The `owner` allows to limit the scope of the searched based on entry ownership.
+This is useful, if you only want to search among all publically downloadable
+entries, or only among your own entries, etc.
+
+These are the possible owner values and their meaning:
+
+* `public`: Consider all entries that can be publically downloaded, i.e. only published entries without embargo.
+* `user`: Only consider entries that belong to you.
+* `shared`: Only consider entries that belong to you or are shared with you.
+* `visible`: Consider all entries that are visible to you. This includes
+ entries with embargo or unpublished entries that belong to you or are
+ shared with you.
+* `staging`: Only search through unpublished entries.
+* `all`: Consider all entries.
+''')
+
+
class Owner(str, enum.Enum):
- '''
- The `owner` allows to limit the scope of the searched based on entry ownership.
- This is useful, if you only want to search among all publically downloadable
- entries, or only among your own entries, etc.
-
- These are the possible owner values and their meaning:
- * `all`: Consider all entries.
- * `public` (default): Consider all entries that can be publically downloaded,
- i.e. only published entries without embargo
- * `user`: Only consider entries that belong to you.
- * `shared`: Only consider entries that belong to you or are shared with you.
- * `visible`: Consider all entries that are visible to you. This includes
- entries with embargo or unpublished entries that belong to you or are
- shared with you.
- * `staging`: Only search through unpublished entries.
- '''
+ owner_documentation
# There seems to be a slight bug in fast API. When it creates the example in OpenAPI
# it will ignore any given default or example and simply take the first enum value.
@@ -213,93 +216,96 @@ Not.update_forward_refs()
Nested.update_forward_refs()
-class WithQuery(BaseModel):
- owner: Optional[Owner] = Body('public')
- query: Optional[Query] = Body(
- None,
- embed=True,
- description=strip('''
- A query can be very simple list of parameters. Different parameters are combined
- with a logical **and**, values of the same parameter with also with a logical **and**.
- The following would search for all entries that are VASP calculations,
- contain *Na* **and** *Cl*, **and** are authored by *Stefano Curtarolo*
- **and** *Chris Wolverton*.
- ```
- {
- "results.material.elements": ["Na", "Cl"],
- "results.method.simulation.program_name": "VASP",
- "authors": ["Stefano Curtarolo", "Chris Wolverton"]
- }
- ```
-
- A short cut to change the logical combination of values in a list, is to
- add a suffix to the quantity `:any`:
- ```
- {
- "results.material.elements": ["Na", "Cl"],
- "results.method.simulation.program_name": "VASP",
- "authors:any": ["Stefano Curtarolo", "Chris Wolverton"]
- }
- ```
-
- Otherwise, you can also write complex logical combinations of parameters like this:
- ```
- {
- "and": [
- {
- "or": [
- {
- "results.material.elements": ["Cl", "Na"]
- },
- {
- "results.material.elements": ["H", "O"]
- }
- ]
- },
- {
- "not": {
- "results.material.symmetry.crystal_system": "cubic"
- }
- }
- ]
- }
- ```
- Other short-cut prefixes are `none:` and `any:` (the default).
-
- By default all quantity values have to **equal** the given values to match. For
- some values you can also use comparison operators like this:
- ```
- {
- "upload_create_time": {
- "gt": "2020-01-01",
- "lt": "2020-08-01"
+query_documentation = strip('''
+A query can be very simple list of parameters. Different parameters are combined
+with a logical **and**, values of the same parameter with also with a logical **and**.
+The following would search for all entries that are VASP calculations,
+contain *Na* **and** *Cl*, **and** are authored by *Stefano Curtarolo*
+**and** *Chris Wolverton*.
+```json
+{
+ "results.material.elements": ["Na", "Cl"],
+ "results.method.simulation.program_name": "VASP",
+ "authors": ["Stefano Curtarolo", "Chris Wolverton"]
+}
+```
+
+A short cut to change the logical combination of values in a list, is to
+add a suffix to the quantity `:any`:
+```json
+{
+ "results.material.elements": ["Na", "Cl"],
+ "results.method.simulation.program_name": "VASP",
+ "authors:any": ["Stefano Curtarolo", "Chris Wolverton"]
+}
+```
+
+Otherwise, you can also write complex logical combinations of parameters like this:
+```json
+{
+ "and": [
+ {
+ "or": [
+ {
+ "results.material.elements": ["Cl", "Na"]
},
- "results.properties.geometry_optimization.final_energy_difference": {
- "lte": 1.23e-18
+ {
+ "results.material.elements": ["H", "O"]
}
+ ]
+ },
+ {
+ "not": {
+ "results.material.symmetry.crystal_system": "cubic"
}
- ```
+ }
+ ]
+}
+```
+Other short-cut prefixes are `none:` and `any:` (the default).
+
+By default all quantity values have to **equal** the given values to match. For
+some values you can also use comparison operators like this:
+```json
+{
+ "upload_create_time": {
+ "gt": "2020-01-01",
+ "lt": "2020-08-01"
+ },
+ "results.properties.geometry_optimization.final_energy_difference": {
+ "lte": 1.23e-18
+ }
+}
+```
+
+or shorter with suffixes:
+```json
+{
+ "upload_create_time:gt": "2020-01-01",
+ "upload_create_time:lt": "2020-08-01",
+ "results.properties.geometry_optimization.final_energy_difference:lte" 1.23e-18
+}
+```
- or shorter with suffixes:
- ```
- {
- "upload_create_time:gt": "2020-01-01",
- "upload_create_time:lt": "2020-08-01",
- "results.properties.geometry_optimization.final_energy_difference:lte" 1.23e-18
- }
- ```
+The searchable quantities are a subset of the NOMAD Archive quantities defined
+in the NOMAD Metainfo. The searchable quantities also depend on the API endpoint.
- The searchable quantities are a subset of the NOMAD Archive quantities defined
- in the NOMAD Metainfo. The searchable quantities also depend on the API endpoint.
+There is also an additional query parameter that you can use to formulate queries based
+on the optimade filter language:
+```json
+{
+ "optimade_filter": "nelements >= 2 AND elements HAS ALL 'Ti', 'O'"
+}
+```
+''')
- There is also an additional query parameter that you can use to formulate queries based
- on the optimade filter language:
- ```
- {
- "optimade_filter": "nelements >= 2 AND elements HAS ALL 'Ti', 'O'"
- }
- ```
- '''), # TODO custom documentation for entry and material API
+
+class WithQuery(BaseModel):
+ owner: Optional[Owner] = Body('public')
+ query: Optional[Query] = Body(
+ None,
+ embed=True,
+ description=query_documentation,
example={
'upload_create_time:gt': '2020-01-01',
'results.material.elements': ['Ti', 'O'],
diff --git a/nomad/app/v1/routers/entries.py b/nomad/app/v1/routers/entries.py
index a6f73db67d661f742a370b929ebd44c11d0bb67f..a27c513d2e3ff35fca2d92d3f67ced9ac7b685c0 100644
--- a/nomad/app/v1/routers/entries.py
+++ b/nomad/app/v1/routers/entries.py
@@ -65,95 +65,98 @@ logger = utils.get_logger(__name__)
query_parameters = QueryParameters(doc_type=entry_type)
-ArchiveRequired = Union[str, Dict[str, Any]]
-
-_archive_required_field = Body(
- '*',
- embed=True,
- description=strip('''
- The `required` part allows you to specify what parts of the requested archives
- should be returned. The NOMAD Archive is a hierarchical data format and
- you can *require* certain branches (i.e. *sections*) in the hierarchy.
- By specifing certain sections with specific contents or all contents (via
- the directive `"*"`), you can determine what sections and what quantities should
- be returned. The default is the whole archive, i.e., `"*"`.
-
- For example to specify that you are only interested in the `metadata`
- use:
-
- ```
- {
- "metadata": "*"
+archive_required_documentation = strip('''
+The `required` part allows you to specify what parts of the requested archives
+should be returned. The NOMAD Archive is a hierarchical data format and
+you can *require* certain branches (i.e. *sections*) in the hierarchy.
+By specifing certain sections with specific contents or all contents (via
+the directive `"*"`), you can determine what sections and what quantities should
+be returned. The default is the whole archive, i.e., `"*"`.
+
+For example to specify that you are only interested in the `metadata`
+use:
+
+```json
+{
+ "metadata": "*"
+}
+```
+
+Or to only get the `energy_total` from each individual calculations, use:
+```json
+{
+ "run": {
+ "configuration": {
+ "energy": "*"
}
- ```
-
- Or to only get the `energy_total` from each individual calculations, use:
- ```
- {
- "run": {
- "configuration": {
- "energy": "*"
+ }
+}
+```
+
+You can also request certain parts of a list, e.g. the last calculation:
+```json
+{
+ "run": {
+ "calculation[-1]": "*"
+ }
+}
+```
+
+These required specifications are also very useful to get workflow results.
+This works because we can use references (e.g. workflow to final result calculation)
+and the API will resolve these references and return the respective data.
+For example just the total energy value and reduced formula from the resulting
+calculation:
+```json
+{
+ "workflow": {
+ "calculation_result_ref": {
+ "energy": "*",
+ "system_ref": {
+ "value": {
+ "chemical_composition": "*"
}
}
}
- ```
+ }
+}
+```
+
+You can also resolve all references in a branch with the `include-resolved`
+directive. This will resolve all references in the branch, and also all references
+in referenced sections:
+```json
+{
+ "workflow":
+ "calculation_result_ref": "include-resolved"
+ }
+}
+```
+
+By default, the targets of "resolved" references are added to the archive at
+their original hierarchy positions.
+This means, all references are still references, but they are resolvable within
+the returned data, since they targets are now part of the data. Another option
+is to add
+`"resolve-inplace": true` to the root of required. Here, the reference targets will
+replace the references:
+```json
+{
+ "resolve-inplace": true,
+ "workflow":
+ "calculation_result_ref": "include-resolved"
+ }
+}
+```
+''')
- You can also request certain parts of a list, e.g. the last calculation:
- ```
- {
- "run": {
- "calculation[-1]": "*"
- }
- }
- ```
-
- These required specifications are also very useful to get workflow results.
- This works because we can use references (e.g. workflow to final result calculation)
- and the API will resolve these references and return the respective data.
- For example just the total energy value and reduced formula from the resulting
- calculation:
- ```
- {
- "workflow": {
- "calculation_result_ref": {
- "energy": "*",
- "system_ref": {
- "value": {
- "chemical_composition": "*"
- }
- }
- }
- }
- }
- ```
-
- You can also resolve all references in a branch with the `include-resolved`
- directive. This will resolve all references in the branch, and also all references
- in referenced sections:
- ```
- {
- "workflow":
- "calculation_result_ref": "include-resolved"
- }
- }
- ```
-
- By default, the targets of "resolved" references are added to the archive at
- their original hierarchy positions.
- This means, all references are still references, but they are resolvable within
- the returned data, since they targets are now part of the data. Another option
- is to add
- `"resolve-inplace": true` to the root of required. Here, the reference targets will
- replace the references:
- ```
- {
- "resolve-inplace": true,
- "workflow":
- "calculation_result_ref": "include-resolved"
- }
- }
- ```
- '''),
+
+ArchiveRequired = Union[str, Dict[str, Any]]
+
+_archive_required_field = Body(
+ '*',
+ embed=True,
+ description=archive_required_documentation,
example={
'run': {
'calculation[-1]': {
diff --git a/nomad/mkdocs.py b/nomad/mkdocs.py
new file mode 100644
index 0000000000000000000000000000000000000000..a0082d1c20dbfc09d7c2de1a1c4f49b13e335b37
--- /dev/null
+++ b/nomad/mkdocs.py
@@ -0,0 +1,43 @@
+#
+# Copyright The NOMAD Authors.
+#
+# This file is part of NOMAD. See https://nomad-lab.eu for further info.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+'''
+Definitions that are used in the documentation via mkdocs-macro-plugin.
+'''
+
+from nomad.app.v1.models import (
+ query_documentation,
+ owner_documentation)
+from nomad.app.v1.routers.entries import archive_required_documentation
+from nomad import config
+
+doc_snippets = {
+ 'query': query_documentation,
+ 'owner': owner_documentation,
+ 'archive-required': archive_required_documentation
+}
+
+
+def define_env(env):
+ @env.macro
+ def nomad_url(): # pylint: disable=unused-variable
+ return config.api_url()
+
+ @env.macro
+ def doc_snippet(key): # pylint: disable=unused-variable
+ return doc_snippets[key]
diff --git a/requirements.txt b/requirements.txt
index 3469856e13ca0a300de44ff9d71349ff9b88039c..d28d6e758f07f88f616b2aa660e7d91ea41743e2 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -109,3 +109,4 @@ devtools==0.8.0
mkdocs==1.2.3
mkdocs-material==8.1.1
mkdocs-material-extensions==1.0.3
+mkdocs-macros-plugin==0.6.3
\ No newline at end of file