diff --git a/docs/api.md b/docs/apis/api.md
similarity index 92%
rename from docs/api.md
rename to docs/apis/api.md
index 29a64c5712a66de6fe1c0185e5e525a79e01beb0..d6b7f796c55b7a0fc64d36b887d05351a012b9b1 100644
--- a/docs/api.md
+++ b/docs/apis/api.md
@@ -1,15 +1,54 @@
-# Using the APIs
+This guide is about using NOMAD's REST APIs directly, e.g. via Python's *request*.
-## Tutorial
+To access the processed data with our client library `nomad-lab` follow
+[How to access the processed data](archive_query.md). You watch our
+[video tutorial on the API](../tutorial.md#access-data-via-api).
-This video tutorial explains the basics of API and shows how to do simple requests
-against the NOMAD api.
+## Different options to use the API
-<div class="youtube">
-<iframe src="https://www.youtube.com/embed/G1frBCrxC0g" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
-</div>
+NOMAD offers all its functionality through application
+programming interfaces (APIs). More specifically [RESTful HTTP APIs](https://en.wikipedia.org/wiki/Representational_state_transfer) that allows you
+to use NOMAD as a set of resources (think data) that can be uploaded, accessed, downloaded,
+searched for, etc. via [HTTP requests](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol).
+
+You can get an overview on all NOMAD APIs on the [API page]({{ nomad_url() }}/v1/gui/analyze/apis).
+We will focus here on NOMAD's main API (v1). In fact, this API is also used by
+the web interface and should provide everything you need.
+
+There are different tools and libraries to use the NOMAD API that come with different
+trade-offs between expressiveness, learning curve, and convenience.
+
+<h4>You can use your browser</h4>
+
+For example to see the metadata for all entries with elements *Ti* and *O* go here:
+[{{ nomad_url() }}/v1/entries?elements=Ti&elements=O]({{ nomad_url() }}/v1/entries?elements=Ti&elements=O)
+
+<h4>Use *curl* or *wget*</h4>
+
+REST API's use resources located via URLs. You access URLs with *curl* or *wget*. Same
+*Ti*, *O* example as before:
+```sh
+curl "{{ nomad_url() }}/v1/entries?results.material.elements=Ti&results.material.elements=O" | python -m json.tool
+```
+
+<h4> Use Python and *requests*</h4>
+
+Requests is a popular Python library to use the internets HTTP protocol that is used to
+communicate with REST APIs. Install with `pip install requests`.
+See [the initial example](#using-request).
+
+<h4>Use our *dashboard*</h4>
+
+The NOMAD API has an [OpenAPI dashboard]({{ nomad_url() }}/v1/api/v1). This is an interactive documentation of all
+API functions that allows you to try these functions in the browser.
+
+<h4>Use NOMAD's Python package</h4>
+
+Install the [NOMAD Python client library](pythonlib.md) and use it's `ArchiveQuery`
+functionality for a more convenient query based access of archive data following the
+[How to access the processed data](archive_query.md) guide.
-## Getting started
+## Using request
If you are comfortable with REST APIs and using Pythons `requests` library, this example
demonstrates the basic concepts of NOMAD's main API. You can get more documentation and
@@ -184,7 +223,7 @@ The result will look like this:
You can work with the results in the given JSON (or respective Python dict/list) data already.
If you have [NOMAD's Python library](pythonlib.md) installed ,
you can take the archive data and use the Python interface.
-The [Python interface](schema/python.md#wrap-data-with-python-schema-classes) will help with code-completion (e.g. in notebook environments),
+The [Python interface](../plugins/schemas.md#wrap-data-with-python-schema-classes) will help with code-completion (e.g. in notebook environments),
resolve archive references (e.g. from workflow to calculation to system), and allow unit conversion:
```py
from nomad.datamodel import EntryArchive
@@ -201,47 +240,6 @@ This will give you an output like this:
OOSrTiOOOSrTiOOOSrTiOFF
-355626.93095025205 electron_volt
```
-## Overview
-
-NOMAD offers all its functionality through application
-programming interfaces (APIs). More specifically [RESTful HTTP APIs](https://en.wikipedia.org/wiki/Representational_state_transfer) that allows you
-to use NOMAD as a set of resources (think data) that can be uploaded, accessed, downloaded,
-searched for, etc. via [HTTP requests](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol).
-
-You can get an overview on all NOMAD APIs on the [API page]({{ nomad_url() }}/v1/gui/analyze/apis).
-We will focus here on NOMAD's main API (v1). In fact, this API is also used by
-the web interface and should provide everything you need.
-
-There are different tools and libraries to use the NOMAD API that come with different
-trade-offs between expressiveness, learning curve, and convenience.
-
-#### You can use your browser
-
-For example to see the metadata for all entries with elements *Ti* and *O* go here:
-[{{ nomad_url() }}/v1/entries?elements=Ti&elements=O]({{ nomad_url() }}/v1/entries?elements=Ti&elements=O)
-
-#### Use *curl* or *wget*
-
-REST API's use resources located via URLs. You access URLs with *curl* or *wget*. Same
-*Ti*, *O* example as before:
-```sh
-curl "{{ nomad_url() }}/v1/entries?results.material.elements=Ti&results.material.elements=O" | python -m json.tool
-```
-
-#### Use Python and *requests*
-
-Requests is a popular Python library to use the internets HTTP protocol that is used to
-communicate with REST APIs. Install with `pip install requests`.
-See [the initial example](#getting-started).
-
-#### Use our *dashboard*
-
-The NOMAD API has an [OpenAPI dashboard]({{ nomad_url() }}/v1/api/v1). This is an interactive documentation of all
-API functions that allows you to try these functions in the browser.
-
-#### Use NOMAD's Python package
-Install the [NOMAD Python client library](pythonlib.md) and use it's `ArchiveQuery`
-functionality for a more convenient query based access of archive data.
## Different kinds of data
@@ -252,7 +250,7 @@ the API:
- Raw files, the files as they were uploaded to NOMAD.
- Archive data, all of the extracted data for an entry.
-There are also different entities (see also [Datamodel](index.md#datamodel-uploads-entries-files-datasets)) with different functions in the API:
+There are also different entities (see also [Datamodel](../learn/how_nomad_works.md#datamodel-uploads-entries-files-datasets)) with different functions in the API:
- Entries
- Uploads
@@ -410,7 +408,7 @@ directly from the shell:
curl "{{ nomad_url() }}/v1/entries/raw?results.material.elements=Ti&results.material.elements=O" -o download.zip
```
-## Access archives
+## Access processed data (archives)
Above under [getting started](#getting started), you've already learned how to access
archive data. A special feature of the archive API functions is that you can define what is `required`
from the archives.
diff --git a/docs/archive_query.md b/docs/apis/archive_query.md
similarity index 96%
rename from docs/archive_query.md
rename to docs/apis/archive_query.md
index d65e903405bade9de86d2c9199b27367263150dc..95e6f7bec60537523dc720fa2eba65517e44fd85 100644
--- a/docs/archive_query.md
+++ b/docs/apis/archive_query.md
@@ -1,10 +1,14 @@
-# Query Processed Data
-
The `ArchiveQuery` allows you to search for entries and access their parsed and processed *archive* data
at the same time. Furthermore, all data is accessible through a convenient Python interface
-based on the [NOMAD metainfo](schema/python.md#wrap-data-with-python-schema-classes) rather than plain JSON.
+based on the schema rather than plain JSON. See also this guide on using
+[NOMAD's Python schemas](../plugins/schemas.md#use-python-schemas-to-work-with-data)
+to work with processed data.
+
+As a requirement, you have to install the `nomad-lab` Python package. Follow the
+[How to install nomad-lab](pythonlib.md) guide.
+
-## Basic Usage
+## Getting started
To define a query, one can, for example, write
diff --git a/docs/local_parsers.md b/docs/apis/local_parsers.md
similarity index 59%
rename from docs/local_parsers.md
rename to docs/apis/local_parsers.md
index c36981be0662a44e6d98bc45d4cb9841f36df0a4..c8de75779c28655adcddbba91fe18330a5ecbd69 100644
--- a/docs/local_parsers.md
+++ b/docs/apis/local_parsers.md
@@ -1,14 +1,35 @@
-# Run NOMAD parser locally
+# How to run a parser
-If you install `nomad-lab`, you can use the NOMAD parsers locally on your computer.
-To use the NOMAD parsers from the command line, you can use the parse CLI command. The parse command will automatically
-match the right parser to your code output file and run the parser. There are two output formats, `--show-metadata` (a
-JSON representation of the basic metadata) and `--show-archive` (a JSON representation of the full parse results).
+You can find a [list of all parsers](../reference/parsers.md) and supported files in the reference.
-```sh
+First you need to have the `nomad-lab` pypi package installed. You find more detailed
+instructions [here](pythonlib.md):
+
+```
+pip install nomad-lab
+```
+
+## From the command line
+
+You can run NOMAD parsers from the [command line interface](../reference/cli.md) (CLI).
+The parse command will automatically match the right parser to your file and run the parser.
+There are two output formats:
+
+- `--show-metadata` a JSON representation of the basic metadata
+- `--show-archive` a JSON representation of the full parse results
+
+```
nomad parse --show-archive <path-to-your-mainfile-code-output-file>
```
+!!! note
+
+ If you run into missing dependency errors, you might want to install additional
+ dependencies via `pip install nomad-lab[parsing]`. Only a view parsers require
+ extra dependencies. Please refer to the parser projects for more details.
+
+## From a python program
+
You can also use the NOMAD parsers within Python, as shown below.
This will give you the parse results as metainfo objects to conveniently analyze the results in Python.
See metainfo for more details on how to use the metainfo in Python.
@@ -31,6 +52,8 @@ for archive in archives:
python_dict = section_run.m_to_dict()
```
+## From cloned parser projects
+
You can also clone a parser project to debug or fix a parser:
```sh
diff --git a/docs/pythonlib.md b/docs/apis/pythonlib.md
similarity index 89%
rename from docs/pythonlib.md
rename to docs/apis/pythonlib.md
index 2b3ad1f5d3af04023dcf9686539a193943fc2236..af8a60d29f39de70e20080678dd67107ef99a985 100644
--- a/docs/pythonlib.md
+++ b/docs/apis/pythonlib.md
@@ -16,7 +16,7 @@ pip install nomad-lab
The latest develop versions might still be considered beta and might not be published to
pypi. If you require specific new features you might need to install `nomad-lab`
from our GitLab package registry. To use features of a specific commit or
- branch, consider to [clone and build the project](./develop/setup.md) yourself.
+ branch, consider to [clone and build the project](../develop/setup.md) yourself.
To install the latest release developer releases from our GitLab use:
@@ -45,6 +45,7 @@ pip install nomad-lab[dev]
The various extras have the following meaning:
- *parsing*, run all parsers, incl. parsers based on HDF5, netCDF, or asr
-- *infrastructure*, everything to run NOMAD services, see also [Oasis documentation](oasis.md#base-linux-without-docker)
+- *infrastructure*, everything to run NOMAD services, see also
+[Oasis documentation](../oasis/install.md#base-linux-without-docker)
- *dev*, necessary to run development and build tools, e.g. pytest, pylint, mypy
diff --git a/docs/data/eln.md b/docs/data/eln.md
new file mode 100644
index 0000000000000000000000000000000000000000..74dcb39b5764da3d2b0ad5629ed923fb325eeefc
--- /dev/null
+++ b/docs/data/eln.md
@@ -0,0 +1,37 @@
+This guide describes how to manually create entries and enter information
+via ELNs (electronic lab notebooks). NOMAD ELNs allow you to acquire
+consistently structured data from users to augment uploaded files.
+
+!!! attention
+
+ This part of the documentation is still work in progress.
+
+## Create a basic ELN entry
+
+Go to `PUBLISH` / `Uploads`. Here you can create an upload with the `CREATE A NEW UPLOAD`
+button. This will bring you to the upload page.
+
+Click the `CREATE ENTRY` button. This will bring-up a dialog to choose an ELN schema.
+All ELNs (as any entry in NOMAD) needs to follow a schema. You can choose from uploaded
+custom schemas or NOMAD build-in schemas. You can choose the `Basic ELN` to create a
+simple ELN entry.
+
+The name of your ELN entry, will be the filename for your ELN without the `.archive.json`
+ending that will be added automatically. You can always find and download your ELNs
+on the `FILES` tab.
+
+The `Basic ELN` offers you simple fields for a *name*, *tags*, a *date/time*, and a rich text
+editor to enter your notes.
+
+## Add your own ELN schema
+
+To make NOMAD ELNs more useful, you can define your own schema to create you own data
+fields, create more sub-sections, reference other entries, and much more.
+
+You should have a look at our ELN example upload. Go to `PUBLISH` / `Uploads` and
+click the `ADD EXAMPLE UPLOADS` button. The `Electronic Lab Notebook` example, will
+contain a schema and entries that instantiate different parts of the schema.
+The *ELN example sample (`sample.archive.json`) demonstrates what you can do.
+
+Follow the [How to write a schema](../schemas/basics.md) and [How to define ELN](../schemas/elns.md)
+guides to create you own customized of ELNs.
diff --git a/docs/data/explore.md b/docs/data/explore.md
new file mode 100644
index 0000000000000000000000000000000000000000..471b127806cba5a3dd9cd415e3af665eeefc6f09
--- /dev/null
+++ b/docs/data/explore.md
@@ -0,0 +1,17 @@
+!!! attention
+
+ This part of the documentation is still work in progress.
+
+While we are still working on this, please use our video tutorial as a starting point:
+
+<div class="youtube">
+<iframe src="https://www.youtube.com/embed/38S2U-TIvxE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+## Choose a search
+
+## Add search filters
+
+## Use the search bar
+
+## Create dashboards and apps
\ No newline at end of file
diff --git a/docs/data/north.md b/docs/data/north.md
new file mode 100644
index 0000000000000000000000000000000000000000..d0ff3d97ff0f06bb885954d14b2af9ac9f46206a
--- /dev/null
+++ b/docs/data/north.md
@@ -0,0 +1,3 @@
+!!! attention
+
+ This part of the documentation is still work in progress.
diff --git a/docs/web.md b/docs/data/upload.md
similarity index 73%
rename from docs/web.md
rename to docs/data/upload.md
index fc649955ac26ab959ab624865fc53b5d9c0d25ac..8de4e64359dd7583dc1039a106b5a1fec9a0c033 100644
--- a/docs/web.md
+++ b/docs/data/upload.md
@@ -1,24 +1,25 @@
-# Using the web interface
+This guide describes how to upload data in NOMAD supported file formats. You find a
+list of supported formats on top of each upload page, see below.
-## Uploading and publishing data
-
-This tutorial guides you through the basics of going from files on your computer
-to a published dataset with DOI.
-
-<div class="youtube">
-<iframe src="https://www.youtube.com/embed/3rVvfYoUbO0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
-</div>
-
-
-### Preparing files
+## Preparing files
You can upload files one by one, but you can also provider larger `.zip` or `.tar.gz`
archive files, if this is easier to you. Also the file upload via frp or command line with
curl or with wget generates an archive files. The specific layout of these files is up to you.
NOMAD will simply extract them and consider the whole directory structure within.
+## Create an upload and add files
+
+Open [NOMAD](https://nomad-lab.eu/prod/v1) and log in (you need to create an account first).
+
+Go to `PUBLISH` / `Uploads`. Here you can create an upload with the `CREATE A NEW UPLOAD`
+button. This will bring you to the upload page.
+
+You can drop your files on (or click) the `CLICK OR DROP FILES` button. On top you will
+see a list of supported file formats and details on the files to upload.
+You can also go to the `FILES` tab. Here you can create directories and drop files into directories.
-### Supported codes
+## Processing files
NOMAD interprets your files. It checks each file and recognizes the main output file of the
supported codes. NOMAD creates an entry for this *mainfile* that represents the respective
@@ -32,17 +33,19 @@ However, all files that are associated to a recognized *mainfile* by being in th
same directory are displayed as *auxiliary* files next to the entry represented
by the *mainfile*.
-**A note for VASP users**
-On the handling of **POTCAR** files: NOMAD takes care of it; you don't
-need to worry about it. We understand that POTCAR files are not supposed to be visible to
-the public according to your VASP license. Thus, in agreement with Georg Kresse, NOMAD extracts
-the most important information of POTCAR files and stores it in the files named
-`POTCAR.stripped`. These files can be accessed and downloaded by anyone, while the original
-POTCAR files are automatically removed.
+!!! note
+ **A note for VASP users**.
+ On the handling of **POTCAR** files: NOMAD takes care of it; you don't
+ need to worry about it. We understand that POTCAR files are not supposed to be visible to
+ the public according to your VASP license. Thus, in agreement with Georg Kresse, NOMAD extracts
+ the most important information of POTCAR files and stores it in the files named
+ `POTCAR.stripped`. These files can be accessed and downloaded by anyone, while the original
+ POTCAR files are automatically removed.
-### User metadata
+
+## Add user metadata
NOMAD automatically extracts as much information as possible from your files but you
can still specify additional metadata. This is what we call *user metadata*. This includes
@@ -75,15 +78,25 @@ User metadata can also be provided in an uploaded file. This can be a `.json` or
This file is only applied during the initial processing of an entry. So make sure you either
upload it first or with everything else as part of an archive file.
+## Publish and get a DOI
+
+After clicking the `PUBLISH` button, the uploaded files will become immutable, but you can still
+edit the metadata.
+
+As part of the *edit metadata* functionality, you can create and assign *datasets*.
+Go to `PUBLISH` / `Datasets` in the menu to see all your datasets. Here you can assign
+a DOI to created *datasets*. For a *dataset* with DOI, you can only add more entries, but
+not remove entries.
+
-### Limits
+## Upload limits
- One upload cannot exceed **32 GB** in size.
- Only **10 non published uploads** are allowed per user.
- Only uploads with at least one recognized entry can be published. See also [supported codes](#supported-codes) below.
-### Strategies for large amounts of data
+## Strategies for large amounts of data
Before attempting to upload large amounts of data, run some experiments with a representative
and small subset of your data. Use this to simulate a larger upload that you can review and edit
@@ -124,11 +137,3 @@ files directly from its servers via an additional `local_path` parameter:
```
curl -X PUT "http://nomad-lab.eu/prod/v1/api/uploads/?token=<your-token>&local_path=<path-to-upload-file>"
```
-
-## Exploring data on NOMAD
-
-This tutorial shows how to use NOMAD's search interface and structured data browsing to explore available data.
-
-<div class="youtube">
-<iframe src="https://www.youtube.com/embed/38S2U-TIvxE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
-</div>
diff --git a/docs/develop/parser.md b/docs/develop/parsers.md
similarity index 99%
rename from docs/develop/parser.md
rename to docs/develop/parsers.md
index 97a82c1fbfe71f68fea084c49e32a64770ff03fe..86b7caf1834e624f17e9e820db51026d6cb77163 100644
--- a/docs/develop/parser.md
+++ b/docs/develop/parsers.md
@@ -1,7 +1,4 @@
-# How to write a parser
-
-NOMAD uses parsers to convert raw code input and output files into NOMAD's common
-Archive format. This is the documentation on how to develop such a parser.
+NOMAD uses parsers to convert raw code input and output files into NOMAD's common Archive format. This is the documentation on how to develop such a parser.
## Getting started
diff --git a/docs/develop/setup.md b/docs/develop/setup.md
index 8c52f24268243579f71500fd85c6544b33f5fd94..d9ac86ca33aa522e8c321216582bffe6281f60e9 100644
--- a/docs/develop/setup.md
+++ b/docs/develop/setup.md
@@ -9,7 +9,7 @@ run the infrastructure in development mode, learn to run out test-suites, and se
*Visual Studio Code* for NOMAD development.
This is not about working with the NOMAD Python package. You can find the `nomad-lab`
-documentation [here](../pythonlib.md).
+documentation [here](../apis/pythonlib.md).
## Clone the sources
If not already done, you should clone nomad. If you have a gitlab@MPCDF account, you can clone with git URL:
diff --git a/docs/index.md b/docs/index.md
index 213094fda9c94ec8f3498209f56c3262ac2ad507..478341f3e51840aea7a83dd749b03cabac6c7c3b 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -1,228 +1,78 @@
-# Materials science data managed and shared
-
-The **NOvel Materials Discovery (NOMAD)** is a data management platform for materials
-science data. Here, NOMAD is a [web-application and database](https://nomad-lab.eu/prod/v1/gui/search)
-that allows to centrally publish data. But you can also use the NOMAD software to build your
-own local [NOMAD Oasis](oasis.md). See more basic information about NOMAD on the
-official [homepage](https://nomad-lab.eu).
-<!--
-
-{ align=right width=400 }
-*More than 12 million of simulations from over 400 authors world-wide*
-
-- Free publication and sharing of data
-- Manage research data though its whole life-cycle
-- Extracts <b>rich metadata</b> from data automatically
-- All data in <b>raw</b> and <b>machine processable</b> form
-- Use integrated tools to <b>explore</b>, <b>visualize</b>, and <b>analyze</b> -->
-
-## How does NOMAD work?
-
-### Managing data based on automatically extract rich metadata
-
-
-NOMAD is based on a *bottom-up* approach. Instead of only managing data of a specific
-predefined format, we use parsers and processing to support an extendable variety of
-data formats. Uploaded *raw* files are analysed and files with a recognized format are parsed.
-Parsers are small programs that transform data from the recognized *mainfiles* into a common machine
-processable version that we call *archive*. The information in the common archive representation
-drives everything else. It is the based for our search interface, the representation of materials
-and their properties, as well as all analytics.
-
-### A common hierarchical machine processable format for all data
-
-
-The *archive* is a hierarchical data format with a strict schema.
-All the information is organized into logical nested *sections*.
-Each *section* comprised a set of *quantities* on a common subject.
-All *sections* and *quantities* are supported by a formal schema that defines names, descriptions, types, shapes, and units.
-We sometimes call this data *archive* and the schema *metainfo*.
-
-### Datamodel: *uploads*, *entries*, *files*, *datasets*
-
-Uploaded *raw* files are managed in *uploads*.
-Users can create *uploads* and use them like projects.
-You can share them with other users, incrementally add and modify data in them, publish (incl. embargo) them, or transfer them between NOMAD installations.
-As long as an *upload* is not published, you can continue to provide files, delete the upload again, or test how NOMAD is processing your files.
-Once an upload is published, it becomes immutable.
-
-<figure markdown>
- { width=600 }
- <figcaption>NOMAD's main entities</figcaption>
-</figure>
-
-An *upload* can contain an arbitrary directory structure of *raw* files.
-For each recognized *mainfile*, NOMAD creates an entry.
-Therefore, an *upload* contains a list of *entries*.
-Each *entry* is associated with its *mainfile*, an *archive*, and all other *auxiliary* files in the same directory.
-*Entries* are automatically aggregated into *materials* based on the extract materials metadata.
-*Entries* (of many uploads) can be manually curated into *datasets*for which you can also get a DOI.
-
-### Using NOMAD software locally (the Oasis)
-
-The software that runs NOMAD is Open-Source and can be used independently of the NOMAD
-*central installation* at [http://nomad-lab.eu](http://nomad-lab.eu).
-We call any NOMAD installation that is not the *central* one a NOMAD Oasis.
+---
+hide: toc
+---
-<figure markdown>
- { width=700 }
- <figcaption>NOMAD Oasis use-cases</figcaption>
-</figure>
+# NOMAD Documentation
-There are several use-cases how the NOMAD software could be used. Of course other
-uses and hybrids are imaginable:
+<!-- A single sentence that says what the product is, succinctly and memorably -->
+NOMAD is a free open-source data management platform for materials science, based on FAIR (findable, accessible, interoperable and reusable) principles.
-- Academia: Use the Oasis for local management of unpublished research data
-- Mirror: Use the Oasis as a mirror that hosts a copy of all published NOMAD data
-- Industry: Use of Oasis to manage private data and full internal use of published data in compliance with strict privacy policies
-- FAIRmat: Use Oasis to form a network of repositories to build a federated data infrastructure
-for materials science.
-This is what we do in the [FAIRmat project](https://www.fair-di.eu/fairmat/consortium).
+<!-- A paragraph of one to three short sentences, that describe what the product does. -->
+NOMAD enables scientists and researchers to manage, share, and publish data.
+The platform lets you structure, explore, and analyze your data and the data of others.
+NOMAD promotes data-centered collaboration through an integrated and extendable set of tools.
-## Architecture
+<!-- A third paragraph of similar length, this time explaining what need the product meets -->
+NOMAD solves the problem of not being able to use data that you can't find or that is too
+heterogenous to be use effectively.
-### A containerized cloud enabled architecture
+<!-- Finally, a paragraph that describes whom the product is useful for. -->
+NOMAD is useful for scientists that work with data, for research groups that need to collaborate on data, and for communities that need to build an archive of FAIR research data.
-NOMAD is a modern web-application that requires a lot of services to run. Some are
-NOMAD specific, others are 3rd party products. While all services can be traditionally
-installed and run on a single sever, NOMAD advocates the use of containers and operating
-NOMAD in a cloud environment.
+<div markdown="block" class="home-grid">
+<div markdown="block">
-<figure markdown>
- 
- <figcaption>NOMAD architecture</figcaption>
-</figure>
+### Tutorial
-NOMAD comprises two main services, its *app* and the *worker*. The *app* services
-our API, graphical user interface, and documentation. It is the outward facing part of
-NOMAD. The worker runs all the processing (parsing, normalization). Their separation allows
-to scale the system for various use-cases.
+This series of [short videos will guide you through the main functionality of NOMAD](tutorial.md).
+It covers th whole publish, explore, analyze cycle:
-Other services are:
+- Upload and publish your own data
+- Use the search interface to identify interesting data
+- Use the API to search and access processed data for analysis
-- rabbitmq: a task queue that we use to distribute tasks for the *worker* containers
-- mongodb: a no-sql database used to maintain processing state and user-metadata
-- elasticsearch: a no-sql database and search engine that drives our search
-- a regular file system to maintain all the files (*raw* and *archive*)
-- jupyterhub: run ai toolkit notebooks
-- keycloak: our SSO user management system (can be used by all Oasises)
-- a content management system to provide other web-page content (not part of the Oasis)
+</div>
+<div markdown="block">
-All NOMAD software is bundled in a single NOMAD docker image and a Python package
-([nomad-lab on pypi](https://pypi.org/project/nomad-lab/)). The NOMAD docker
-image can be downloaded from our public registry.
-NOMAD software is organized in multiple git repositories. We use continuous integration
-to constantly provide the latest version of docker image and Python package.
+### How to guides
-### NOMAD uses a modern and rich stack frameworks, systems, and libraries
+The documentation provides step-by-step instructions for a wide range of tasks. For example:
-Besides various scientific computing, machine learning, and computational material
-science libraries (e.g. numpy, skikitlearn, tensorflow, ase, spglib, matid, and many more),
-Nomad uses a set of freely available technologies that already solve most
-of its processing, storage, availability, and scaling goals. The following is a non
-comprehensive overview of used languages, libraries, frameworks, and services.
+- [How to upload and publish data](data/upload.md)
+- [How to write a custom ELN](schemas/elns.md)
+- [How to run a parser locally](apis/local_parsers.md)
+- [How to install NOMAD Oasis](oasis/install.md)
-<figure markdown>
- 
- <figcaption>NOMAD components and dependencies</figcaption>
-</figure>
+</div>
-#### Python 3
+<div markdown="block">
-The *backend* of nomad is written in Python. This includes all parsers, normalizers,
-and other data processing. We only use Python 3 and there is no compatibility with
-Python 2. Code is formatted close to [pep8](https://www.python.org/dev/peps/pep-0008/),
-critical parts use [pep484](https://www.python.org/dev/peps/pep-0484/) type-hints.
-[Pycodestyle](https://pypi.org/project/pycodestyle/),
-[pylint](https://www.pylint.org/), and
-[mypy](http://mypy-lang.org/) (static type checker) are used to ensure quality.
-Tests are written with [pytest](https://docs.pytest.org/en/latest/contents.html).
-Logging is done with [structlog](https://www.structlog.org/en/stable/) and *logstash* (see
-Elasticstack below). Documentation is driven by [Sphinx](http://www.sphinx-doc.org/en/master/).
+### Learn
+This section provides background knowledge on what are
+schemas and structured data, how does processing work, the NOMAD architecture, and more.
-#### celery
+</div>
+<div markdown="block">
-[Celery](http://celeryproject.org) (+ [rabbitmq](https://www.rabbitmq.com/))
-is a popular combination for realizing long running tasks in internet applications.
-We use it to drive the processing of uploaded files.
-It allows us to transparently distribute processing load while keeping processing state
-available to inform the user.
+### Reference
+The reference includes all CLI commands and arguments, all configuration options,
+the possible schema annotations and their arguments, and a glossary of used terms.
-#### elastic search
+</div>
+</div>
-[Elasticsearch](https://www.elastic.co/webinars/getting-started-elasticsearch)
-is used to store repository data (not the raw files).
-Elasticsearch enables flexible, scalable search and analytics.
+<h2>Project and community</h2>
+NOMAD is an open source project that warmly welcomes community projects, contributions, suggestions, fixes and constructive feedback.
+NOMAD is developed by FAIRmat, an open NFDI consortium of over 30 partners building a shared
+data structure of for materials science together.
-#### mongodb
+- [Get support](https://nomad-lab.eu/nomad-lab/help.html)
+- [Join our online forum](https://matsci.org/c/nomad/32)
+- [Contribute](https://nomad-lab.eu/nomad-lab/source-code.html)
+- [Roadmap](https://nomad-lab.eu/nomad-lab/features.html)
+- [Code guidelines](develop/guides.md)
-[Mongodb](https://docs.mongodb.com/) is used to store and track the state of the
-processing of uploaded files and the generated entries. We use
-[mongoengine](http://docs.mongoengine.org/) to program with mongodb.
-
-
-#### Keycloak
-
-[Keycloak](https://www.keycloak.org/) is used for user management. It manages users and
-provides functions for registration, forgetting passwords, editing user accounts, and single
-sign-on to fairdi@nomad and other related services.
-
-
-#### FastAPI
-
-The ReSTful API is build with the [FastAPI](https://fastapi.tiangolo.com/)
-framework. This allows us to automatically derive a [OpenAPI](https://swagger.io/specification/) description
-of the nomad API.
-Fruthermore, you can browse and use the API via [OpenAPI dashboard](https://swagger.io/tools/swagger-ui/).
-
-
-#### Elasticstack
-
-The [elastic stack](https://www.elastic.co/guide/index.html)
-(previously *ELK* stack) is a centralized logging, metrics, and monitoring
-solution that collects data within the cluster and provides a flexible analytics front end
-for that data.
-
-
-#### Javascript, React, Material-UI
-
-The frontend (GUI) of **nomad@FAIRDI** is built on the
-[React](https://reactjs.org/docs/getting-started.html) component framework.
-This allows us to build the GUI as a set of re-usable components to
-achieve a coherent representations for all aspects of nomad, while keeping development
-efforts manageable. React uses [JSX](https://reactjs.org/docs/introducing-jsx.html)
-(a ES6 variety) that allows to mix HTML with Javascript code.
-The component library [Material-UI](https://material-ui.com/)
-(based on Google's popular material design framework) provides a consistent look-and-feel.
-
-
-#### docker
-
-To run a **nomad@FAIRDI** instance, many services have to be orchestrated:
-the nomad app, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
-Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together.
-Further services might be needed (e.g. JypiterHUB), when nomad grows.
-The container platform [Docker](https://docs.docker.com/) allows us to provide all services
-as pre-build images that can be run flexibly on all types of platforms, networks,
-and storage solutions. [Docker-compose](https://docs.docker.com/compose/) allows us to
-provide configuration to run the whole nomad stack on a single server node.
-
-
-#### kubernetes + helm
-
-To run and scale nomad on a cluster, you can use [kubernetes](https://kubernetes.io/docs/home/)
-to orchestrated the necessary containers. We provide a [helm](https://docs.helm.sh/)
-chart with all necessary service and deployment descriptors that allow you to set up and
-update nomad with only a few commands.
-
-
-#### GitLab
-
-Nomad as a software project is managed via [GitLab](https://docs.gitlab.com/).
-The **nomad@FAIRDI** project is hosted [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR).
-GitLab is used to manage versions, different branches of development, tasks and issues,
-as a [registry for Docker images](https://docs.gitlab.com/ee/user/packages/container_registry/index.html),
-and [CI/CD platform](https://docs.gitlab.com/ee/ci/).
+Thinking about using NOMAD for your next project? Get in touch!
diff --git a/docs/learn/architecture.md b/docs/learn/architecture.md
new file mode 100644
index 0000000000000000000000000000000000000000..fa76aed08cb349e2947489aabe11028b96fcf57d
--- /dev/null
+++ b/docs/learn/architecture.md
@@ -0,0 +1,147 @@
+# Architecture
+
+## A containerized cloud enabled architecture
+
+NOMAD is a modern web-application that requires a lot of services to run. Some are
+NOMAD specific, others are 3rd party products. While all services can be traditionally
+installed and run on a single sever, NOMAD advocates the use of containers and operating
+NOMAD in a cloud environment.
+
+<figure markdown>
+ 
+ <figcaption>NOMAD architecture</figcaption>
+</figure>
+
+NOMAD comprises two main services, its *app* and the *worker*. The *app* services
+our API, graphical user interface, and documentation. It is the outward facing part of
+NOMAD. The worker runs all the processing (parsing, normalization). Their separation allows
+to scale the system for various use-cases.
+
+Other services are:
+
+- rabbitmq: a task queue that we use to distribute tasks for the *worker* containers
+- mongodb: a no-sql database used to maintain processing state and user-metadata
+- elasticsearch: a no-sql database and search engine that drives our search
+- a regular file system to maintain all the files (*raw* and *archive*)
+- jupyterhub: run ai toolkit notebooks
+- keycloak: our SSO user management system (can be used by all Oasises)
+- a content management system to provide other web-page content (not part of the Oasis)
+
+All NOMAD software is bundled in a single NOMAD docker image and a Python package
+([nomad-lab on pypi](https://pypi.org/project/nomad-lab/)). The NOMAD docker
+image can be downloaded from our public registry.
+NOMAD software is organized in multiple git repositories. We use continuous integration
+to constantly provide the latest version of docker image and Python package.
+
+## NOMAD uses a modern and rich stack frameworks, systems, and libraries
+
+Besides various scientific computing, machine learning, and computational material
+science libraries (e.g. numpy, skikitlearn, tensorflow, ase, spglib, matid, and many more),
+Nomad uses a set of freely available technologies that already solve most
+of its processing, storage, availability, and scaling goals. The following is a non
+comprehensive overview of used languages, libraries, frameworks, and services.
+
+<figure markdown>
+ 
+ <figcaption>NOMAD components and dependencies</figcaption>
+</figure>
+
+#### Python 3
+
+The *backend* of nomad is written in Python. This includes all parsers, normalizers,
+and other data processing. We only use Python 3 and there is no compatibility with
+Python 2. Code is formatted close to [pep8](https://www.python.org/dev/peps/pep-0008/),
+critical parts use [pep484](https://www.python.org/dev/peps/pep-0484/) type-hints.
+[Pycodestyle](https://pypi.org/project/pycodestyle/),
+[pylint](https://www.pylint.org/), and
+[mypy](http://mypy-lang.org/) (static type checker) are used to ensure quality.
+Tests are written with [pytest](https://docs.pytest.org/en/latest/contents.html).
+Logging is done with [structlog](https://www.structlog.org/en/stable/) and *logstash* (see
+Elasticstack below). Documentation is driven by [Sphinx](http://www.sphinx-doc.org/en/master/).
+
+
+#### celery
+
+[Celery](http://celeryproject.org) (+ [rabbitmq](https://www.rabbitmq.com/))
+is a popular combination for realizing long running tasks in internet applications.
+We use it to drive the processing of uploaded files.
+It allows us to transparently distribute processing load while keeping processing state
+available to inform the user.
+
+
+#### elastic search
+
+[Elasticsearch](https://www.elastic.co/webinars/getting-started-elasticsearch)
+is used to store repository data (not the raw files).
+Elasticsearch enables flexible, scalable search and analytics.
+
+
+#### mongodb
+
+[Mongodb](https://docs.mongodb.com/) is used to store and track the state of the
+processing of uploaded files and the generated entries. We use
+[mongoengine](http://docs.mongoengine.org/) to program with mongodb.
+
+
+#### Keycloak
+
+[Keycloak](https://www.keycloak.org/) is used for user management. It manages users and
+provides functions for registration, forgetting passwords, editing user accounts, and single
+sign-on to fairdi@nomad and other related services.
+
+
+#### FastAPI
+
+The ReSTful API is build with the [FastAPI](https://fastapi.tiangolo.com/)
+framework. This allows us to automatically derive a [OpenAPI](https://swagger.io/specification/) description
+of the nomad API.
+Fruthermore, you can browse and use the API via [OpenAPI dashboard](https://swagger.io/tools/swagger-ui/).
+
+
+#### Elasticstack
+
+The [elastic stack](https://www.elastic.co/guide/index.html)
+(previously *ELK* stack) is a centralized logging, metrics, and monitoring
+solution that collects data within the cluster and provides a flexible analytics front end
+for that data.
+
+
+#### Javascript, React, Material-UI
+
+The frontend (GUI) of **nomad@FAIRDI** is built on the
+[React](https://reactjs.org/docs/getting-started.html) component framework.
+This allows us to build the GUI as a set of re-usable components to
+achieve a coherent representations for all aspects of nomad, while keeping development
+efforts manageable. React uses [JSX](https://reactjs.org/docs/introducing-jsx.html)
+(a ES6 variety) that allows to mix HTML with Javascript code.
+The component library [Material-UI](https://material-ui.com/)
+(based on Google's popular material design framework) provides a consistent look-and-feel.
+
+
+#### docker
+
+To run a **nomad@FAIRDI** instance, many services have to be orchestrated:
+the nomad app, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
+Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together.
+Further services might be needed (e.g. JypiterHUB), when nomad grows.
+The container platform [Docker](https://docs.docker.com/) allows us to provide all services
+as pre-build images that can be run flexibly on all types of platforms, networks,
+and storage solutions. [Docker-compose](https://docs.docker.com/compose/) allows us to
+provide configuration to run the whole nomad stack on a single server node.
+
+
+#### kubernetes + helm
+
+To run and scale nomad on a cluster, you can use [kubernetes](https://kubernetes.io/docs/home/)
+to orchestrated the necessary containers. We provide a [helm](https://docs.helm.sh/)
+chart with all necessary service and deployment descriptors that allow you to set up and
+update nomad with only a few commands.
+
+
+#### GitLab
+
+Nomad as a software project is managed via [GitLab](https://docs.gitlab.com/).
+The **nomad@FAIRDI** project is hosted [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR).
+GitLab is used to manage versions, different branches of development, tasks and issues,
+as a [registry for Docker images](https://docs.gitlab.com/ee/user/packages/container_registry/index.html),
+and [CI/CD platform](https://docs.gitlab.com/ee/ci/).
diff --git a/docs/assets/architecture.png b/docs/learn/architecture.png
similarity index 100%
rename from docs/assets/architecture.png
rename to docs/learn/architecture.png
diff --git a/docs/assets/archive-example.png b/docs/learn/archive-example.png
similarity index 100%
rename from docs/assets/archive-example.png
rename to docs/learn/archive-example.png
diff --git a/docs/assets/data-flow.png b/docs/learn/data-flow.png
similarity index 100%
rename from docs/assets/data-flow.png
rename to docs/learn/data-flow.png
diff --git a/docs/assets/datamodel.png b/docs/learn/datamodel.png
similarity index 100%
rename from docs/assets/datamodel.png
rename to docs/learn/datamodel.png
diff --git a/docs/assets/how-does-nomad-work.png b/docs/learn/how-does-nomad-work.png
similarity index 100%
rename from docs/assets/how-does-nomad-work.png
rename to docs/learn/how-does-nomad-work.png
diff --git a/docs/learn/how_nomad_works.md b/docs/learn/how_nomad_works.md
new file mode 100644
index 0000000000000000000000000000000000000000..265509f06b1a0ab2ba8743b500a2d26dc2792840
--- /dev/null
+++ b/docs/learn/how_nomad_works.md
@@ -0,0 +1,41 @@
+# How does NOMAD work?
+
+## Managing data based on automatically extract rich metadata
+
+
+NOMAD is based on a *bottom-up* approach. Instead of only managing data of a specific
+predefined format, we use parsers and processing to support an extendable variety of
+data formats. Uploaded *raw* files are analysed and files with a recognized format are parsed.
+Parsers are small programs that transform data from the recognized *mainfiles* into a common machine
+processable version that we call *archive*. The information in the common archive representation
+drives everything else. It is the based for our search interface, the representation of materials
+and their properties, as well as all analytics.
+
+## A common hierarchical machine processable format for all data
+
+
+The *archive* is a hierarchical data format with a strict schema.
+All the information is organized into logical nested *sections*.
+Each *section* comprised a set of *quantities* on a common subject.
+All *sections* and *quantities* are supported by a formal schema that defines names, descriptions, types, shapes, and units.
+We sometimes call this data *archive* and the schema *metainfo*.
+
+## Datamodel: *uploads*, *entries*, *files*, *datasets*
+
+Uploaded *raw* files are managed in *uploads*.
+Users can create *uploads* and use them like projects.
+You can share them with other users, incrementally add and modify data in them, publish (incl. embargo) them, or transfer them between NOMAD installations.
+As long as an *upload* is not published, you can continue to provide files, delete the upload again, or test how NOMAD is processing your files.
+Once an upload is published, it becomes immutable.
+
+<figure markdown>
+ { width=600 }
+ <figcaption>NOMAD's main entities</figcaption>
+</figure>
+
+An *upload* can contain an arbitrary directory structure of *raw* files.
+For each recognized *mainfile*, NOMAD creates an entry.
+Therefore, an *upload* contains a list of *entries*.
+Each *entry* is associated with its *mainfile*, an *archive*, and all other *auxiliary* files in the same directory.
+*Entries* are automatically aggregated into *materials* based on the extract materials metadata.
+*Entries* (of many uploads) can be manually curated into *datasets*for which you can also get a DOI.
\ No newline at end of file
diff --git a/docs/assets/oasis-use-cases.png b/docs/learn/oasis-use-cases.png
similarity index 100%
rename from docs/assets/oasis-use-cases.png
rename to docs/learn/oasis-use-cases.png
diff --git a/docs/learn/oasis.md b/docs/learn/oasis.md
new file mode 100644
index 0000000000000000000000000000000000000000..8a7ae97ca840806e7e574b025a12746e9e8b47a5
--- /dev/null
+++ b/docs/learn/oasis.md
@@ -0,0 +1,19 @@
+The software that runs NOMAD is Open-Source and can be used independently of the NOMAD
+*central installation* at [http://nomad-lab.eu](http://nomad-lab.eu).
+We call any NOMAD installation that is not the *central* one a NOMAD Oasis.
+
+<figure markdown>
+ { width=700 }
+ <figcaption>NOMAD Oasis use-cases</figcaption>
+</figure>
+
+There are several use-cases how the NOMAD software could be used. Of course other
+uses and hybrids are imaginable:
+
+- Academia: Use the Oasis for local management of unpublished research data
+- Mirror: Use the Oasis as a mirror that hosts a copy of all published NOMAD data
+- Industry: Use of Oasis to manage private data and full internal use of published data in compliance with strict privacy policies
+- FAIRmat: Use Oasis to form a network of repositories to build a federated data infrastructure
+for materials science.
+This is what we do in the [FAIRmat project](https://www.fair-di.eu/fairmat/consortium).
+
diff --git a/docs/schema/introduction.md b/docs/learn/schemas.md
similarity index 94%
rename from docs/schema/introduction.md
rename to docs/learn/schemas.md
index 70df1b6fda6d3068a9a242431d4cfe70dfd9ddc0..3e335e7df6b5415ddc770625eed9a702f17f2e15 100644
--- a/docs/schema/introduction.md
+++ b/docs/learn/schemas.md
@@ -1,8 +1,4 @@
----
-title: Introduction
----
-
-# An Introduction to Schemas and Structured Data in NOMAD
+# Schemas and Structured Data in NOMAD
NOMAD stores all processed data in a *well defined*, *structured*, and *machine readable*
format. Well defined means that each element is supported by a formal definition that provides
@@ -11,7 +7,7 @@ hierarchical structure that logically organizes data in sections and subsections
cross-references between pieces of data. Formal definitions and corresponding
data structures enable the machine processing of NOMAD data.
-
+
## The Metainfo is the schema for Archive data.
The Archive stores descriptive and structured information about materials-science
@@ -54,7 +50,7 @@ Metainfo schema can be programmed with Python classes, but can also be uploaded
archive files (the Metainfo itself is just a specific Archive schema). The following
chart provides a sense of various ways that data can be entered into NOMAD:
-
+
There are various interface to provide or retrieve Archive data and Metainfo schemas.
The following documentation sections will explain a few of them.
diff --git a/docs/assets/stack.png b/docs/learn/stack.png
similarity index 100%
rename from docs/assets/stack.png
rename to docs/learn/stack.png
diff --git a/docs/oasis/admin.md b/docs/oasis/admin.md
new file mode 100644
index 0000000000000000000000000000000000000000..3dcd1cf3bc6b98045d39d8b376afdcec962961be
--- /dev/null
+++ b/docs/oasis/admin.md
@@ -0,0 +1,183 @@
+## Backups
+
+To backup your Oasis at least the file data and mongodb data needs to be saved. You determined the path to your file data (your uploads) during the installation. By
+default all data is stored in a directory called `.volumes` that is created in the
+current working directory of your installation/docker-compose. This directory can be backed up like any other file backup (e.g. rsync).
+
+To backup the mongodb, please refer to the official [mongodb documentation](https://docs.mongodb.com/manual/core/backups/). We suggest a simple mongodump export that is backed up alongside your files. The default configuration mounts `.volumes/mongo` into the mongodb container (as `/backup`) for this purpose. You can use this to export the NOMAD mongo database. Combine this with rsync on the `.volumes` directory and everything should be set. To create a new mongodump run:
+
+```sh
+docker exec nomad_oasis_mongo mongodump -d nomad_oasis_v1 -o /backup
+```
+
+The elasticsearch contents can be reproduced with the information in the files and the mongodb.
+
+To create a new oasis with the backup data, create the `.volumes` directory from your backup. Start the new oasis. Use mongorestore:
+
+```sh
+docker exec nomad_oasis_mongo mongorestore /backup
+```
+
+Now you still have to recreate the elasticsearch index:
+```sh
+docker exec nomad_oasis_app python -m nomad.cli admin uploads index
+```
+
+## Managing data with the CLI
+
+The NOMAD command line interface (CLI) provides a few useful administrative functions. To use the NOMAD CLI, open a shell into the app container and run it from there:
+
+```sh
+docker exec -ti nomad_oasis_app bash
+```
+
+For example you can ls or remove uploads:
+```sh
+nomad admin uploads ls
+nomad admin uploads rm -- <upload_id> <upload_id>
+```
+
+You can also reset the processing (of "stuck") uploads and reprocess:
+```sh
+nomad admin uploads reset -- <upload_id>
+nomad admin uploads process -- <upload_id>
+```
+
+You can also use the CLI to wipe the whole installation:
+```sh
+nomad admin reset --i-am-really-sure
+```
+
+## Upload commands
+
+The `nomad admin uploads` group of CLI commands allow you to inspect and modify
+all or some uploads in your installation. Sub-commands include `ls`, `rm`,
+`chown`, `process` (see below), `index` (see below).
+
+The command group takes many different parameters to target specific subsets of uploads.
+Here are a few examples:
+
+- `--unpublished`
+- `--published`
+- `--outdated` Select published uploads with older NOMAD versions than the current
+- `--processing-failure` Uploads with processing failures.
+
+For a complete list refer to the [CLI reference documentation](../reference/cli.md#nomad-admin-uploads).
+
+Alternatively, you can use a list of upload ids at the end of the command, e.g.:
+
+```
+nomad admin uploads ls -- <id1> <id2>
+```
+
+If you have a list of ids (e.g. in a file), you could use `xargs`:
+
+```
+cat file_with_ids.txt | xargs nomad admin uploads ls --
+```
+
+## Re-processing
+
+Processing includes the conversion of raw files into NOMAD entries. Files are parsed,
+normalizers are called, the processing results are stored, and the search index
+is updated. In certain scenarios (failed processing, [migration](migrate.md#migration-steps),
+changed plugins) might require that admins process certain uploads again.
+
+```
+nomad admin uploads process
+```
+
+## Re-Indexing
+
+Each NOMAD entry is represented in NOMAD's search index. Only if an entry is in this
+index, you can find it via the search interface. Some changes between NOMAD versions
+(see also our [migration guide](migrate.md#migration-steps)), might require that
+you re-index all uploads.
+
+```
+nomad admin uploads index
+```
+
+## Restricting access to your Oasis
+
+An Oasis works exactly the same way the official NOMAD works. It is open and everybody
+can access published data. Everybody with an account can upload data. This might not be
+what you want.
+
+Currently there are two ways to restrict access to your Oasis. First, you do not
+expose the Oasis to the public internet, e.g. you only make it available on an intra-net or
+through a VPN.
+
+Second, we offer a simple white-list mechanism. As the Oasis administrator you provide a
+list of accounts as part of your Oasis configuration. To use the Oasis, all users have to
+be logged in and be on your white list of allowed users. To enable white-listing, you
+can provide a list of NOMAD account email addresses in your `nomad.yaml` like this:
+
+```
+oasis:
+ allowed_users:
+ - user1@gmail.com
+ - user2@gmail.com
+```
+
+## Configuring for performance
+
+If you run the OASIS on a single computer, like described here (either with docker or bare
+linux), you might run into problems with processing large uploads. If the NOMAD worker
+and app are run on the same computer, the app might become unresponsive, when the worker
+consumes all system resources.
+
+By default, the worker container might have as many worker processes as the system as CPU cores.
+In addition, each worker process might spawn additional threads and consume
+more than one CPU core.
+
+There are multiple ways to restrict the worker's resource consumption:
+
+- limit the number of worker processes and thereby lower the number of used cores
+- disable or restrict multithreading
+- limit available CPU utilization of the worker's docker container with docker
+
+### Limit the number of worker processes
+
+The worker uses the Python package celery. Celery can be configured to use less than the
+default number of worker processes (which equals the number of available cores). To use only
+a single core only, you can alter the worker service command in the `docker-compose.yml` and
+add a `--concurrency` argument:
+
+```
+command: python -m celery -A nomad.processing worker -l info --concurrency=1 -Q celery
+```
+
+See also the [celery documentation](https://docs.celeryproject.org/en/stable/userguide/workers.html#id1).
+
+### Limiting the use of threads
+
+You can also reduce the usable threads that Python packages based on OpenMP could use to
+reduce the threads that might be spawned by a single worker process. Simply set the `OMP_NUM_THREADS`
+environment variable in the worker container in your `docker-compose.yml`:
+
+```
+services:
+ worker:
+ ...
+ environment:
+ ...
+ OMP_NUM_THREADS: 1
+```
+
+### Limit CPU with docker
+
+You can add a `deploy.resources.limits` section to the worker service in the `docker-compose.yml`:
+
+```
+services:
+ worker:
+ ...
+ deploy:
+ resources:
+ limits:
+ cpus: '0.50'
+```
+
+The number refers to the percentage use of a single CPU core.
+See also the [docker-compose documentation](https://docs.docker.com/compose/compose-file/compose-file-v3/#resources).
diff --git a/docs/oasis/customize.md b/docs/oasis/customize.md
new file mode 100644
index 0000000000000000000000000000000000000000..30609f00655a8b0be2fc3f293e39ada799d1393b
--- /dev/null
+++ b/docs/oasis/customize.md
@@ -0,0 +1,14 @@
+!!! attention
+
+ This part of the documentation is still work in progress.
+
+This is an incomplete list of potential customizations. Please read the respective
+guides to learn more.
+
+- Installation specific changes (domain, path-prefix): [How to install an Oasis](install.md)
+- [Restricting user access](admin.md#restricting-access-to-your-oasis)
+- Write .yaml based [schemas](../schemas/basics.md) and [ELNs](../schemas/elns.md)
+- Learn how to use the [tabular parser](../schemas/tabular.md) to manage data from .xls or .csv
+- Develop a [schema plugin](../plugins/schemas.md) for more powerful schemas and ELNs
+- Use and develop [parser plugins](../plugins/parsers.md) for specific file formats
+- Add specialized [NORTH tools](../data/north.md)
\ No newline at end of file
diff --git a/docs/oasis.md b/docs/oasis/install.md
similarity index 64%
rename from docs/oasis.md
rename to docs/oasis/install.md
index 1f075e07c90e64361bc542170e5f357f596bda11..badf2d415d9adcfe6087341cbe5f6ef3490ab7be 100644
--- a/docs/oasis.md
+++ b/docs/oasis/install.md
@@ -12,7 +12,7 @@ central NOMAD installation.
- Make sure you have [docker](https://docs.docker.com/engine/install/) installed.
Docker nowadays comes with `docker compose` build in. Prior, you needed to
install the stand alone [docker-compose](https://docs.docker.com/compose/install/).
-- Download our basic configuration files [nomad-oasis.zip](assets/nomad-oasis.zip)
+- Download our basic configuration files [nomad-oasis.zip](../assets/nomad-oasis.zip)
- Run the following commands (skip `chown` on MacOS and Windows computers)
```sh
@@ -106,7 +106,7 @@ compatible and you might chose to can replace `docker compose` with `docker-comp
The following will run all necessary services with docker. These comprise: a **mongo**
database, an **elasticsearch**, a **rabbitmq** distributed task queue, the NOMAD **app**,
-NOMAD **worker**, and NOMAD **gui**. In this [introduction](index.md#architecture),
+NOMAD **worker**, and NOMAD **gui**. In this [introduction](../index.md#architecture),
you will learn what each service does and why it is necessary.
### Configuration
@@ -121,7 +121,7 @@ There are three files to configure:
- `configs/nginx.conf`
In this example, we have all files in the same directory (the directory we are also working in).
-You can download minimal example files [here](assets/nomad-oasis.zip).
+You can download minimal example files [here](../assets/nomad-oasis.zip).
#### docker-compose.yaml
@@ -350,56 +350,6 @@ your own host.
- We disabled the https requirement on the default realm for simplicity. You should change
this for a production system.
-### Backups
-
-To backup your Oasis at least the file data and mongodb data needs to be saved. You determined the path to your file data (your uploads) during the installation. By
-default all data is stored in a directory called `.volumes` that is created in the
-current working directory of your installation/docker-compose. This directory can be backed up like any other file backup (e.g. rsync).
-
-To backup the mongodb, please refer to the official [mongodb documentation](https://docs.mongodb.com/manual/core/backups/). We suggest a simple mongodump export that is backed up alongside your files. The default configuration mounts `.volumes/mongo` into the mongodb container (as `/backup`) for this purpose. You can use this to export the NOMAD mongo database. Combine this with rsync on the `.volumes` directory and everything should be set. To create a new mongodump run:
-
-```sh
-docker exec nomad_oasis_mongo mongodump -d nomad_oasis_v1 -o /backup
-```
-
-The elasticsearch contents can be reproduced with the information in the files and the mongodb.
-
-To create a new oasis with the backup data, create the `.volumes` directory from your backup. Start the new oasis. Use mongorestore:
-
-```sh
-docker exec nomad_oasis_mongo mongorestore /backup
-```
-
-Now you still have to recreate the elasticsearch index:
-```sh
-docker exec nomad_oasis_app python -m nomad.cli admin uploads index
-```
-
-### A few useful NOMAD commands
-
-The NOMAD command line interface (CLI) provides a few useful administrative functions. To use the NOMAD CLI, open a shell into the app container and run it from there:
-
-```sh
-docker exec -ti nomad_oasis_app bash
-```
-
-For example you can ls or remove uploads:
-```sh
-nomad admin uploads ls
-nomad admin uploads rm -- <upload_id> <upload_id>
-```
-
-You can also reset the processing (of "stuck") uploads and reprocess:
-```sh
-nomad admin uploads reset -- <upload_id>
-nomad admin uploads process -- <upload_id>
-```
-
-You can also use the CLI to wipe the whole installation:
-```sh
-nomad admin reset --i-am-really-sure
-```
-
## Base Linux (without docker)
### Pre-requisites
@@ -514,190 +464,3 @@ This should give you a working OASIS at `http://<your-host>/<your-path-prefix>`.
*This is not yet documented.*
-## Migrating from an older version (0.8.x to 1.x)
-
-Between versions 0.10.x and 1.x we needed to change how archive and metadata data is stored
-internally in files and databases. This means you cannot simply start a new version of
-NOMAD on top of the old data. But there is a strategy to adapt the data. This should
-work for data based on NOMAD >0.8.0 and <= 0.10.x.
-
-The overall strategy is to create a new mongo database, copy all information, and then
-reprocess all data for the new version.
-
-First, shutdown the OASIS and remove all old containers.
-```sh
-docker compose stop
-docker compose rm -f
-```
-
-Update your config files (`docker-compose.yaml`, `nomad.yaml`, `nginx.conf`) according
-to the latest documentation (see above). Make sure to use index and database names that
-are different. The default values contain a version number in those names, if you don't
-overwrite those defaults, you should be safe.
-
-Make sure you get the latest images and start the OASIS with the new version of NOMAD:
-```sh
-docker compose pull
-docker compose up -d
-```
-
-If you go to the GUI of your OASIS, it should now show the new version and appear empty,
-because we are using a different database and search index now.
-
-To migrate the data, we created a command that you can run within your OASIS' NOMAD
-application container. This command takes the old database name as an argument, it will copy
-all data from the old mongodb to the current one. The default v8.x database name
-was `nomad_fairdi`, but you might have changed this to `nomad_v0_8` as recommended by
-our old Oasis documentation.
-
-```sh
-docker exec -ti nomad_oasis_app bash -c 'nomad admin upgrade migrate-mongo --src-db-name nomad_v0_8'
-docker exec -ti nomad_oasis_app bash -c 'nomad admin uploads reprocess'
-```
-
-Now all your data should appear in your OASIS again. If you like, you can remove the
-old index and database:
-
-```sh
-docker exec nomad_oasis_elastic bash -c 'curl -X DELETE http://elastic:9200/nomad_fairdi'
-docker exec nomad_oasis_mongo bash -c 'mongo nomad_fairdi --eval "printjson(db.dropDatabase())"'
-```
-
-## Restricting access to your Oasis
-
-An Oasis works exactly the same way the official NOMAD works. It is open and everybody
-can access published data. Everybody with an account can upload data. This might not be
-what you want.
-
-Currently there are two ways to restrict access to your Oasis. First, you do not
-expose the Oasis to the public internet, e.g. you only make it available on an intra-net or
-through a VPN.
-
-Second, we offer a simple white-list mechanism. As the Oasis administrator you provide a
-list of accounts as part of your Oasis configuration. To use the Oasis, all users have to
-be logged in and be on your white list of allowed users. To enable white-listing, you
-can provide a list of NOMAD account email addresses in your `nomad.yaml` like this:
-
-```
-oasis:
- allowed_users:
- - user1@gmail.com
- - user2@gmail.com
-```
-
-## Performance considerations
-
-If you run the OASIS on a single computer, like described here (either with docker or bare
-linux), you might run into problems with processing large uploads. If the NOMAD worker
-and app are run on the same computer, the app might become unresponsive, when the worker
-consumes all system resources.
-
-By default, the worker container might have as many worker processes as the system as CPU cores.
-In addition, each worker process might spawn additional threads and consume
-more than one CPU core.
-
-There are multiple ways to restrict the worker's resource consumption:
-
-- limit the number of worker processes and thereby lower the number of used cores
-- disable or restrict multithreading
-- limit available CPU utilization of the worker's docker container with docker
-
-#### Limit the number of worker processes
-
-The worker uses the Python package celery. Celery can be configured to use less than the
-default number of worker processes (which equals the number of available cores). To use only
-a single core only, you can alter the worker service command in the `docker-compose.yml` and
-add a `--concurrency` argument:
-
-```
-command: python -m celery -A nomad.processing worker -l info --concurrency=1 -Q celery
-```
-
-See also the [celery documentation](https://docs.celeryproject.org/en/stable/userguide/workers.html#id1).
-
-#### Limiting the use of threads
-
-You can also reduce the usable threads that Python packages based on OpenMP could use to
-reduce the threads that might be spawned by a single worker process. Simply set the `OMP_NUM_THREADS`
-environment variable in the worker container in your `docker-compose.yml`:
-
-```
-services:
- worker:
- ...
- environment:
- ...
- OMP_NUM_THREADS: 1
-```
-
-#### Limit CPU with docker
-
-You can add a `deploy.resources.limits` section to the worker service in the `docker-compose.yml`:
-
-```
-services:
- worker:
- ...
- deploy:
- resources:
- limits:
- cpus: '0.50'
-```
-
-The number refers to the percentage use of a single CPU core.
-See also the [docker-compose documentation](https://docs.docker.com/compose/compose-file/compose-file-v3/#resources).
-
-
-## NOMAD Oasis FAQ
-
-#### Why use an Oasis?
-There are three reasons:
-
-- You want to manage data in private,
-- you want local availability of public NOMAD data without network restrictions,
-- or you want to manage large amounts of low quality and preliminary data that is not intended for publication.
-
-#### How to organize data in NOMAD Oasis?
-
-##### How can I categorize or label my data?
-Currently, NOMAD supports the following mechanism to organize data:
-
-- Data always belongs to one upload, which has a main author, and is assigned various metadata, like the time the upload was created, a custom human friendly upload name, etc.
-- Data can be assigned to multiple independent datasets.
-- Data can hold a proprietary id called “external_idâ€.
-- Data can be assigned to multiple coauthors in addition to the main author.
-- Data can be filtered and searched in various ways.
-
-##### Is there some rights-based visibility?
-An upload can be viewed at any time by the main author, the upload coauthors, and the
-upload reviewers. Other users will not be able to see the upload until it is published and
-the embargo (if requested) has been lifted. The main author decides which users to register as coauthors and reviewers, when to publish and if it should be published with an embargo.
-The main author and upload coauthors are the only users who can edit the upload while it is in staging.
-
-#### How to share data with the central NOMAD?
-
-Keep in mind, it is not entirely clear, how we will do this.
-
-##### How to designate Oasis data for publishing to NOMAD?
-Currently, you should use one of the organizational mechanism to designate data for being published to NOMAD. You can use a dedicated dataset for publishable data.
-
-##### How to upload?
-
-We will probably provide functionality in the API of the central NOMAD to upload data from an Oasis to the central NOMAD and provide the necessary scripts and detailed instructions. Most likely the data that is uploaded to the central NOMAD can be selected via a search query. Therefore, using a dedicated dataset, would be an easy to select criteria.
-
-#### How to maintain an Oasis installation?
-
-##### How to install a NOMAD Oasis?
-Follow our guide: https://nomad-lab.eu/prod/v1/docs/ops.html#operating-a-nomad-oasis
-
-##### How do version numbers work?
-There are still a lot of things in NOMAD that are subject to change. Currently, changes in the minor version number (0.x.0) designate major changes that require data migration. Changes in the patch version number (0.7.x) just contain minor changes and fixes and do not require data migration. Once we reach 1.0.0, NOMAD will use the regular semantic versioning conventions.
-
-##### How to upgrade a NOMAD Oasis?
-When we release a new version of the NOMAD software, it will be available as a new Docker image with an increased version number. You simply change the version number in your docker-compose.yaml and restart.
-
-##### What about major releases?
-Going from NOMAD 0.7.x to 0.8.x will require data migration. This means the layout of the data has changed and the new version cannot be used on top of the old data. This requires a separate installation of the new version and mirroring the data from the old version via NOMAD’s API. Detailed instructions will be made available with the new version.
-
-##### How to move data between installations?
-With the release of 0.8.x, we will clarify how to move data between installations. (See last question)
\ No newline at end of file
diff --git a/docs/oasis/migrate.md b/docs/oasis/migrate.md
new file mode 100644
index 0000000000000000000000000000000000000000..9abbc6b712da45e77effabc6b261771b1b2c4f87
--- /dev/null
+++ b/docs/oasis/migrate.md
@@ -0,0 +1,129 @@
+## Software versions
+
+We distribute NOMAD as docker images that are available in our
+[public docker registry](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/container_registry/36).
+The a NOMAD image names looks like this:
+
+```
+gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:v1.2.0
+```
+
+The version tag (e.g. `v1.2.0`) follows **semantic versioning** (`major.minor.patch`). Images
+released under a version tag do not change. There are also variable tags like
+`stable`, `latest`, and `develop`. The image tag for recent feature branches use a
+encoded variant of the respective merge request name. It is generally the safes to use
+version tags.
+
+Our semantic interpretation of **"minor"** is the following:
+
+- there are only additions to programming interfaces and config options
+- NOMAD will still operate on existing data, but **the structure of newly processed data might change**
+- minor version might introduce new features that are only available after certain
+actions [migration steps](#migration-steps).
+
+A road-map for major features can be found on our homepage [here](https://nomad-lab.eu/nomad-lab/features.html). You'll find a detailed change log in the source code [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/blob/develop/CHANGELOG.md).
+
+
+## Configuration versions
+
+Depending on the versions you need to update your docker-compose or
+NOMAD configuration. This might be necessary for breaking changes, or advisable to
+activate new features. Therefore, it is important to understand that the
+installations files that run NOMAD (e.g. `docker-compose.yaml` or `nomad.yaml`) are independent
+of the NOMAD image and they won't automatically change just because you use a new image.
+
+We will list respective changes and guides under [migration steps](#migration-steps) below.
+
+## Data versions
+
+Different version of NOMAD might represent data differently. For example, the
+definition of the search index, database schema, file system layout, or internal file
+formats might change.
+For such changes, we might offer a migration period, where NOMAD supports two different
+data representations at the same time.
+However, eventually this requires some migration of the existing data. This might be necessary
+for breaking changes (or because an old representation is deprecated) or advisable
+because a new representation offers new features.
+Therefore, it is important to understand that the used
+NOMAD software and data are independent things. Using a new image, does not change the
+data in your installation.
+
+We will list respective changes and guides under [migration steps](#migration-steps) below.
+
+## Using a new version
+
+To use a different version of NOMAD, it can be as simple as swapping the tag in the
+docker-compose services that use the nomad images (i.e. `app` and `worker`). Depending
+on the version change, further steps might be necessary:
+
+- for patch releases no further actions should be necessary
+- for minor releases some additional actions might be required to unlock new features
+- for major releases breaking changes are likely and further actions will be required
+
+For changing the minor or major version, please check the [migration steps](#migration-steps) below.
+
+## Migration steps
+
+### to 1.2.0
+
+- We introduced the plugin mechanism. There are now more options to control which schemas
+and parsers are available in your installation. By default all the existing and shipped
+schemas and parsers are enabled. See also [here](customize.md).
+
+- We changed the archive file format. [Re-processing](admin.md#re-process) might yield better performance.
+
+- Parsers are now using a different workflow model and the UI now includes a
+workflow card on the overview page of entries with workflows for the new model.
+[Re-processing](admin.md#re-process) all data will enable this feature for old data. Any analysis build on
+the old workflow model, might not work for new data.
+
+- We introduce the *log-transfer* service. This is currently an opt-in feature.
+
+### from 0.8.x to 1.x
+
+Between versions 0.10.x and 1.x we needed to change how archive and metadata data is stored
+internally in files and databases. This means you cannot simply start a new version of
+NOMAD on top of the old data. But there is a strategy to adapt the data. This should
+work for data based on NOMAD >0.8.0 and <= 0.10.x.
+
+The overall strategy is to create a new mongo database, copy all information, and then
+reprocess all data for the new version.
+
+First, shutdown the OASIS and remove all old containers.
+```sh
+docker compose stop
+docker compose rm -f
+```
+
+Update your config files (`docker-compose.yaml`, `nomad.yaml`, `nginx.conf`) according
+to the latest documentation (see above). Make sure to use index and database names that
+are different. The default values contain a version number in those names, if you don't
+overwrite those defaults, you should be safe.
+
+Make sure you get the latest images and start the OASIS with the new version of NOMAD:
+```sh
+docker compose pull
+docker compose up -d
+```
+
+If you go to the GUI of your OASIS, it should now show the new version and appear empty,
+because we are using a different database and search index now.
+
+To migrate the data, we created a command that you can run within your OASIS' NOMAD
+application container. This command takes the old database name as an argument, it will copy
+all data from the old mongodb to the current one. The default v8.x database name
+was `nomad_fairdi`, but you might have changed this to `nomad_v0_8` as recommended by
+our old Oasis documentation.
+
+```sh
+docker exec -ti nomad_oasis_app bash -c 'nomad admin upgrade migrate-mongo --src-db-name nomad_v0_8'
+docker exec -ti nomad_oasis_app bash -c 'nomad admin uploads reprocess'
+```
+
+Now all your data should appear in your OASIS again. If you like, you can remove the
+old index and database:
+
+```sh
+docker exec nomad_oasis_elastic bash -c 'curl -X DELETE http://elastic:9200/nomad_fairdi'
+docker exec nomad_oasis_mongo bash -c 'mongo nomad_fairdi --eval "printjson(db.dropDatabase())"'
+```
\ No newline at end of file
diff --git a/docs/plugins/parsers.md b/docs/plugins/parsers.md
new file mode 100644
index 0000000000000000000000000000000000000000..c060e9ace6f3875c8a089812958d4aed50ca654a
--- /dev/null
+++ b/docs/plugins/parsers.md
@@ -0,0 +1,7 @@
+NOMAD uses parsers to convert raw code input and output files into NOMAD's common Archive format. This is the documentation on how to develop such a parser.
+
+## Getting started
+
+Fork and clone the [schema example project](https://github.com/nomad-coe/nomad-parser-plugin-example) as described in [before](plugins.md). Follow the original [How to write a parser](../develop/parsers.md) documentation.
+
+{{pydantic_model('nomad.config.plugins.Parser', heading='### Parser plugin metadata', hide=['code_name','code_category','code_homepage','metadata'])}}
\ No newline at end of file
diff --git a/docs/plugins.md b/docs/plugins/plugins.md
similarity index 89%
rename from docs/plugins.md
rename to docs/plugins/plugins.md
index d733a06ee8866a5a2cd408902709a59776f84674..69c95319df534704c3055c02c0f7f3221f7e00d8 100644
--- a/docs/plugins.md
+++ b/docs/plugins/plugins.md
@@ -5,8 +5,8 @@ plugins.
We support different kinds of plugins:
-- Python **schema**, read also [Python schema documentation](schema/python.md).
-- **parser**, read also [parser development documentation](develop/parser.md).
+- Python **schema**
+- **parser**
- **normalizer** (coming soon...)
- additional custom **APIs** (coming soon...)
@@ -50,7 +50,7 @@ my-nomad-schema
### Plugin code
The directory `nomadschemaexample` is our Python package *plugin code*. In this case,
-it contains a simple `schema.py`. Read the [Python schema documentation](schema/python.md)
+it contains a simple `schema.py`. Read the [Schema plugin documentation](schemas.md)
for more details:
```python
@@ -72,7 +72,7 @@ metadata that is necessary to use them. See below for a reference of the *plugin
models.
The file `nomad.yaml` shows how to add the plugin to a nomad installation. As a plugin
-developer you have [installed our Python package](./pythonlib.md) and can run the `nomad parse`
+developer you have [installed our Python package](../apis/pythonlib.md) and can run the `nomad parse`
command as your "installation" to try your schema:
```yaml
@@ -93,9 +93,6 @@ Please note that `python_package` is the name of a Python package and not a path
code. This also means that the package has to be in your `PYTHONPATH` (see below).
-{{pydantic_model('nomad.config.plugins.Schema', heading='### Parser plugin metadata')}}
-{{pydantic_model('nomad.config.plugins.Parser', heading='### Schema plugin metadata', hide=['code_name','code_category','code_homepage','metadata'])}}
-
Now follow the instructions for one of our examples and try for yourself:
- [schema plugin](https://github.com/nomad-coe/nomad-schema-plugin-example)
@@ -145,7 +142,7 @@ export COMPOSE_FILE=docker-compose.yaml:docker-compose.plugins.yaml
docker compose up -d
```
-Here is a complete Oasis setup [nomad-oasis-with-plugins.zip](assets/nomad-oasis-with-plugins.zip).
+Here is a complete Oasis setup [nomad-oasis-with-plugins.zip](../assets/nomad-oasis-with-plugins.zip).
Simply download, extract, and start like any other Oasis:
```sh
@@ -158,7 +155,7 @@ docker compose up -d
curl localhost/nomad-oasis/alive
```
-Read the [Oasis documentation](oasis.md) for more details.
+Read the [Oasis install guide](../oasis/install.md) for more details.
### Other means
diff --git a/docs/schema/python.md b/docs/plugins/schemas.md
similarity index 87%
rename from docs/schema/python.md
rename to docs/plugins/schemas.md
index 6472a3a7e4d07ad6acfae76504288e93ae65b856..c73dfc9321a47cf79e97f670f53715d6a7a94899 100644
--- a/docs/schema/python.md
+++ b/docs/plugins/schemas.md
@@ -1,16 +1,12 @@
----
-title: Python schemas
----
+## Getting started
-# Write NOMAD schemas in Python
+Fork and clone the [schema example project](https://github.com/nomad-coe/nomad-schema-plugin-example) as described in [before](plugins.md).
-Writing and uploading schemas in `.archive.yaml` format is a good way for NOMAD users
-to add schemas. But it has limitations. As a NOMAD developer or Oasis administrator
-you can add Python schemas to NOMAD. All build in NOMAD schemas (e.g. for electronic structure code data) are written an Python and are part of the NOMAD sources (`nomad.datamodel.metainfo.*`).
+## Writing schemas in Python compared to YAML schemas
-There is a 1-1 translation between Python schemas (written in classes) and YAML (or JSON)
-schemas (written in objects). Both use the same fundamental concepts, like
-*section*, *quantity*, or *sub-section*, introduced in [YAML schemas](basics.md).
+In this [guide](../schemas/basics.md), we explain how to write and upload schemas in the `.archive.yaml` format. Writing and uploading such YAML schemas is a good way for NOMAD users to add schemas. But it has limitations. As a NOMAD developer or Oasis administrator you can add Python schemas to NOMAD. All build in NOMAD schemas (e.g. for electronic structure code data) are written an Python and are part of the NOMAD sources (`nomad.datamodel.metainfo.*`).
+
+There is a 1-1 translation between Python schemas (written in classes) and YAML (or JSON) schemas (written in objects). Both use the same fundamental concepts, like *section*, *quantity*, or *sub-section*, introduced in [YAML schemas](../schemas/basics.md).
## Starting example
@@ -304,7 +300,7 @@ short handle of a code name or other special method prefix.
### Access structured data via API
-The [API section](../api.md#access-archives) demonstrates how to access an Archive, i.e.
+The [API section](../apis/api.md#access-archives) demonstrates how to access an Archive, i.e.
retrieve the processed data from a NOAMD entry. This API will give you JSON data likes this:
```json title="https://nomad-lab.eu/prod/v1/api/v1/entries/--dLZstNvL_x05wDg2djQmlU_oKn/archive"
@@ -359,7 +355,7 @@ To learn what each key means, you need to look up its definition in the Metainfo
In Python, JSON data is typically represented as nested combinations of dictionaries
and lists. Of course, you could work with this right away. To make it easier for Python
-programmers the [NOMAD Python package](../pythonlib.md) allows you to use this
+programmers the [NOMAD Python package](../apis/pythonlib.md) allows you to use this
JSON data with a higher level interface, which provides the following advantages:
- code completion in dynamic coding environments like Jupyter notebooks
@@ -392,3 +388,45 @@ print(json.dumps(calc.m_to_dict(), indent=2))
The NOMAD Python package provides utilities to [query large amounts of
archive data](/archive_query.md). This uses the build-in Python schema classes as
an interface to the data.
+
+## Custom normalizers
+
+For custom schemas, you might want to add custom normalizers. All files are parsed
+and normalized when they are uploaded or changed. The NOMAD metainfo Python interface
+allows you to add functions that are called when your data is normalized.
+
+Here is an example:
+
+```python
+--8<-- "examples/archive/custom_schema.py"
+```
+
+To add a `normalize` function, your section has to inherit from `ArchiveSection` which
+provides the base for this functionality. Now you can overwrite the `normalize` function
+and add you own behavior. Make sure to call the `super` implementation properly to
+support schemas with multiple inheritance.
+
+If we parse an archive like this:
+
+```yaml
+--8<-- "examples/archive/custom_data.archive.yaml"
+```
+
+we will get a final normalized archive that contains our data like this:
+
+```json
+{
+ "data": {
+ "m_def": "examples.archive.custom_schema.SampleDatabase",
+ "samples": [
+ {
+ "added_date": "2022-06-18T00:00:00+00:00",
+ "formula": "NaCl",
+ "sample_id": "2022-06-18 00:00:00+00:00--NaCl"
+ }
+ ]
+ }
+}
+```
+
+{{pydantic_model('nomad.config.plugins.Schema', heading='## Schema plugin metadata')}}
\ No newline at end of file
diff --git a/docs/reference/annotations.md b/docs/reference/annotations.md
new file mode 100644
index 0000000000000000000000000000000000000000..38c2ac7ce0c3334052085e31170ae1a499a75301
--- /dev/null
+++ b/docs/reference/annotations.md
@@ -0,0 +1,25 @@
+# Schema annotations
+
+Definitions in a schema can have annotations. These annotations provide additional information that NOMAD can use to alter its behavior around these definitions. Annotations are named blocks of key-value pairs:
+
+```yaml
+definitions:
+ sections:
+ MyAnnotatedSection:
+ m_annotations:
+ annotation_name:
+ key1: value
+ key2: value
+```
+
+Many annotations control the representation of data in the GUI. This can be for plots or data entry/editing capabilities.
+
+{{ pydantic_model('nomad.datamodel.metainfo.annotations.ELNAnnotation', heading='## eln') }}
+
+## Tabular data
+{{ pydantic_model('nomad.datamodel.metainfo.annotations.TabularParserAnnotation', heading='### tabular_parser') }}
+{{ pydantic_model('nomad.datamodel.metainfo.annotations.TabularAnnotation', heading='### tabular') }}
+
+{{ pydantic_model('nomad.datamodel.metainfo.annotations.PlotAnnotation', heading='## plot') }}
+
+{{ pydantic_model('nomad.datamodel.metainfo.annotations.BrowserAnnotation', heading='## browser') }}
diff --git a/docs/reference/cli.md b/docs/reference/cli.md
new file mode 100644
index 0000000000000000000000000000000000000000..62e94ea077af6af9a2292bd0a52b38cff7d49fce
--- /dev/null
+++ b/docs/reference/cli.md
@@ -0,0 +1,13 @@
+---
+toc_depth: 6
+---
+
+# Command Line Interface (CLI)
+
+::: mkdocs-click
+ :module: nomad.cli.cli
+ :command: cli
+ :prog_name: nomad
+ :depth: 1
+ :list_subcommands: True
+
diff --git a/docs/config.md b/docs/reference/config.md
similarity index 100%
rename from docs/config.md
rename to docs/reference/config.md
diff --git a/docs/glossary.md b/docs/reference/glossary.md
similarity index 99%
rename from docs/glossary.md
rename to docs/reference/glossary.md
index 10d24f5ae23c3e65225820646c8daec3a346187e..7fda59e53bd8a1d9b5b765da78540d1c370af624 100644
--- a/docs/glossary.md
+++ b/docs/reference/glossary.md
@@ -18,7 +18,7 @@ out the application and this documentation.
*Annotations* are part of data [schemas](#schema) and they describe aspects that are not
directly defining the type or shape of data. They often allow to alter how certain data is
-managed, represented, or edited. See [annotations in the schema documentation](schema/elns.md#annotations).
+managed, represented, or edited. See [annotations in the schema documentation](../schemas/elns.md#annotations).
### Archive
diff --git a/docs/reference/parsers.md b/docs/reference/parsers.md
new file mode 100644
index 0000000000000000000000000000000000000000..0279a50906c2efdd6ec66c8888ec240aba3557bd
--- /dev/null
+++ b/docs/reference/parsers.md
@@ -0,0 +1,5 @@
+# Supported parsers
+
+!!! attention
+
+ This part of the documentation is still work in progress.
\ No newline at end of file
diff --git a/docs/schema/basics.md b/docs/schemas/basics.md
similarity index 99%
rename from docs/schema/basics.md
rename to docs/schemas/basics.md
index 3a5565808eb63896343db8454cf41aa3ff29a971..e9e3a907987a32d23fc0340ddc2c802550a4f8a4 100644
--- a/docs/schema/basics.md
+++ b/docs/schemas/basics.md
@@ -1,11 +1,6 @@
----
-title: Yaml schemas
----
-
# Write NOMAD Schemas in YAML
-This page explains the fundamental concepts behind NOMAD schemas and how you can
-write and upload schemas in our `.archive.yaml` format.
+This guide explains how to write and upload NOMAD schemas in our `.archive.yaml` format. For more information visit the [learn section on schemas](../learn/schemas.md).
## Example data
diff --git a/docs/schemas/elns.md b/docs/schemas/elns.md
new file mode 100644
index 0000000000000000000000000000000000000000..64ed0b0bf7e6e0c4f88ab8c76bc1d73411b6f945
--- /dev/null
+++ b/docs/schemas/elns.md
@@ -0,0 +1,25 @@
+# Schemas for ELNs
+
+A schema defines all possible data structures. With small editions to our schemas, we can instruct NOMAD to provide respective editors for data. This allows us to build Electronic Lab Notebooks (ELNs) as tools to acquire data in a formal and structured way. For schemas with ELN annotations, users can create new entries in NOMAD GUI and edit the archive (structured data) of these entries directly in the GUI.
+
+## Annotations
+
+Definitions in a schema can have annotations. With these annotations you can provide additional information that NOMAD can use to alter its behavior around these definitions. Annotations are named blocks of key-value pairs:
+
+```yaml
+definitions:
+ sections:
+ MyAnnotatedSection:
+ m_annotations:
+ annotation_name:
+ key1: value
+ key2: value
+```
+
+Many annotations control the representation of data in the GUI. This can be for plots or data entry/editing capabilities. There are three main categories of annotations relevant to ELNs. You find a reference of all annotations [here](../reference/annotations.md).
+
+## Example ELN schema
+The is the commented ELN schema from our ELN example upload that can be created from NOMAD's upload page:
+```yaml
+--8<-- "examples/data/eln/schema.archive.yaml"
+```
diff --git a/docs/assets/example-workflow.png b/docs/schemas/example-workflow.png
similarity index 100%
rename from docs/assets/example-workflow.png
rename to docs/schemas/example-workflow.png
diff --git a/docs/schema/hdf5.md b/docs/schemas/hdf5.md
similarity index 100%
rename from docs/schema/hdf5.md
rename to docs/schemas/hdf5.md
diff --git a/docs/schema/elns.md b/docs/schemas/tabular.md
similarity index 85%
rename from docs/schema/elns.md
rename to docs/schemas/tabular.md
index bab62239a55255cfd8b1bc1ea019308c23bff3b3..2944dc79c7b58d0624c438554fea0d320b6e1d9d 100644
--- a/docs/schema/elns.md
+++ b/docs/schemas/tabular.md
@@ -1,62 +1,16 @@
-# Schemas for ELNs
-
-A schema defines all possible data structures. With small editions to our schemas,
-we can instruct NOMAD to provide respective editors for data. This allows us
-to build Electronic Lab Notebooks (ELNs) as tools to acquire data in a formal and
-structured way. For schemas with ELN annotations, users can create new entries
-in NOMAD GUI and edit the archive (structured data) of these entries directly in the GUI.
-
-## Annotations
-
-Definitions in a schema can have annotations. These annotations provide additional information that NOMAD can use to alter its behavior around these definitions. Annotations
-are named blocks of key-value pairs:
-
-```yaml
-definitions:
- sections:
- MyAnnotatedSection:
- m_annotations:
- annotation_name:
- key1: value
- key2: value
-```
-
-Many annotations control the representation of data in the GUI. This can be for plots
-or data entry/editing capabilities. There are three main categories of annotations
-relevant to ELNs.
-
-- `eln` annotations shape the data editor, i.e. allow you to control which type of forms to use to edit quantities.
-- With `tabular` annotation data from linked `.csv` or `Excel` files can be parsed and added
-to data.
-- `plot` annotation allows you to plot numerical data (e.g. added via tables) directly
-in the ELN (or)
-
-## Example ELN
-The is the commented ELN schema from our ELN example upload that can be created from
-NOMAD's upload page:
-```yaml
---8<-- "examples/data/eln/schema.archive.yaml"
-```
-
-{{ pydantic_model('nomad.datamodel.metainfo.annotations.ELNAnnotation', heading='## ELN Annotation') }}
-
-{{ pydantic_model('nomad.datamodel.metainfo.annotations.BrowserAnnotation', heading='## Browser Annotation') }}
-
-
-## Tabular Annotations
In order to import your data from a `.csv` or `Excel` file, NOMAD provides three distinct (and separate) ways, that
with each comes unique options for importing and interacting with your data. In order to better understand how to use
NOMAD tabular parser to import your data, follow three sections below. In each section you
can find a commented sample schema with a step-by-step guide on how to import your tabular data.
-Tabular parser, implicitly, parse the data into the same NOMAD entry where the datafile is loaded. Also, explicitly,
+Tabular parser, implicitly, parse the data into the same NOMAD entry where the datafile is loaded. Also, explicitly,
this can be defined by putting the corresponding annotations under `current_entry` (check the examples below).
-In addition, tabular parser can be set to parse the data into new entry (or entries). For this, the proper annotations
+In addition, tabular parser can be set to parse the data into new entry (or entries). For this, the proper annotations
should be appended to `new_entry` annotation in your schema file.
Two main components of any tabular parser schema are:
1) implementing the correct base-section(s), and
-2) providing a `data_file` `Quantity` with the correct `m_annotations`.
+2) providing a `data_file` `Quantity` with the correct `m_annotations`.
Please bear in mind that the schema files should 1) follow the NOMAD naming convention
(i.e. `My_Name.archive.yaml`), and 2) be accompanied by your data file in order for NOMAD to parse them.
@@ -65,23 +19,23 @@ In the examples provided below, an `Excel` file is assumed to contain all the da
should match the name of the `Excel` data file, which in case of using a `.csv` data file, it can be replaced by the
`.csv` file name.
-`TableData` (and any other section(s) that is inheriting from `TableData`) has a customizable checkbox Quantity
+`TableData` (and any other section(s) that is inheriting from `TableData`) has a customizable checkbox Quantity
(i.e. `fill_archive_from_datafile`) to turn the tabular parser `on` or `off`.
If you do not want to have the parser running everytime you make a change to your archive data, it is achievable then via
unchecking the checkbox. It is customizable in the sense that if you do not wish to see this checkbox at all,
-you can configure the `hide` parameter of the section's `m_annotations` to hide the checkbox. This in turn sets
-the parser to run everytime you save your archive.
+you can configure the `hide` parameter of the section's `m_annotations` to hide the checkbox. This in turn sets
+the parser to run everytime you save your archive.
Be cautious though! Turning on the tabular parser (or checking the box) on saving your data will cause
-losing/overwriting your manually-entered data by the parser!
+losing/overwriting your manually-entered data by the parser!
-#### Column-mode Sample:
+## Column-mode
The following sample schema creates one quantity off the entire column of an excel file (`column mode`).
For example, suppose in an excel sheet, several rows contain information of a chemical product (e.g. `purity` in one
column). In order to list all the purities under the column `purity` and import them into NOMAD, you can use the
following schema by substituting `My_Quantity` with any name of your choice (e.g. `Purity`),
`tabular-parser.data.xlsx` with the name of the `csv/excel` file where the data lies, and `My_Sheet/My_Column` with
-sheet_name/column_name of your targeted data. The `Tabular_Parser` can also be changed to any arbitrary name of your
+sheet_name/column_name of your targeted data. The `Tabular_Parser` can also be changed to any arbitrary name of your
choice.
Important notes:
@@ -89,7 +43,7 @@ Important notes:
- `shape: ['*']` under `My_Quantity` is essential to parse the entire column of the data file.
- The `data_file` `Quantity` can have any arbitrary name (e.g. `xlsx_file`)
- `My_Quantity` can also be defined within another subsection (see next sample schema)
-- Use `current_entry` and append `column_to_sections` to specify which sub_section(s) is to be filled in
+- Use `current_entry` and append `column_to_sections` to specify which sub_section(s) is to be filled in
this mode. `Leaving this field empty` causes the parser to parse the entire schema under column mode.
```yaml
@@ -166,7 +120,7 @@ your data is populated under the `data` sub_section.
<b>Advanced options to use/set in tabular parser:</b>
- If you want to populate your schema from multiple `excel/csv` files, you can
-define multiple data_file `Quantity`s annotated with `tabular_parser` in the root level of your schema
+define multiple data_file `Quantity`s annotated with `tabular_parser` in the root level of your schema
(root level of your schema is where you inherit from `TableData` class under `base_sections`).
Each individual data_file quantity can now contain a list of sub_sections which are expected to be filled
using one- or all of the modes mentioned above. Check the `MyOverallSchema` section in
@@ -248,7 +202,7 @@ we will get a final normalized archive that contains our data like this:
NOMAD offers integration with third-party ELN providers, simplifying the process of connecting
and interacting with external platforms. Three main external ELN solutions that are integrated into NOMAD
-are: [elabFTW](https://www.elabftw.net/), [Labfolder](https://labfolder.com/) and [chemotion](https://chemotion.net/).
+are: [elabFTW](https://www.elabftw.net/), [Labfolder](https://labfolder.com/) and [chemotion](https://chemotion.net/).
The process of data retrieval and data mapping onto NOMAD's schema
varies for each of these third-party ELN provider as they inherently allow for certain ways of communicating with their
database. Below you can find a <b>How-to</b> guide on importing your data from each of these external
@@ -267,17 +221,17 @@ your experiments.
Go to your elabFTW experiment and export your project as `ELN Archive`. Save the file to your filesystem under
your preferred name and location (keep the `.eln` extension intact).
To parse your ebalFTW data into NOMAD,
-go to the upload page of NOMAD and create a new upload. In the `overview` page, upload your exported file (either by
+go to the upload page of NOMAD and create a new upload. In the `overview` page, upload your exported file (either by
drag-dropping it into the <i>click or drop files</i> box or by navigating to the path where you stored the file).
This causes triggering NOMAD's parser to create as many new entries in this upload as there are experiments in your
elabFTW project.
-You can inspect the parsed data of each of your entries (experiments) by going to the <b>DATA</b>
-tab of each entry page. Under <i>Entry</i> column, click on <i>data</i> section. Now a new lane titled
+You can inspect the parsed data of each of your entries (experiments) by going to the <b>DATA</b>
+tab of each entry page. Under <i>Entry</i> column, click on <i>data</i> section. Now a new lane titled
`ElabFTW Project Import` should be visible. Under this section, (some of) the metadata of your project is listed.
There two sub-sections: 1) <b>experiment_data</b>, and 2) <b>experiment_files</b>.
-<b>experiment_data</b> section contains detailed information of the given elabFTW experiment, such as
+<b>experiment_data</b> section contains detailed information of the given elabFTW experiment, such as
links to external resources and extra fields. <b>experiment_files</b> section is a list of sub-sections
containing metadata and additional info of the files associated with the experiment.
@@ -320,4 +274,4 @@ contains data of any text field in your Labfodler project.
### Chemotion integration
-Coming soon
\ No newline at end of file
+Coming soon
diff --git a/docs/assets/workflow-schema.png b/docs/schemas/workflow-schema.png
similarity index 100%
rename from docs/assets/workflow-schema.png
rename to docs/schemas/workflow-schema.png
diff --git a/docs/schema/workflows.md b/docs/schemas/workflows.md
similarity index 97%
rename from docs/schema/workflows.md
rename to docs/schemas/workflows.md
index ef4ed97a17c7075d953aec15cd650b5f7e1ceb87..0f64fc1fec47e1d6f768d3142834cdea38d69d3b 100644
--- a/docs/schema/workflows.md
+++ b/docs/schemas/workflows.md
@@ -12,7 +12,7 @@ performed . This often is also referred to as *data provenance* or *provenance g
The following shows the overall abstract schema for *worklows* that can be found
in `nomad.datamodel.metainfo.workflow` (blue):
-
+
The idea is that *workflows* are stored in a top-level archive section along-side other
sections that contain the *inputs* and *outputs*. This way the *workflow* or *provenance graph*
@@ -27,7 +27,7 @@ top-level section.
Here is a logical depiction of the workflow and all its tasks, inputs, and outputs.
-
+
### Simple workflow entry
diff --git a/docs/stylesheets/extra.css b/docs/stylesheets/extra.css
index 18bc8723212f37c43eaa14590bba9b07c0ec12bc..321e087066b2696d28bf0d1c05085d2afca25a85 100644
--- a/docs/stylesheets/extra.css
+++ b/docs/stylesheets/extra.css
@@ -30,6 +30,11 @@
background-color: rgba(0,0,0,.13);
}
+.md-typeset h1 {
+ color: black;
+ font-weight: 700;
+}
+
.youtube {
position: relative;
width: 100%;
@@ -43,4 +48,22 @@
left: 0;
width: 100%;
height: 100%;
+}
+
+.home-grid {
+ display: grid;
+ grid-template-columns: 1fr 1fr;
+ grid-column-gap: 24px;
+ row-gap: 24px;
+}
+
+.home-grid div {
+ border-radius: 4px;
+ padding: 24px;
+ background-color: #f3e9d9;
+}
+
+.home-grid h3 {
+ margin-top: 0;
+ font-weight: 700;
}
\ No newline at end of file
diff --git a/docs/theme/partials/header.html b/docs/theme/partials/header.html
new file mode 100644
index 0000000000000000000000000000000000000000..5b091f3e96f16ff11ec8de8955194cbc37172c45
--- /dev/null
+++ b/docs/theme/partials/header.html
@@ -0,0 +1,86 @@
+{#-
+ This file was automatically generated - do not edit
+-#}
+{% set class = "md-header" %}
+{% if "navigation.tabs.sticky" in features %}
+ {% set class = class ~ " md-header--lifted" %}
+{% endif %}
+<header class="{{ class }}" data-md-component="header">
+ <nav class="md-header__inner md-grid" aria-label="{{ lang.t('header.title') }}">
+ <a href="{{ config.extra.homepage | d(nav.homepage.url, true) | url }}" title="{{ config.site_name | e }}" class="md-header__button md-logo" aria-label="{{ config.site_name }}" data-md-component="logo">
+ {% include "partials/logo.html" %}
+ </a>
+ <label class="md-header__button md-icon" for="__drawer">
+ {% include ".icons/material/menu" ~ ".svg" %}
+ </label>
+ <div class="md-header__title" data-md-component="header-title">
+ <div class="md-header__ellipsis">
+ <div class="md-header__topic">
+ <span class="md-ellipsis">
+ {{ config.site_name }}
+ </span>
+ </div>
+ <div class="md-header__topic" data-md-component="header-topic">
+ <span class="md-ellipsis">
+ {% if page and page.meta and page.meta.title %}
+ {{ page.meta.title }}
+ {% else %}
+ {{ page.title }}
+ {% endif %}
+ </span>
+ </div>
+ </div>
+ </div>
+ {% if not config.theme.palette is mapping %}
+ <form class="md-header__option" data-md-component="palette">
+ {% for option in config.theme.palette %}
+ {% set primary = option.primary | replace(" ", "-") | lower %}
+ {% set accent = option.accent | replace(" ", "-") | lower %}
+ <input class="md-option" data-md-color-media="{{ option.media }}" data-md-color-scheme="{{ option.scheme }}" data-md-color-primary="{{ primary }}" data-md-color-accent="{{ accent }}" {% if option.toggle %} aria-label="{{ option.toggle.name }}" {% else %} aria-hidden="true" {% endif %} type="radio" name="__palette" id="__palette_{{ loop.index }}">
+ {% if option.toggle %}
+ <label class="md-header__button md-icon" title="{{ option.toggle.name }}" for="__palette_{{ loop.index0 or loop.length }}" hidden>
+ {% include ".icons/" ~ option.toggle.icon ~ ".svg" %}
+ </label>
+ {% endif %}
+ {% endfor %}
+ </form>
+ {% endif %}
+ {% if config.extra.alternate %}
+ <div class="md-header__option">
+ <div class="md-select">
+ {% set icon = config.theme.icon.alternate or "material/translate" %}
+ <button class="md-header__button md-icon" aria-label="{{ lang.t('select.language.title') }}">
+ {% include ".icons/" ~ icon ~ ".svg" %}
+ </button>
+ <div class="md-select__inner">
+ <ul class="md-select__list">
+ {% for alt in config.extra.alternate %}
+ <li class="md-select__item">
+ <a href="{{ alt.link | url }}" hreflang="{{ alt.lang }}" class="md-select__link">
+ {{ alt.name }}
+ </a>
+ </li>
+ {% endfor %}
+ </ul>
+ </div>
+ </div>
+ </div>
+ {% endif %}
+ {% if "search" in config["plugins"] %}
+ <label class="md-header__button md-icon" for="__search">
+ {% include ".icons/material/magnify.svg" %}
+ </label>
+ {% include "partials/search.html" %}
+ {% endif %}
+ {% if config.repo_url %}
+ <div class="md-header__source">
+ {% include "partials/source.html" %}
+ </div>
+ {% endif %}
+ </nav>
+ {% if "navigation.tabs.sticky" in features %}
+ {% if "navigation.tabs" in features %}
+ {% include "partials/tabs.html" %}
+ {% endif %}
+ {% endif %}
+</header>
\ No newline at end of file
diff --git a/docs/todo.md b/docs/todo.md
new file mode 100644
index 0000000000000000000000000000000000000000..3dc876699559b0086b6f5eb32253a56e096ac35e
--- /dev/null
+++ b/docs/todo.md
@@ -0,0 +1,3 @@
+# Coming soon ...
+
+We still have to write this.
\ No newline at end of file
diff --git a/docs/tutorial.md b/docs/tutorial.md
new file mode 100644
index 0000000000000000000000000000000000000000..e8e4945ebff94551ea6e7526297c340ecf092cb2
--- /dev/null
+++ b/docs/tutorial.md
@@ -0,0 +1,43 @@
+This is a series of short videos that guide you through the main functionality of NOMAD.
+It covers the whole data-life cycle: starting with data on your hard drive,
+you will learn how to prepare, upload, publish data, and reference them with a DOI.
+Furthermore, you will learn how to explore, download, and use data that were published on NOMAD before.
+We will perform these steps with NOMAD's graphical user interface and its APIs.
+
+- [Example data and exercises](https://www.fairmat-nfdi.eu/events/fairmat-tutorial-1/tutorial-1-materials)
+- [More videos and tutorials](https://youtube.com/playlist?list=PLrRaxjvn6FDW-_DzZ4OShfMPcTtnFoynT)
+
+!!! note
+ The NOMAD seen in the tutorials is an older version with a different color theme,
+ but all the demonstrated functionality is still available on the current version.
+ You'll find the NOMAD test installation mentioned in the first video
+ [here](https://nomad-lab.eu/prod/v1/test/gui/search/entries).
+
+## Uploading and publishing data
+
+This tutorial guides you through the basics of going from files on your computer
+to a published dataset with DOI.
+
+<div class="youtube">
+<iframe src="https://www.youtube.com/embed/3rVvfYoUbO0" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+## Exploring data on
+
+This tutorial shows how to use NOMAD's search interface and structured data browsing to explore available data.
+
+<div class="youtube">
+<iframe src="https://www.youtube.com/embed/38S2U-TIvxE" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+
+## Access data via API
+
+This video tutorial explains the basics of API and shows how to do simple requests
+against the NOMAD api.
+
+<div class="youtube">
+<iframe src="https://www.youtube.com/embed/G1frBCrxC0g" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
+</div>
+
+
diff --git a/mkdocs.yml b/mkdocs.yml
index 075a27fb5ba78e2385a77118e2ac947e89b7e176..5ef8924a1638a995663510373e3215c10e1f54ac 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -3,33 +3,52 @@ site_description: |
The documentation for NOMAD v1.
site_author: The NOMAD Authors
nav:
- - Introduction: index.md
- - web.md
- - api.md
- - Python package:
- - pythonlib.md
- - archive_query.md
- - local_parsers.md
- - Schemas:
- - schema/introduction.md
- - schema/basics.md
- - schema/python.md
- - schema/elns.md
- - schema/workflows.md
- - schema/hdf5.md
- - Plugins: plugins.md
- # - Using the AI Toolkit and other remote tools: aitoolkit.md
- - Developing NOMAD:
- - develop/setup.md
- - develop/guides.md
- - develop/gitlab.md
- - develop/search.md
- - develop/parser.md
- - develop/normalizers.md
- - Operating NOMAD:
- - oasis.md
- - config.md
- - glossary.md
+ - Home: index.md
+ - Tutorial: tutorial.md
+ - How to guides:
+ - Data Management:
+ - How to upload/publish data: data/upload.md
+ - How to use ELNs: data/eln.md
+ - How to explore data: data/explore.md
+ - How to use NORTH: data/north.md
+ - Schemas:
+ - How to write a schema: schemas/basics.md
+ - How to define ELNs: schemas/elns.md
+ - How to define tabular data: schemas/tabular.md
+ - How to define workflows: schemas/workflows.md
+ - How to reference hdf5: schemas/hdf5.md
+ - Programming interfaces:
+ - How to use the API: apis/api.md
+ - How to install nomad-lab: apis/pythonlib.md
+ - How to access processed data: apis/archive_query.md
+ - How to run a parser: apis/local_parsers.md
+ - Plugins:
+ - How to develop and use plugins: plugins/plugins.md
+ - How to write schema plugins: plugins/schemas.md
+ - How to write parser plugins: plugins/parsers.md
+ - Development:
+ - How to get started: develop/setup.md
+ - Code guidelines: develop/guides.md
+ - How to contribute: develop/gitlab.md
+ - How to extend the search: develop/search.md
+ - How to write a parser: develop/parsers.md
+ - How to write a normalizer: develop/normalizers.md
+ - Oasis:
+ - How to install an Oasis: oasis/install.md
+ - How to customize an Oasis: oasis/customize.md
+ - How to migrate Oasis versions: oasis/migrate.md
+ - Administrative tasks: oasis/admin.md
+ - Learn:
+ - Files, processing, data: learn/how_nomad_works.md
+ - Schemas and structured data: learn/schemas.md
+ - Architecture: learn/architecture.md
+ - Why you need an Oasis: learn/oasis.md
+ - Reference:
+ - reference/config.md
+ - reference/annotations.md
+ - reference/cli.md
+ - reference/parsers.md
+ - reference/glossary.md
theme:
name: material
palette:
@@ -41,6 +60,7 @@ theme:
favicon: assets/favicon.png
features:
- navigation.instant
+ custom_dir: docs/theme
# repo_url: https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/
markdown_extensions:
- attr_list
@@ -54,9 +74,11 @@ markdown_extensions:
- pymdownx.superfences
- toc:
permalink: True
- toc_depth: 3
- pymdownx.arithmatex:
generic: true
+ - attr_list
+ - mkdocs-click
+ - pymdownx.extra
extra:
generator: false
homepage: https://nomad-lab.eu
@@ -65,6 +87,10 @@ plugins:
- search
- macros:
module_name: nomad/mkdocs
+ - redirects:
+ redirect_maps:
+ 'pythonlib.md': 'apis/pythonlib.md'
+ 'oasis.md': 'oasis/install.md'
extra_css:
- stylesheets/extra.css
extra_javascript:
diff --git a/pyproject.toml b/pyproject.toml
index af6fae87d0c768d9a489d6d4a90f4d4342b83657..080618429ffe511f57910bfed4b327ebddedfb00 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -130,7 +130,9 @@ dev = [
'mkdocs-material==8.1.1',
'mkdocs-material-extensions==1.0.3',
'mkdocs-macros-plugin==0.6.3',
- 'aiosmtpd'
+ 'aiosmtpd',
+ 'mkdocs-click==0.8.0',
+ 'mkdocs-redirects==1.2.0'
]
[project.scripts]