diff --git a/docs/explanation/processing.md b/docs/explanation/processing.md index 5acf566f9b42dfb6a6d804b1d33534bc5e16decd..25e745ba9e4e234dd54d67f7ccdc0d2e6df36a3f 100644 --- a/docs/explanation/processing.md +++ b/docs/explanation/processing.md @@ -90,7 +90,7 @@ section, might indirectly use custom processing functionality. A parser plugin can define a new parser and therefore add to the *matching*, *parsing*, (and *normalizing*). A schema plugin defines new sections that can contain `normalize` functions that add to the *normalizing*. -See also the how-tos on [plugins installation](../howto/oasis/plugins_install.md), and development of [parsers and schemas](../howto/plugins/plugins.md). +See also the how-tos on the development of [parsers and schemas](../howto/plugins/plugins.md). #### Matching diff --git a/docs/howto/customization/basics.md b/docs/howto/customization/basics.md index 33e3e1910de062602701da502c019eed57f45e06..281c6f769382adc690a119cdd09accae54fd082f 100644 --- a/docs/howto/customization/basics.md +++ b/docs/howto/customization/basics.md @@ -320,7 +320,7 @@ Here are a few other built-in section definitions and packages of definitions: |nomad.datamodel.metainfo.eln.*|A package of section definitions to inherit commonly used quantities for ELNs. These quantities are indexed and allow specialization to utilize the NOMAD search.| |nomad.datamodel.metainfo.workflow.*|A package of section definitions use by NOMAD to define workflows| |nomad.metainfo.*|A package that contains all *definitions* of *definitions*, e.g. NOMAD's "schema language". Here you find *definitions* for what a sections, quantity, subsections, etc. is.| -|nomad.parsing.tabular.TableData|Allows to inherit parsing of references .csv and .xls files. See the [detailed description](tabular.html) to learn how to include this class and its annotations in a yaml schema.| +|nomad.parsing.tabular.TableData|Allows to inherit parsing of references .csv and .xls files. See the [detailed description](tabular.md) to learn how to include this class and its annotations in a yaml schema.| |nomad.datamodel.metainfo.basesections.HDF5Normalizer|Allows to link quantities to hdf5 dataset, improving performance for large data. This class and the related annotations are included in a yaml schema. [Dedicated classes](hdf5.md#how-to-use-hdf5-to-handle-large-quantities) can be used to write a parser.| ### HDF5Normalizer diff --git a/docs/howto/customization/hdf5.md b/docs/howto/customization/hdf5.md index 22a4a2548964f47346ed6318cb8b89d319c1ad6e..42b52680b29e936b008dc12cc4e541b5a3405372 100644 --- a/docs/howto/customization/hdf5.md +++ b/docs/howto/customization/hdf5.md @@ -136,7 +136,7 @@ provide respective functionality (e.g. showing a H5Web view). When multiple quantities need to be displayed in the same plot, some attributes in the HDF5 file groups are needed, in order for h5web to be able to render a plot. -The [H5WebAnnotation class](../../reference/annotations.html#h5web) contains the attributes to be included in the groups of HDF5 file, +The [H5WebAnnotation class](../../reference/annotations.md#h5web) contains the attributes to be included in the groups of HDF5 file, provided as section annotations. In the following example, the `value` quantity has a dedicated default h5web rendering. diff --git a/docs/howto/oasis/admin.md b/docs/howto/oasis/admin.md index ccd27b5eae690875c501aec5325dc80cb7f7dc13..8a1309c0035ae68e1f5910ae0f305d087247b43b 100644 --- a/docs/howto/oasis/admin.md +++ b/docs/howto/oasis/admin.md @@ -21,6 +21,7 @@ docker exec nomad_oasis_mongo mongorestore /backup ``` Now you still have to recreate the elasticsearch index: + ```sh docker exec nomad_oasis_app python -m nomad.cli admin uploads index ``` @@ -34,18 +35,21 @@ docker exec -ti nomad_oasis_app bash ``` For example you can ls or remove uploads: + ```sh nomad admin uploads ls nomad admin uploads rm -- <upload_id> <upload_id> ``` You can also reset the processing (of "stuck") uploads and reprocess: + ```sh nomad admin uploads reset -- <upload_id> nomad admin uploads process -- <upload_id> ``` You can also use the CLI to wipe the whole installation: + ```sh nomad admin reset --i-am-really-sure ``` @@ -82,7 +86,7 @@ cat file_with_ids.txt | xargs nomad admin uploads ls -- Processing includes the conversion of raw files into NOMAD entries. Files are parsed, normalizers are called, the processing results are stored, and the search index -is updated. In certain scenarios (failed processing, [migration](migrate.md#migration-steps), +is updated. In certain scenarios (failed processing, [migration](update.md#migration-steps), changed plugins) might require that admins process certain uploads again. ``` @@ -93,7 +97,7 @@ nomad admin uploads process Each NOMAD entry is represented in NOMAD's search index. Only if an entry is in this index, you can find it via the search interface. Some changes between NOMAD versions -(see also our [migration guide](migrate.md#migration-steps)), might require that +(see also our [update guide](update.md#migration-steps)), might require that you re-index all uploads. ``` diff --git a/docs/howto/oasis/customize.md b/docs/howto/oasis/customize.md deleted file mode 100644 index 079c0ff4506a63256f0048bd1a744d3cf1aa11b2..0000000000000000000000000000000000000000 --- a/docs/howto/oasis/customize.md +++ /dev/null @@ -1,15 +0,0 @@ -# How to customize an Oasis - -!!! warning "Attention" - - This part of the documentation is still work in progress. - -This is an incomplete list of potential customizations. Please read the respective -guides to learn more. - -- Installation specific changes (domain, path-prefix): [How to install an Oasis](install.md) -- [Restricting user access](admin.md#restricting-access-to-your-oasis) -- Write .yaml based [schemas](../customization/basics.md) and [ELNs](../customization/elns.md) -- Learn how to use the [tabular parser](../customization/tabular.md) to manage data from .xls or .csv -- Learn [how to develop plugins](../plugins/plugins.md) that can be installed in an Oasis -- Add specialized [NORTH tools](../manage/north.md) \ No newline at end of file diff --git a/docs/howto/oasis/install.md b/docs/howto/oasis/install.md index dd7ddee78f06254b3d3a0296ca7fd41ad6e8ea25..03919b1da6d13bab3c2aef7e8cb9d14ac3a79879 100644 --- a/docs/howto/oasis/install.md +++ b/docs/howto/oasis/install.md @@ -4,7 +4,7 @@ Originally, the NOMAD Central Repository is a service that runs at the Max-Planck's computing facility in Garching, Germany. However, the NOMAD software is Open-Source, and everybody can run it. Any service that -uses NOMAD software independently is called a *NOMAD Oasis*. A *NOMAD Oasis* does not +uses NOMAD software independently is called a _NOMAD Oasis_. A _NOMAD Oasis_ does not need to be fully isolated. For example, you can publish uploads from your NOMAD Oasis to the central NOMAD installation. @@ -15,40 +15,85 @@ central NOMAD installation. [register your Oasis with FAIRmat](https://www.fairmat-nfdi.eu/fairmat/oasis_registration) and help us to assist you in the future. -## Quick-start +## Basic installation -- Find a linux computer. -- Make sure you have [docker](https://docs.docker.com/engine/install/){:target="_blank"} installed. -Docker nowadays comes with `docker compose` build in. Prior, you needed to -install the stand alone [docker-compose](https://docs.docker.com/compose/install/){:target="_blank"}. -- Download our basic configuration files [nomad-oasis.zip](../../assets/nomad-oasis.zip) -- Run the following commands (skip `chown` on MacOS and Windows computers) +You install NOMAD Oasis from a NOMAD distribution project. We provide a +[template](https://github.com/FAIRmat-NFDI/nomad-distro-template) +for these distribution projects. +!!! warning -```sh -unzip nomad-oasis.zip -cd nomad-oasis -sudo chown -R 1000 .volumes -docker compose pull -docker compose up -d -curl localhost/nomad-oasis/alive -``` + You can install NOMAD Oasis by directly using our template as + described below. However, for a production installation, we recommend to create your own distribution + project based on the template by pressing the + "use this template" button on the top right of the [templat's github page](https://github.com/FAIRmat-NFDI/nomad-distro-template). + + The project contains all necessery config files and will allow you to version your + configuration, build custom images with plugins, and much more. You own distro + project is mandatory for configuring plugins. + + The same documentation will also be awailable on the created projects README.md + with the necessary github URLs already changed for you. + +1. Make sure you have [docker](https://docs.docker.com/engine/install/) installed. + Docker nowadays comes with `docker compose` built in. Prior, you needed to + install the stand-alone [docker-compose](https://docs.docker.com/compose/install/). + +2. Clone the `nomad-distro-template` repository or download the repository as a zip file. + + ```sh + git clone https://github.com/FAIRmat-NFDI/nomad-distro-template.git + cd nomad-distro-template + ``` + + or + + ```sh + curl-L -o nomad-distro-template.zip "https://github.com/FAIRmat-NFDI/nomad-distro-template/archive/main.zip" + unzip nomad-distro-template.zip + cd nomad-distro-template + ``` + +3. _On Linux only,_ recursively change the owner of the `.volumes` directory to the nomad user (1000) -- Open [http://localhost/nomad-oasis](http://localhost/nomad-oasis){:target="_blank"} in your browser. + ```sh + sudo chown -R 1000 .volumes + ``` -To run NORTH (the NOMAD Remote Tools Hub), the `north` container needs to run docker and +4. Pull the images specified in the `docker-compose.yaml` + + Note that the image needs to be public or you need to provide a PAT (see "Important" note above). + + ```sh + docker compose pull + ``` + +5. And run it with docker compose in detached (--detach or -d) mode + + ```sh + docker compose up -d + ``` + +6. Optionally you can now test that NOMAD is running with + + ``` + curl localhost/nomad-oasis/alive + ``` + +7. Finally, open [http://localhost/nomad-oasis](http://localhost/nomad-oasis) in your browser to start using your new NOMAD Oasis. + +To run NORTH (the NOMAD Remote Tools Hub), the `hub` container needs to run docker and the container has to be run under the docker group. You need to replace the default group -id `991` in the `docker-compose.yaml`'s `north` section with your systems docker group id. -Run `id` if you are a docker user, or `getent group | grep docker` to find our your +id `991` in the `docker-compose.yaml`'s `hub` section with your systems docker group id. +Run `id` if you are a docker user, or `getent group | grep docker` to find your systems docker gid. The user id 1000 is used as the nomad user inside all containers. -This is good as a quick test. We strongly recommend to read the following instructions -carefully and adapt the configuration files accordingly. The following might also -include meaningful help, if you run into problems. +Please see the [Jupyter image](#the-jupyter-image) section below for more information on +the jupyter NORTH image being generated in this repository. -## Before you start +## Planning your installation -### Hardware considerations +### Hardware consideration Of course this depends on how much data you need to manage and process. Data storage is the obvious aspect here. NOMAD keeps all files that it manages as they are. The files @@ -78,7 +123,7 @@ RAM and CPU for running tools like jupyter, if you opt to use NOMAD NORTH. ### Sharing data through log transfer and data privacy notice -NOMAD includes a *log transfer* functions. When enabled this it automatically collects +NOMAD includes a _log transfer_ functions. When enabled this it automatically collects and transfers non-personalized logging data to us. Currently, this functionality is experimental and requires opt-in. However, in upcoming versions of NOMAD Oasis, we might change to out-out. @@ -90,9 +135,9 @@ contains any uploaded data. All data is in an aggregated and anonymized form. The data is solely used by the NOMAD developers and FAIRmat, including but not limited to: -* Analyzing and monitoring system performance to identify and resolve issues. -* Improving our NOMAD software based on usage patterns. -* Generating aggregated and anonymized reports. +- Analyzing and monitoring system performance to identify and resolve issues. +- Improving our NOMAD software based on usage patterns. +- Generating aggregated and anonymized reports. We do not share any collected data with any third parties. @@ -121,42 +166,23 @@ and generally recommend to use the central system. But in principle, you can also run your own user management. See the section on [your own user management](#provide-and-connect-your-own-user-management). -## Docker and docker compose - -We recommend the installation via docker and docker-compose. It is the most documented, simplest, -easiest to update, and generally the most frequently chosen option. - -### Pre-requisites - -NOMAD software is distributed as a set of docker containers and there are also other services required that can be run with docker. -Further, we use docker-compose to setup all necessary containers in the simplest way possible. - -You will need a single computer, with **docker** and **docker-compose** installed. Refer -to the official [docker](https://docs.docker.com/engine/install/){:target="_blank"} (and [docker-compose](https://docs.docker.com/compose/install/){:target="_blank"}) -documentation for installation instructions. Newer version of docker have a re-implementation -of docker-compose integrated as the `docker compose` sub-command. This should be fully -compatible and you might chose to can replace `docker compose` with `docker-compose` in this tutorial. +## Configuration files -The following will run all necessary services with docker. These comprise: a **mongo** -database, an **elasticsearch**, a **rabbitmq** distributed task queue, the NOMAD **app**, -NOMAD **worker**, and NOMAD **gui**. In this [architecture documentation](../../explanation/architecture.md), -you will learn what each service does and why it is necessary. +The [`nomad-distro-template`](https://github.com/FAIRmat-NFDI/nomad-distro-template) +provides all the neccessary configuration files. We strongly recommend to create your own distribution +project based on the template. This will allow you to version your configuration, build custom +images with plugins, and much more. -### Configuration - -All docker containers are configured via docker-compose and the respective `docker-compose.yaml` file. -Further, we will need to mount some configuration files to configure the NOMAD services within their respective containers. - -There are three files to configure: +In this section, you can learn about settings that you might need to change. The config files are: - `docker-compose.yaml` - `configs/nomad.yaml` - `configs/nginx.conf` -In this example, we have all files in the same directory (the directory we are also working in). -You can download minimal example files [here](../../assets/nomad-oasis.zip). +All docker containers are configured via docker-compose and the respective `docker-compose.yaml` file. +The other files are mounted into the docker containers. -#### docker-compose.yaml +### docker-compose.yaml The most basic `docker-compose.yaml` to run an OASIS looks like this: @@ -167,25 +193,25 @@ The most basic `docker-compose.yaml` to run an OASIS looks like this: Changes necessary: - The group in the value of the hub's user parameter needs to match the docker group -on the host. This should ensure that the user which runs the hub, has the rights to access the host's docker. + on the host. This should ensure that the user which runs the hub, has the rights to access the host's docker. - On Windows or MacOS computers you have to run the `app` and `worker` container without `user: '1000:1000'` and the `north` container with `user: root`. A few things to notice: - The app, worker, and north service use the NOMAD docker image. Here we use the `latest` tag, which -gives you the latest *beta* version of NOMAD. You might want to change this to `stable`, -a version tag (format is `vX.X.X`, you find all releases [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/tags){:target="_blank"}), or a specific [branch tag](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/branches){:target="_blank"}. + gives you the latest _beta_ version of NOMAD. You might want to change this to `stable`, + a version tag (format is `vX.X.X`, you find all releases [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/tags){:target="\_blank"}), or a specific [branch tag](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/branches){:target="\_blank"}. - All services use docker volumes for storage. This could be changed to host mounts. - It mounts two configuration files that need to be provided (see below): `nomad.yaml`, `nginx.conf`. - The only exposed port is `80` (proxy service). This could be changed to a desired port if necessary. -- The NOMAD images are pulled from our gitlab at MPCDF, the other services use images from a public registry (*dockerhub*). +- The NOMAD images are pulled from our gitlab at MPCDF, the other services use images from a public registry (_dockerhub_). - All containers will be named `nomad_oasis_*`. These names can be used later to reference the container with the `docker` cmd. - The services are setup to restart `always`, you might want to change this to `no` while debugging errors to prevent indefinite restarts. - Make sure that the `PWD` environment variable is set. NORTH needs to create bind mounts that require absolute paths and we need to pass the current working directory to the configuration from the PWD variable (see hub service in the `docker-compose.yaml`). - The `north` service needs to run docker containers. We have to use the systems docker group as a group. You might need to replace `991` with your -systems docker group id. + systems docker group id. -#### nomad.yaml +### nomad.yaml NOMAD app and worker read a `nomad.yaml` for configuration. @@ -196,22 +222,22 @@ NOMAD app and worker read a `nomad.yaml` for configuration. You should change the following: - Replace `localhost` with the hostname of your server. I user-management will redirect your -users back to this host. Make sure this is the hostname, your users can use. + users back to this host. Make sure this is the hostname, your users can use. - Replace `deployment`, `deployment_url`, and `maintainer_email` with representative values. -The `deployment_url` should be the url to the deployment's api (should end with `/api`). -- To enable the *log transfer* set `logtransfer.enable: true` ([data privacy notice above](#sharing-data-through-log-transfer-and-data-privacy-notice)). + The `deployment_url` should be the url to the deployment's api (should end with `/api`). +- To enable the _log transfer_ set `logtransfer.enable: true` ([data privacy notice above](#sharing-data-through-log-transfer-and-data-privacy-notice)). - You can change `api_base_path` to run NOMAD under a different path prefix. - You should generate your own `north.jupyterhub_crypt_key`. You can generate one -with `openssl rand -hex 32`. + with `openssl rand -hex 32`. - On Windows or MacOS, you have to add `hub_connect_ip: 'host.docker.internal'` to the `north` section. A few things to notice: - Under `mongo` and `elastic` you can configure database and index names. This might -be useful, if you need to run multiple NOMADs with the same databases. + be useful, if you need to run multiple NOMADs with the same databases. - All managed files are stored under `.volumes` of the current directory. -#### nginx.conf +### nginx.conf The GUI container serves as a proxy that forwards requests to the app container. The proxy is an nginx server and needs a configuration similar to this: @@ -227,11 +253,12 @@ A few things to notice: - `client_max_body_size` sets a limit to the possible upload size. You can add an additional reverse proxy in front or modify the nginx in the docker-compose.yaml -to [support https](http://nginx.org/en/docs/http/configuring_https_servers.html){:target="_blank"}. +to [support https](http://nginx.org/en/docs/http/configuring_https_servers.html){:target="\_blank"}. If you operate the GUI container behind another proxy, keep in mind that your proxy should not buffer requests/responses to allow streaming of large requests/responses for `api/v1/uploads` and `api/v1/.*/download`. An nginx reverse proxy location on an additional reverse proxy, could have these directives to ensure the correct http headers and allows the download and upload of large files: + ```nginx client_max_body_size 35g; proxy_set_header Host $host; @@ -243,22 +270,43 @@ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://<your-oasis-host>/nomad-oasis; ``` -### Running NOMAD +## Plugins + +[Plugins](../plugins/plugins.md) allow the customization of a NOMAD deployment in terms of +which search apps, normalizers, parsers and schema packages are available. In order for these +customization to be activated, they have to be configured and installed into an Oasis. +The basic template comes with a core set of plugins. If you want to configure +your own set of plugins, using the template and creating your own distro project +is mandatory. + +Plugins are configured via the `pyproject.toml` file. Based on this file +the distro project CI pipeline creates the NOMAD docker image that is used by your installation. +Only plugins configured in the `pyproject.toml` files, will be installed into the docker +image and only those plugins installed in the used docker image are available in your +Oasis. + +Please refer to the [template README](https://github.com/FAIRmat-NFDI/nomad-distro-template?tab=readme-ov-file#adding-a-plugin) +to learn how to add your own plugins. + +## Starting and stopping NOMAD services If you prepared the above files, simply use the usual `docker compose` commands to start everything. To make sure you have the latest docker images for everything, run this first: + ```sh docker compose pull ``` In the beginning and to simplify debugging, it is recommended to start the services separately: + ```sh docker compose up -d mongo elastic rabbitmq docker compose up app worker gui ``` -The `-d` option runs container in the background as *daemons*. Later you can run all at once: +The `-d` option runs container in the background as _daemons_. Later you can run all at once: + ```sh docker compose up -d ``` @@ -267,18 +315,21 @@ Running all services also contains NORTH. When you use a tool in NORTH for the f your docker needs to pull the image that contains this tool. Be aware that this might take longer than timeouts allow and starting a tool for the very first time might fail. -You can also use docker to stop and remove faulty containers that run as *daemons*: +You can also use docker to stop and remove faulty containers that run as _daemons_: + ```sh docker stop nomad_oasis_app docker rm nomad_oasis_app ``` You can wait for the start-up with curl using the apps `alive` "endpoint": + ```sh curl http://<your host>/nomad-oasis/alive ``` If everything works, the gui should be available under: + ```none http://<your host>/nomad-oasis/gui/ ``` @@ -287,6 +338,7 @@ If you run into problems, use the dev-tools of your browser to check the javascr or monitor the network traffic for HTTP 500/400/404/401 responses. To see if at least the api works, check + ```none http://<your host>/nomad-oasis/alive http://<your host>/nomad-oasis/api/info @@ -309,9 +361,9 @@ If you want to report problems with your OASIS. Please provide the logs for - nomad_oasis_worker - nomad_oasis_gui -### Provide and connect your own user management +## Provide and connect your own user management -NOMAD uses [keycloak](https://www.keycloak.org/){:target="_blank"} for its user management. NOMAD uses +NOMAD uses [keycloak](https://www.keycloak.org/){:target="\_blank"} for its user management. NOMAD uses keycloak in two ways. First, the user authentication uses the OpenID Connect/OAuth interfaces provided by keycloak. Second, NOMAD uses the keycloak realm-management API to get a list of existing users. Keycloak is highly customizable and numerous options to connect keycloak to existing @@ -333,6 +385,7 @@ The download also contains an additional `configs/nomad-realm.json` that allows to create an initial keycloak realm that is configured for NOMAD automatically. First, the `docker-compose.yaml`: + ```yaml --8<-- "ops/docker-compose/nomad-oasis-with-keycloak/docker-compose.yaml" ``` @@ -346,6 +399,7 @@ A few notes: - We mount and use the downloaded `configs/nomad-realm.json` to configure a NOMAD compatible realm on the first startup of keycloak. Second, we add a keycloak location to the nginx config: + ```nginx --8<-- "ops/docker-compose/nomad-oasis-with-keycloak/configs/nginx.conf" ``` @@ -353,9 +407,10 @@ Second, we add a keycloak location to the nginx config: A few notes: - Again, we are using `keycloak` as a path prefix. We configure the headers to allow -keycloak to pick up the rewritten url. + keycloak to pick up the rewritten url. Third, we modify the keycloak configuration in the `nomad.yaml`: + ```yaml --8<-- "ops/docker-compose/nomad-oasis-with-keycloak/configs/nomad.yaml" ``` @@ -363,14 +418,14 @@ Third, we modify the keycloak configuration in the `nomad.yaml`: You should change the following: - There are two urls to configure for keycloak. The `server_url` is used by the nomad -services to directly communicate with keycloak within the docker network. The `public_server_url` -is used by the UI to perform the authentication flow. You need to replace `localhost` -in `public_server_url` with `<yourhost>`. + services to directly communicate with keycloak within the docker network. The `public_server_url` + is used by the UI to perform the authentication flow. You need to replace `localhost` + in `public_server_url` with `<yourhost>`. A few notes: - The particular `admin_user_id` is the Oasis admin user in the provided example realm -configuration. See below. + configuration. See below. If you open `http://<yourhost>/keycloak/auth` in a browser, you can access the admin console. The default user and password are `admin` and `password`. @@ -383,120 +438,27 @@ A few notes on the realm configuration: - Realm and client settings are almost all default keycloak settings. - You should change the password of the admin user in the nomad realm. - The admin user in the nomad realm has the additional `view-users` client role for `realm-management` -assigned. This is important, because NOMAD will use this user to retrieve the list of possible -users for managing co-authors and reviewers on NOMAD uploads. + assigned. This is important, because NOMAD will use this user to retrieve the list of possible + users for managing co-authors and reviewers on NOMAD uploads. - The realm has one client `nomad_public`. This has a basic configuration. You might -want to adapt this to your own policies. In particular you can alter the valid redirect URIs to -your own host. + want to adapt this to your own policies. In particular you can alter the valid redirect URIs to + your own host. - We disabled the https requirement on the default realm for simplicity. You should change -this for a production system. - -## Base Linux (without docker) + this for a production system. -### Pre-requisites +## Further steps -We will run NOMAD from the *nomad-lab* Python package. This package contains all the necessary -code to run the processing, api, and gui. But, it is missing the necessary databases. You might be -able to run NOMAD in user space. +This is an incomplete list of potential things to customize your NOMAD experience. -You will need: +- Learn [how to develop plugins](../plugins/plugins.md) that can be installed in an Oasis +- Write .yaml based [schemas](../customization/basics.md) and [ELNs](../customization/elns.md) +- Learn how to use the [tabular parser](../customization/tabular.md) to manage data from .xls or .csv +- Add specialized [NORTH tools](../manage/north.md) +- [Restricting user access](admin.md#restricting-access-to-your-oasis) -- preferably a linux computer, which Python 3.9, preferable a Python virtual environment -- elasticsearch 7.x, running without users and authentication, preferable on the default settings -- mongodb 5.x, running without users and authentication, preferable on the default settings -- rabbitmq 3.x, running without users and authentication, preferable on the default settings -- nginx -- an empty directory to work in +## Other installation options -### Install the NOMAD Python package - -You should install everything in a virtual environment. For example like this: -```sh -virtualenv -p `which python3` nomadpyenv -source nomadpyenv/bin/activate -``` - -You can simply install the Python package from pypi: -```sh -pip install nomad-lab[all] -``` - -If you need the latest version, you can also download the latest package from our -"beta" installation. -```sh -curl "https://nomad-lab.eu/prod/v1/staging/dist/nomad-lab.tar.gz" -o nomad-lab.tar.gz -pip install nomad-lab.tar.gz[all] -``` - -### nomad.yaml - -The `nomad.yaml` is our central config file. You should write a `nomad.yaml` like this: - -```yaml -client: - url: 'http://<your-host>/nomad-oasis/api' - -services: - api_base_path: '/nomad-oasis' - admin_user_id: '<your admin user id>' - -keycloak: - realm_name: fairdi_nomad_prod - username: '<your admin username>' - password: '<your admin user password>' - oasis: true - -mongo: - db_name: nomad_v0_8 - -elastic: - index_name: nomad_v0_8 -``` - -You need to change the following: - -- Replace `your-host` and admin credentials respectively. -- `api_base_path` defines the path under which the app is run. It needs to be changed, if you use a different base path. - -A few things to notice: - -- Be secretive about your admin credentials; make sure this file is not publicly readable. - -### nginx - -You can generate a suitable `nginx.conf` with the `nomad` command line command that -comes with the *nomad-lab* Python package. If you server other content but NOMAD with -your nginx, you need to incorporate the config accordingly. - -If you have a standard installation of nginx, this might work. Adapt to your set-up: -```sh -nomad admin ops nginx-conf > /etc/nginx/conf.d/default.conf -nginx -t -nginx -s reload -``` - -If you want to run nginx in docker, this might work. Adapt to your set-up: -```sh -nomad admin ops nginx-conf --host host.docker.internal > nginx.conf -docker run --rm -v `pwd`/nginx.conf:/etc/nginx/conf.d/default.conf -p 80:80 nginx:stable nginx -g 'daemon off;' & -``` - -### Running NOMAD - -To run NOMAD, you must run two services. One is the NOMAD app, it serves the API and GUI. -Here is startup script to run the app. This is exactly, what the docker based install would use. It takes the port (`$@`) as a parameter. -```sh ---8<-- "scripts/run.sh" -``` - -The second service is the NOMAD worker, that runs the NOMAD processing. Again this is the startup script. -``` ---8<-- "scripts/run-worker.sh" -``` - -This should give you a working OASIS at `http://<your-host>/<your-path-prefix>`. - -## Kubernetes +### Kubernetes !!! warning "Attention" @@ -525,16 +487,21 @@ Run the Helm chart and install NOMAD: helm update --install nomad nomad/nomad -f values.yaml ``` +### Base Linux (without docker) + +Not recommended. We do not provide official support for this type of installation. It is possible +to run NOMAD without docker. You can infer the necessary steps from the provided `docker-compose.yaml`. + ## Troubleshooting Here are some common problems that may occur in an OASIS installation: - `jwt.exceptions.ImmatureSignatureError: The token is not yet valid (iat)`: - The authentication information from central authentication is contained in a special piece of signed information (JWT) that contains details about the signed in person. This information also contains a timestamp, which indicates a point in time at which the information was issued at, called `iat`. The above error indicates that the server looking at the token thinks that it has not been issued yet. + The authentication information from central authentication is contained in a special piece of signed information (JWT) that contains details about the signed in person. This information also contains a timestamp, which indicates a point in time at which the information was issued at, called `iat`. The above error indicates that the server looking at the token thinks that it has not been issued yet. - The underlying reason is a time difference between the two different servers (the one creating the JWT, and the one that is validating it) as these might very well be different physical machines. To fix this problem, you should ensure that the time on the servers is up to date (e.g. a network port on the server may be closed, preventing it from synchronizing the time). Note that the servers do not need to be on the same timezone, as internally everything is converted to UTC+0. + The underlying reason is a time difference between the two different servers (the one creating the JWT, and the one that is validating it) as these might very well be different physical machines. To fix this problem, you should ensure that the time on the servers is up to date (e.g. a network port on the server may be closed, preventing it from synchronizing the time). Note that the servers do not need to be on the same timezone, as internally everything is converted to UTC+0. - ### NOMAD in networks with restricted Internet access +### NOMAD in networks with restricted Internet access Some network environments do not allow direct Internet connections, and require the use of an outbound proxy. However, NOMAD needs to connect to the central user management or elasticsearch thus requires an active Internet @@ -559,27 +526,27 @@ Since not all used services respect proxy variables, one also has to change the ```yaml hl_lines="7 8" elastic: - restart: unless-stopped - image: elasticsearch:7.17.24 - container_name: nomad_oasis_elastic - environment: - - ES_JAVA_OPTS=-Xms512m -Xmx512m - - ES_JAVA_OPTS=-Djava.net.useSystemProxies=true - - ES_JAVA_OPTS=-Dhttps.proxyHost=<proxy> -Dhttps.proxyPort=port -Dhttps.nonProxyHosts=localhost|127.0.0.1|elastic - - discovery.type=single-node - volumes: - - elastic:/usr/share/elasticsearch/data - healthcheck: - test: - - "CMD" - - "curl" - - "--fail" - - "--silent" - - "http://elastic:9200/_cat/health" - interval: 10s - timeout: 10s - retries: 30 - start_period: 60s + restart: unless-stopped + image: elasticsearch:7.17.24 + container_name: nomad_oasis_elastic + environment: + - ES_JAVA_OPTS=-Xms512m -Xmx512m + - ES_JAVA_OPTS=-Djava.net.useSystemProxies=true + - ES_JAVA_OPTS=-Dhttps.proxyHost=<proxy> -Dhttps.proxyPort=port -Dhttps.nonProxyHosts=localhost|127.0.0.1|elastic + - discovery.type=single-node + volumes: + - elastic:/usr/share/elasticsearch/data + healthcheck: + test: + - "CMD" + - "curl" + - "--fail" + - "--silent" + - "http://elastic:9200/_cat/health" + interval: 10s + timeout: 10s + retries: 30 + start_period: 60s ``` Unfortunately there is no way yet to use the NORTH tools with the central user management, since the jupyterhub spawner does not respect proxy variables. @@ -598,4 +565,16 @@ Furthermore, inbound traffic needs to be enabled for the port used on the `nginx In this case you should make sure this test runs through: [https://docs.docker.com/network/network-tutorial-standalone/](https://docs.docker.com/network/network-tutorial-standalone/) -If not please contact your server provider for help. \ No newline at end of file +If not please contact your server provider for help. + +### Elasticsearch and open files limit + +Even when run in docker elasticsearch might require you to change your systems resource +limits as described in the elasticsearch documentation +[here](https://www.elastic.co/guide/en/elasticsearch/reference/current/setting-system-settings.html). + +You can temporarely change the open files limit like this: + +```sh +sudo ulimit -n 65535 +``` diff --git a/docs/howto/oasis/plugins_install.md b/docs/howto/oasis/plugins_install.md deleted file mode 100644 index 22a936511448dfa7a05453137c02d0c0ea68507b..0000000000000000000000000000000000000000 --- a/docs/howto/oasis/plugins_install.md +++ /dev/null @@ -1,107 +0,0 @@ -# How to install plugins into a NOMAD Oasis - -[Plugins](../plugins/plugins.md) allow the customization of a NOMAD deployment in terms of which apps, normalizers, parsers and schema packages are available. In order for these customization to be activated, they have to be installed into an Oasis. - -Oasis is controlled and run through a `docker-compose.yaml` file, which specifies the different software services and how they interact. Some of these services are using a Docker image that contains the actual NOMAD software. It is in this image where we will need to install any additional plugins with `pip install`. - -The following sections contain some alternatives for achieving this, with the first option being the preferred one. - -## Option 1: Create a new customized NOMAD Oasis distribution with your plugins - -When initially starting to create a customized NOMAD Oasis distribution, it is strongly advised that you create a GitHub repository to persist your work, collaborate with coworkers and also to automate the building and distribution of your custom image. To streamline this process, we have created a [GitHub template repository](https://github.com/FAIRmat-NFDI/nomad-distribution-template) that helps with all of this. It can do the following for you: - -- Plugins are controlled with a `pyproject.toml` file where it is easy to install plugins from PyPI, Git repositories, local files, etc. -- The automatic pipeline will create a new Docker image for your Oasis. This image will also be stored on GitHub servers. -- Initial modifications to the `docker-compose.yaml` are done automatically so you can boot up the software directly. - -To learn more, head over to the [template repository](https://github.com/FAIRmat-NFDI/nomad-distribution-template) and follow the instructions there. - -## Option 2: Only create a customized Docker image - -If you already have an existing NOMAD Oasis setup, or do not wish to use the template, you can also just create a new Docker image which has your plugin installed as a `pip` package. For this approach, you need to create a new `Dockerfile`, which runs the installation step on top of our default image. The basic idea is that your Dockerfile looks something like this: - -```Dockerfile -FROM gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:latest - -# Switch to root user to install packages to the system with pip -USER root - -# Install your plugin here, e.g.: -RUN pip install git+https://<repository_url> - -# Remember to switch back to the 'nomad' user -USER nomad -``` - -Depending on how your plugin code is distributed, you have several options for the actual install steps: - -1. Plugin published in PyPI: - - ```sh - RUN pip install <package_name> - ``` - -2. Plugin code available in GitHub: - - ```sh - RUN pip install git+https://<repository_url> - ``` - -3. Plugin published in MPCDF GitLab registry: - - ```sh - RUN pip install nomad-example-schema-plugin --index-url https://gitlab.mpcdf.mpg.de/api/v4/projects/2187/packages/pypi/simple - ``` - -4. Copy plugin folder from host machine. Note that the folder needs to be in the [Docker build context](https://docs.docker.com/build/building/context/): - - ```sh - COPY <nomad-plugin-folder-name> <nomad-plugin-folder-name> - RUN cd <nomad-plugin-folder-name> && pip install . - ``` - -The customized image can then be built like this: - -``` -docker build -t nomad-with-plugins . -``` - -This will create a new image with the tag `nomad-with-plugins`, which you can use in your `docker-compose.yaml` file: - -```yaml -#image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:latest -image: nomad-with-plugins -``` - -## Option 3 (deprecated): Mount the plugin code directly into the container - -!!! warning "Attention" - This option only works with the old plugin mechanism that is based on `nomad_plugin.yaml` files instead of [Python entry points](https://setuptools.pypa.io/en/latest/userguide/entry_point.html). - -The NOMAD docker image adds the folder `/app/plugins` to the `PYTHONPATH`. This means that you can mount your code into the `/app/plugins` directory via the volumes section of the `app` and `worker` services in your `docker-compose.yaml`. - -For example, you can do this by adding an extension to the `docker-compose.yaml`, e.g. a file called `docker-compose.plugins.yaml`. Assuming you have cloned three plugins into the Oasis folder as `./nomad-schema-plugin-example`, `./nomad-parser-plugin-example` and `./nomad-normalizer-plugin-example`, -your `docker-compose.plugins.yaml` should look like this: - -```yaml -services: - worker: - volumes: - - ./nomad-schema-plugin-example/nomadschemaexample:/app/plugins/nomadschemaexample - - ./nomad-parser-plugin-example/nomadparserexample:/app/plugins/nomadparserexample - - ./nomad-normalizer-plugin-example/nomadparserexample:/app/plugins/nomadparserexample - app: - volumes: - - ./nomad-schema-plugin-example/nomadschemaexample:/app/plugins/nomadschemaexample - - ./nomad-parser-plugin-example/nomadparserexample:/app/plugins/nomadparserexample - - ./nomad-normalizer-plugin-example/nomadparserexample:/app/plugins/nomadparserexample -``` - -You have to tell docker that there are now two compose files. This can be done via the `COMPOSE_FILE` environment variable. This is how you can start the Oasis with the plugins: - -```sh -export COMPOSE_FILE=docker-compose.yaml:docker-compose.plugins.yaml -docker compose up -d -``` - -Read the [Oasis install guide](install.md) for more details. diff --git a/docs/howto/oasis/migrate.md b/docs/howto/oasis/update.md similarity index 87% rename from docs/howto/oasis/migrate.md rename to docs/howto/oasis/update.md index b5302b1f1d9b1cdd76ad1b31ef4edc268169fd48..2ddbed2154971d8856d7e05a555c28201460aa3e 100644 --- a/docs/howto/oasis/migrate.md +++ b/docs/howto/oasis/update.md @@ -1,9 +1,9 @@ -# How to migrate Oasis versions +# How to Update Oasis versions ## Software versions We distribute NOMAD as docker images that are available in our -[public docker registry](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/container_registry/36){:target="_blank"}. +[public docker registry](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/container_registry/36){:target="\_blank"}. The a NOMAD image names looks like this: ``` @@ -21,10 +21,9 @@ Our semantic interpretation of **"minor"** is the following: - there are only additions to programming interfaces and config options - NOMAD will still operate on existing data, but **the structure of newly processed data might change** - minor version might introduce new features that are only available after certain -actions [migration steps](#migration-steps). - -A road-map for major features can be found on our homepage [here](https://nomad-lab.eu/nomad-lab/features.html){:target="_blank"}. You'll find a detailed change log in the source code [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/blob/develop/CHANGELOG.md){:target="_blank"}. + actions [migration steps](#migration-steps). +A road-map for major features can be found on our homepage [here](https://nomad-lab.eu/nomad-lab/features.html){:target="\_blank"}. You'll find a detailed change log in the source code [here](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/blob/develop/CHANGELOG.md){:target="\_blank"}. ## Configuration versions @@ -93,17 +92,17 @@ docker compose up -d ### to 1.2.0 - We introduced the plugin mechanism. There are now more options to control which schemas -and parsers are available in your installation. By default all the existing and shipped -schemas and parsers are enabled. See also [here](customize.md). + and parsers are available in your installation. By default all the existing and shipped + schemas and parsers are enabled. See also [here](install.md#plugins). - We changed the archive file format. [Re-processing](admin.md#re-processing) might yield better performance. - Parsers are now using a different workflow model and the UI now includes a -workflow card on the overview page of entries with workflows for the new model. -[Re-processing](admin.md#re-processing) all data will enable this feature for old data. Any analysis build on -the old workflow model, might not work for new data. + workflow card on the overview page of entries with workflows for the new model. + [Re-processing](admin.md#re-processing) all data will enable this feature for old data. Any analysis build on + the old workflow model, might not work for new data. -- We introduce the *log-transfer* service. This is currently an opt-in feature. +- We introduce the _log-transfer_ service. This is currently an opt-in feature. ### from 0.8.x to 1.x @@ -116,6 +115,7 @@ The overall strategy is to create a new mongo database, copy all information, an reprocess all data for the new version. First, shutdown the OASIS and remove all old containers. + ```sh docker compose stop docker compose rm -f @@ -127,6 +127,7 @@ are different. The default values contain a version number in those names, if yo overwrite those defaults, you should be safe. Make sure you get the latest images and start the OASIS with the new version of NOMAD: + ```sh docker compose pull docker compose up -d @@ -152,4 +153,4 @@ old index and database: ```sh docker exec nomad_oasis_elastic bash -c 'curl -X DELETE http://elastic:9200/nomad_fairdi' docker exec nomad_oasis_mongo bash -c 'mongo nomad_fairdi --eval "printjson(db.dropDatabase())"' -``` \ No newline at end of file +``` diff --git a/docs/howto/overview.md b/docs/howto/overview.md index 59a87edaf5ef820d09ba10f9c85f6fbea1e4dcfa..c8968052d9fe6452abfefdced09bff2676da5558 100644 --- a/docs/howto/overview.md +++ b/docs/howto/overview.md @@ -4,8 +4,8 @@ hide: toc # NOMAD How-to guides - ## Users + These how-to guides target NOMAD users and cover data management, exploration, analysis with NOMAD graphical web-interface and APIs. <div markdown="block" class="home-grid"> @@ -37,6 +37,7 @@ Use NOMAD's functions programmatically and via its APIs. </div> ## Data stewards, administrators, and developers + These how-to guides allow advanced users, NOMAD administrators, data stewards, and developers to customize and operate NOMAD and NOMAD Oasis or contribute to NOMAD's development. <div markdown="block" class="home-grid"> @@ -47,9 +48,8 @@ These how-to guides allow advanced users, NOMAD administrators, data stewards, a Host NOMAD for your lab or institution. - [Install an Oasis](oasis/install.md) -- [Customize an Oasis](oasis/customize.md) -- [Install plugins](oasis/plugins_install.md) -- [Migrate Oasis versions](oasis/migrate.md) +- [Develop plugins](plugins/plugins.md) +- [Update an Oasis](oasis/update.md) - [Administrate and maintain an Oasis](oasis/admin.md) </div> diff --git a/docs/howto/plugins/apis.md b/docs/howto/plugins/apis.md index 50e28d79e51465c97a0770f282add396a9a54147..3f87105fe9b5563b7cc9d0436e640862df360f5d 100644 --- a/docs/howto/plugins/apis.md +++ b/docs/howto/plugins/apis.md @@ -89,7 +89,8 @@ async def root(): Read the official [FastAPI documentation](https://fastapi.tiangolo.com/tutorial/) to learn how to build apps and APIs with FastAPI. -If you run NOMAD with this plugin following our [Oasis installation documentation](../oasis/install.md) and our [plugin installation documentation](../oasis/plugins_install.md), you can curl this API and should receive the message: +If you run NOMAD with this plugin following our [Oasis installation documentation](../oasis/install.md) +you can curl this API and should receive the message: ```sh curl localhost/nomad-oasis/myapi diff --git a/docs/howto/plugins/plugins.md b/docs/howto/plugins/plugins.md index e1662dac247fa89b3a663371d769adb6d1e50893..47be45319fc74afac064283893c0c647e14a59e8 100644 --- a/docs/howto/plugins/plugins.md +++ b/docs/howto/plugins/plugins.md @@ -2,10 +2,12 @@ The main way to customize a NOMAD installation is through the use of **plugins**. A NOMAD plugin is a Python package that an administrator can install into a NOMAD distribution to add custom features. This page contains shows you how to create, develop and publish a NOMAD plugin. For a high-level overview of the plugin mechanism, see the [NOMAD plugin system](../../explanation/plugin_system.md) -page. -See the [FAIRmat-NFDI GitHub organization page](https://github.com/FAIRmat-NFDI) for a list of plugins developed by FAIRmat. You can also see the list of activated plugins and plugin entry points at the bottom of the *Information page* (`about/information`) of any NOMAD installation, for example check out the [central NOMAD installation](https://nomad-lab.eu/prod/v1/gui/about/information). +See the [FAIRmat-NFDI GitHub organization page](https://github.com/FAIRmat-NFDI) for a list of plugins developed by FAIRmat. You can also see the list of activated plugins and plugin entry points at the bottom of the _Information page_ (`about/information`) of any NOMAD installation, for example check out the [central NOMAD installation](https://nomad-lab.eu/prod/v1/gui/about/information). + ## Plugin anatomy !!! tip + We provide a [template repository](https://github.com/FAIRmat-NFDI/nomad-plugin-template) which you can use to create the initial plugin layout for you. Plugin Git repositories should roughly follow this layout: @@ -34,8 +36,8 @@ Plugin Git repositories should roughly follow this layout: We suggest using the following convention for naming the repository name and the plugin package: - - repository name: `nomad-<plugin name>` - - package name: `nomad_<plugin name>` +- repository name: `nomad-<plugin name>` +- package name: `nomad_<plugin name>` In the folder structure you can see that a single plugin can contain multiple types of customizations: apps, parsers, schema packages and normalizers. These are called a **plugin entry points** and you will learn more about them next. @@ -182,6 +184,7 @@ As your plugin matures, you should also think about documenting its usage. We re ## Publishing a plugin !!! warning "Attention" + The standard processes for publishing plugins and using plugins from other developers are still being worked out. The "best" practices mentioned in the following are preliminary. We aim to set up a dedicated plugin registry that allows you to publish your plugin and find plugins from others. ### GitHub repository @@ -194,13 +197,14 @@ pip install git+https://<repository_url> ``` !!! note + If you develop a plugin in the context of [FAIRmat](https://github.com/fairmat-nfdi) or the [NOMAD CoE](https://github.com/nomad-coe), put your plugin repositories in the corresponding GitHub organization. ### PyPI/pip package -You may additionally publish the plugin package in PyPI. Learn from the PyPI documentation how to [create a package for PyPI](https://packaging.python.org/en/latest/tutorials/packaging-projects/){:target="_blank"}. We recommend to use the `pyproject.toml`-based approach. +You may additionally publish the plugin package in PyPI. Learn from the PyPI documentation how to [create a package for PyPI](https://packaging.python.org/en/latest/tutorials/packaging-projects/){:target="\_blank"}. We recommend to use the `pyproject.toml`-based approach. -The PyPI documentation provides further information about how to [publish a package to PyPI](https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives){:target="_blank"}. If you have access to the MPCDF GitLab and NOMAD's presence there, you can also +The PyPI documentation provides further information about how to [publish a package to PyPI](https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives){:target="\_blank"}. If you have access to the MPCDF GitLab and NOMAD's presence there, you can also use the `nomad-FAIR` package registry: ``` @@ -213,4 +217,4 @@ twine upload \ ## Installing a plugin -See our documentation on [How to install plugins into a NOMAD Oasis](../oasis/plugins_install.md). \ No newline at end of file +See our documentation on [How to install plugins into a NOMAD Oasis](../oasis/install.md#plugins). diff --git a/docs/reference/annotations.md b/docs/reference/annotations.md index fd12100b882dbb3b9cde630e13d8cc4f4cd6c24d..12d9866734e8b6f501c983c714f974d102748dce 100644 --- a/docs/reference/annotations.md +++ b/docs/reference/annotations.md @@ -30,7 +30,6 @@ This annotation goes in the section that we want to be filled with tabular data, It is used to give a name to the instances that might be created by the parser. If it is not provided, the name of the section itself will be used as name. Many times it is useful because, i. e., one might want to create a bundle of instances of, say, a "Substrate" class, each instance filename not being "Substrate_1", "Substrate_2", etc., but being named after a quantity contained in the class that is, for example, the specific ID of that sample. - ```yaml MySection: more: @@ -38,7 +37,7 @@ MySection: quantities: my_quantity: type: np.float64 - shape: ['*'] + shape: ["*"] description: "my quantity to be filled from the tabular data file" unit: K m_annotations: @@ -50,7 +49,10 @@ MySection: ``` !!! important - The quantity designated as `label_quantity` should not be an array but a integer, float or string, to be set as the name of a file. If an array quantity is chosen, the parser would fall back to the use of the section as name. + + The quantity designated as `label_quantity` should not be an array but a integer, float or string, + to be set as the name of a file. If an array quantity is chosen, the parser would fall back to + the use of the section as name. {{ pydantic_model('nomad.datamodel.metainfo.annotations.SchemaAnnotation', heading='## Schema annotation') }} @@ -64,7 +66,7 @@ A practical example is provided in [How To](../howto/customization/tabular.md#pr ```yaml my_quantity: type: np.float64 - shape: ['*'] + shape: ["*"] description: "my quantity to be filled from the tabular data file" unit: K m_annotations: @@ -83,20 +85,20 @@ One special quantity will be dedicated to host the tabular data file. In the fol ### Available Combinations -|Tutorial ref.|`file_mode`|`mapping_mode`|`sections`|How to ref.| -|---|---|---|---|---| -|1|`current_entry`|`column`|`root`|[HowTo](../howto/customization/tabular.md#1-column-mode-current-entry-parse-to-root)| -|2|`current_entry`|`column`|my path|[HowTo](../howto/customization/tabular.md#2-column-mode-current-entry-parse-to-my-path)| -|<span style="color:red">np1</span>|`current_entry`|`row`|`root`|<span style="color:red">Not possible</span>| -|3|`current_entry`|`row`|my path|[HowTo](../howto/customization/tabular.md#3-row-mode-current-entry-parse-to-my-path)| -|<span style="color:red">np2</span>|`single_new_entry`|`column`|`root`|<span style="color:red">Not possible</span>| -|4|`single_new_entry`|`column`|my path|[HowTo](../howto/customization/tabular.md#4-column-mode-single-new-entry-parse-to-my-path)| -|<span style="color:red">np3</span>|`single_new_entry`|`row`|`root`|<span style="color:red">Not possible</span>| -|5|`single_new_entry`|`row`|my path|[HowTo](../howto/customization/tabular.md#5-row-mode-single-new-entry-parse-to-my-path)| -|<span style="color:red">np4</span>|`multiple_new_entries`|`column`|`root`|<span style="color:red">Not possible</span>| -|<span style="color:red">np5</span>|`multiple_new_entries`|`column`|my path|<span style="color:red">Not possible</span>| -|6|`multiple_new_entries`|`row`|`root`|[HowTo](../howto/customization/tabular.md#6-row-mode-multiple-new-entries-parse-to-root)| -|7|`multiple_new_entries`|`row`|my path|[HowTo](../howto/customization/tabular.md#7-row-mode-multiple-new-entries-parse-to-my-path)| +| Tutorial ref. | `file_mode` | `mapping_mode` | `sections` | How to ref. | +| ---------------------------------- | ---------------------- | -------------- | ---------- | ------------------------------------------------------------------------------------------- | +| 1 | `current_entry` | `column` | `root` | [HowTo](../howto/customization/tabular.md#1-column-mode-current-entry-parse-to-root) | +| 2 | `current_entry` | `column` | my path | [HowTo](../howto/customization/tabular.md#2-column-mode-current-entry-parse-to-my-path) | +| <span style="color:red">np1</span> | `current_entry` | `row` | `root` | <span style="color:red">Not possible</span> | +| 3 | `current_entry` | `row` | my path | [HowTo](../howto/customization/tabular.md#3-row-mode-current-entry-parse-to-my-path) | +| <span style="color:red">np2</span> | `single_new_entry` | `column` | `root` | <span style="color:red">Not possible</span> | +| 4 | `single_new_entry` | `column` | my path | [HowTo](../howto/customization/tabular.md#4-column-mode-single-new-entry-parse-to-my-path) | +| <span style="color:red">np3</span> | `single_new_entry` | `row` | `root` | <span style="color:red">Not possible</span> | +| 5 | `single_new_entry` | `row` | my path | [HowTo](../howto/customization/tabular.md#5-row-mode-single-new-entry-parse-to-my-path) | +| <span style="color:red">np4</span> | `multiple_new_entries` | `column` | `root` | <span style="color:red">Not possible</span> | +| <span style="color:red">np5</span> | `multiple_new_entries` | `column` | my path | <span style="color:red">Not possible</span> | +| 6 | `multiple_new_entries` | `row` | `root` | [HowTo](../howto/customization/tabular.md#6-row-mode-multiple-new-entries-parse-to-root) | +| 7 | `multiple_new_entries` | `row` | my path | [HowTo](../howto/customization/tabular.md#7-row-mode-multiple-new-entries-parse-to-my-path) | ```yaml data_file: @@ -105,12 +107,12 @@ data_file: m_annotations: tabular_parser: parsing_options: - comment: '#' + comment: "#" mapping_options: - - mapping_mode: column - file_mode: single_new_entry - sections: - - my_section/my_quantity + - mapping_mode: column + file_mode: single_new_entry + sections: + - my_section/my_quantity ``` <!-- The available options are: @@ -120,8 +122,8 @@ data_file: |`parsing_options`|group of options|some pandas `Dataframe` options.| |`mapping_options`|list of groups of options|they allow to choose among all the possible modes of parsing data from the spreadsheet file to the NOMAD archive file. Each group of options can be repeated in a list. | --> - ## Plot + The PlotSection base section serves as an additional functionality to your sections. This base section is designed to simplify the process of creating various types of plots, making it easy to use Plotly Express, Plotly Subplot, and the general Plotly graph objects. @@ -200,6 +202,7 @@ and additionally utilizing different flavours of plot annotations. The different {{ pydantic_model('nomad.datamodel.metainfo.annotations.PlotlySubplotsAnnotation', heading='### PlotlySubplotsAnnotation') }} ### plot annotations in python + For simple plots in Python schema one could use the annotations without normalizer: ```python @@ -286,8 +289,9 @@ class CustomSection(PlotSection, EntryData): {{ pydantic_model('nomad.datamodel.metainfo.annotations.PlotAnnotation', heading='### PlotAnnotation (Deprecated)') }} ## H5Web + The H5WebAnnotation provides a way to control how H5Web renders visualization for -[HDF5Dataset](../howto/customization/hdf5.html#hdf5dataset) quantities. The annotation values are written to the +[HDF5Dataset](../howto/customization/hdf5.md#hdf5dataset) quantities. The annotation values are written to the corresponding HDF5 object attributes and are subsequently read by H5Web. Usage: @@ -296,7 +300,9 @@ Usage: - Add addictional annotations to trigger the H5Web visualizer to the section containing the Quantity, and, optionally, to its parent sections. !!! note - If an EntryData class contain the `a_h5web` annotation, the H5Web plot is shown in the [entry overview page](../examples/computational_data/uploading.html#entries-overview-page). + + If an EntryData class contain the `a_h5web` annotation, the H5Web plot is shown in the + [entry overview page](../examples/computational_data/uploading.md#entries-overview-page). An example of use of H5WebAnnotation in Python schemas: @@ -347,10 +353,11 @@ class B(EntryData): ``` In this example, an H5Web view of the variables `x`, `y`, and `y_err` is displayed in the page of the subsection `A`. -The plot of variables `x_value` and `y_value` is also displayed; as the `B` class is an `EntryData`, the plot is shown in the [entry overview page](../examples/computational_data/uploading.html#entries-overview-page). Additionally, alongside with the plot of the class `B`, the overview page presents also the plot contained in the subsection `A`, due to the `paths` attribute. The `paths` attribute allows to show in a parent section or subsection the plots originally contained in children subsections. Parents and children refer here to **composed** object. +The plot of variables `x_value` and `y_value` is also displayed; as the `B` class is an `EntryData`, the plot is shown in the [entry overview page](../examples/computational_data/uploading.md#entries-overview-page). Additionally, alongside with the plot of the class `B`, the overview page presents also the plot contained in the subsection `A`, due to the `paths` attribute. The `paths` attribute allows to show in a parent section or subsection the plots originally contained in children subsections. Parents and children refer here to **composed** object. !!! note - The `paths` variable points in the example above to a [repeated subsection](../howto/plugins/schema_packages.html#schemapackage-class), hence the path provided includes a serial number pointing to the subsection object to be displayed in the entry overview page. To show in the overview page a non-repeatable subsection, no serial number is required in the path. + + The `paths` variable points in the example above to a [repeated subsection](../howto/plugins/schema_packages.md#schemapackage-class), hence the path provided includes a serial number pointing to the subsection object to be displayed in the entry overview page. To show in the overview page a non-repeatable subsection, no serial number is required in the path. H5Web implements visualization features through the attributes shown above, they can be attached to datasets and groups of an HDF5 file. The conventions for the attributes are rooted in the NeXus language and more explanations can be found in the [NXData documentation page](https://manual.nexusformat.org/classes/base_classes/NXdata.html) and in the [Associating plottable data documentation page](https://manual.nexusformat.org/datarules.html#design-findplottable-niac2014). @@ -358,4 +365,3 @@ The conventions for the attributes are rooted in the NeXus language and more exp ### H5WebAnnotation {{ pydantic_model('nomad.datamodel.metainfo.annotations.H5WebAnnotation', heading = '') }} - diff --git a/mkdocs.yml b/mkdocs.yml index d81c0d8ca55110b30a25aa388c4403b9f85d0ed1..4fa2bc73058f0316339b91ae8a986de7f7aa2428 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -29,9 +29,7 @@ nav: - Transform data: howto/programmatic/json_transformer.md - NOMAD Oasis: - Install an Oasis: howto/oasis/install.md - - Customize an Oasis: howto/oasis/customize.md - - Install plugins: howto/oasis/plugins_install.md - - Migrate Oasis versions: howto/oasis/migrate.md + - Update an Oasis: howto/oasis/update.md - Perform admin tasks: howto/oasis/admin.md - Plugins: - Get started with plugins: howto/plugins/plugins.md