diff --git a/.dockerignore b/.dockerignore index 75ef39777a2ccb30592d53e72cfba69226578954..ea6c2dbd72f876115c47b902350976cac795dc8e 100644 --- a/.dockerignore +++ b/.dockerignore @@ -36,6 +36,7 @@ gui/public/env.js !gui/tests/nomad.yaml !ops/docker-compose/nomad-oasis/configs/nomad.yaml !ops/docker-compose/nomad-oasis-with-keycloak/configs/nomad.yaml +!ops/docker-compose/nomad-oasis-with-plugins/nomad.yaml !examples/plugins/schema/nomad.yaml !examples/plugins/parser/nomad.yaml diff --git a/Dockerfile b/Dockerfile index 6ed864f47a0b56c28b0f9d61a20a27cd10438071..9f1a56d99067c74817b3d382fa2b9a2e128218f7 100644 --- a/Dockerfile +++ b/Dockerfile @@ -221,4 +221,6 @@ USER nomad EXPOSE 8000 EXPOSE 9000 +ENV PYTHONPATH=/app/plugins + VOLUME /app/.volumes/fs diff --git a/docs/plugins.md b/docs/plugins.md index a0a662973067c8e72710fe41b76b3daf7e2f6f3c..795cabc0054121b6f38e329bcffdf8423aa40ba2 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -1,5 +1,9 @@ Plugins allow you to add Python-based functionality to NOMAD without a custom build -NOMAD image or package. We support different kinds of plugins: +NOMAD image or release. Plugins can be installed at NOMAD start-up time. Therefore, you can +configure each NOMAD (Oasis) with a different custom set of plugins or disable unnecessary +plugins. + +We support different kinds of plugins: - Python **schema**, read also [Python schema documentation](schema/python.md). - **parser**, read also [parser development documentation](develop/parser.md). @@ -10,7 +14,9 @@ NOMAD image or package. We support different kinds of plugins: We provide template projects on GitHub. You can fork these projects and follow the instructions in their `README.md`. These instructions will give you everything you -need to run your plugin as a plugin developer: +need to run and test your plugin as a plugin developer. +The following sections here contain more background information and explain how to +add plugins to a NOMAD installation. - [schema plugin](https://github.com/nomad-coe/nomad-schema-plugin-example) - [parser plugin](https://github.com/nomad-coe/nomad-parser-plugin-example) @@ -53,7 +59,7 @@ for more details: ### Plugin metadata -The file `nomad_plugin.yaml` contains the installation independent plugin metadata: +The file `nomad_plugin.yaml` contains the installation independent *plugin metadata*: ```yaml {{ file_contents('examples/plugins/schema/nomadschemaexample/nomad_plugin.yaml') }} @@ -86,20 +92,76 @@ each `options` entry with `python_package` will be merged with the data in the p Please note that `python_package` is the name of a Python package and not a path to the code. This also means that the package has to be in your `PYTHONPATH` (see below). + +{{pydantic_model('nomad.config.plugins.Schema', heading='### Parser plugin metadata')}} +{{pydantic_model('nomad.config.plugins.Parser', heading='### Schema plugin metadata')}} + Now follow the instructions for one of our examples and try for yourself: - [schema plugin](https://github.com/nomad-coe/nomad-schema-plugin-example) - [parser plugin](https://github.com/nomad-coe/nomad-parser-plugin-example) -## Add a plugin to your NOMAD (Oasis) +## Add a plugin to your NOMAD -To add a plugin, you need to add the *plugin metadata* to `nomad.yaml` and you need +To add a plugin, you need to add the *plugin metadata* to `nomad.yaml` (see above) and you need to add the *plugin code* to the `PYTHONPATH` of your NOMAD. The `nomad.yaml` needs to be edited manually in the usual way. There are several ways to -add *plugin code* to a NOMAD installation: +add *plugin code* to a NOMAD installation. + +### Development setup of NOMAD + +Simply add the plugin directory to the `PYTHONPATH` environment variable. When you start +the application (e.g. `nomad admin run appworker`), Python will find your code when NOMAD +imports the `python_package` given in the `plugins.options` of your `nomad.yaml`. + +### NOMAD Oasis + +The NOMAD docker image adds the folder `/app/plugins` to the `PYTHONPATH`. You simply have +to add the *plugin metadata* to your Oasis' `nomad.yaml` and mount your code into the `/app/plugins` +directory via the volumes section of the `app` and `worker` services in your `docker-compose.yaml`. + +For example, you can do this by adding an extension to the `docker-compose.yaml`, e.g. a file called +`docker-compose.plugins.yaml`. Assuming you cloned the example plugins above into the Oasis folder as +`./nomad-schema-plugin-example` and `./nomad-parser-plugin-example`, +your `docker-compose.plugins.yaml` should look like this: + +```yaml +services: + worker: + volumes: + - ./nomad-schema-plugin-example/nomadschemaexample:/app/plugins/nomadschemaexample + - ./nomad-parser-plugin-example/nomadparserexample:/app/plugins/nomadparserexample + app: + volumes: + - ./nomad-schema-plugin-example/nomadschemaexample:/app/plugins/nomadschemaexample + - ./nomad-parser-plugin-example/nomadparserexample:/app/plugins/nomadparserexample +``` + +You have to tell docker that there are now two compose files. This can be done via the +`COMPOSE_FILE` environment variable. This is how you can start the Oasis with the plugins: + +```sh +export COMPOSE_FILE=docker-compose.yaml:docker-compose.plugins.yaml +docker compose up -d +``` + +Here is a complete Oasis setup [nomad-oasis-with-plugins.zip](assets/nomad-oasis-with-plugins.zip). +Simply download, extract, and start like any other Oasis: + +```sh +unzip nomad-oasis-with-plugins.zip +cd nomad-oasis-with-plugins +sudo chown -R 1000 .volumes +export COMPOSE_FILE=docker-compose.yaml:docker-compose.plugins.yaml +docker compose pull +docker compose up -d +curl localhost/nomad-oasis/alive +``` + +Read the [Oasis documentation](oasis.md) for more details. + +### Other means -- adding it to the `PYTHONPATH`, if you run nomad directly, e.g. as a developer -- mounting plugin sources into NOMAD (Oasis) containers, e.g. as an Oasis admin (coming soon...) - via python packages (coming soon...) - via github projects (coming soon...) diff --git a/ops/docker-compose/nomad-oasis-with-plugins/.gitignore b/ops/docker-compose/nomad-oasis-with-plugins/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..1854ee70bb08645488918b27ff3b03148b947cb4 --- /dev/null +++ b/ops/docker-compose/nomad-oasis-with-plugins/.gitignore @@ -0,0 +1,6 @@ +nomad-schema-plugin-example/ +nomad-parser-plugin-example/ +configs/ +docker-compose.yaml +!nomad.yaml +.volumes/ \ No newline at end of file diff --git a/ops/docker-compose/nomad-oasis-with-plugins/docker-compose.plugins.yaml b/ops/docker-compose/nomad-oasis-with-plugins/docker-compose.plugins.yaml new file mode 100644 index 0000000000000000000000000000000000000000..61e7931c22258307465d2228aba9def13b6b6cb0 --- /dev/null +++ b/ops/docker-compose/nomad-oasis-with-plugins/docker-compose.plugins.yaml @@ -0,0 +1,11 @@ +services: + worker: + image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:1008-plugin-mechanism + volumes: + - ./nomad-schema-plugin-example/nomadschemaexample:/app/plugins/nomadschemaexample + - ./nomad-parser-plugin-example/nomadparserexample:/app/plugins/nomadparserexample + app: + image: gitlab-registry.mpcdf.mpg.de/nomad-lab/nomad-fair:1008-plugin-mechanism + volumes: + - ./nomad-schema-plugin-example/nomadschemaexample:/app/plugins/nomadschemaexample + - ./nomad-parser-plugin-example/nomadparserexample:/app/plugins/nomadparserexample \ No newline at end of file diff --git a/ops/docker-compose/nomad-oasis-with-plugins/nomad.yaml b/ops/docker-compose/nomad-oasis-with-plugins/nomad.yaml new file mode 100644 index 0000000000000000000000000000000000000000..31f7f8946462c67722d5666f877d97c1c1388d48 --- /dev/null +++ b/ops/docker-compose/nomad-oasis-with-plugins/nomad.yaml @@ -0,0 +1,11 @@ + + +plugins: + include: + - 'schemas/example' + - 'parsers/example' + options: + schemas/example: + python_package: nomadschemaexample + parsers/example: + python_package: nomadparserexample \ No newline at end of file diff --git a/scripts/generate_docs_artifacts.sh b/scripts/generate_docs_artifacts.sh index 2783a8976a2d4c1016b5dc4f324331dd77c4e593..c75787731e164217a746ec50ff1c6ae5b553edca 100755 --- a/scripts/generate_docs_artifacts.sh +++ b/scripts/generate_docs_artifacts.sh @@ -10,3 +10,10 @@ cd $project_dir/ops/docker-compose rm -rf $project_dir/docs/assets/nomad-oasis* zip -r $project_dir/docs/assets/nomad-oasis.zip nomad-oasis -x "**/.gitignore" zip -r $project_dir/docs/assets/nomad-oasis-with-keycloak.zip nomad-oasis-with-keycloak -x "**/.gitignore" + +rm -rf nomad-oasis-with-plugins/.volumes nomad-oasis-with-plugins/configs nomad-oasis-with-plugins/docker-compose.yaml nomad-oasis-with-plugins/nomad-* +cp -r nomad-oasis/configs nomad-oasis/docker-compose.yaml nomad-oasis/.volumes nomad-oasis-with-plugins/ +git clone https://github.com/nomad-coe/nomad-schema-plugin-example.git ./nomad-oasis-with-plugins/nomad-schema-plugin-example +git clone https://github.com/nomad-coe/nomad-parser-plugin-example.git ./nomad-oasis-with-plugins/nomad-parser-plugin-example +cat nomad-oasis-with-plugins/nomad.yaml >> nomad-oasis-with-plugins/configs/nomad.yaml +zip -r $project_dir/docs/assets/nomad-oasis-with-plugins.zip nomad-oasis-with-plugins -x "**/.gitignore" -x "nomad-oasis-with-plugins/nomad.yaml" \ No newline at end of file