diff --git a/docs/explanation/plugin_system.md b/docs/explanation/plugin_system.md
new file mode 100644
index 0000000000000000000000000000000000000000..d34f43db2ce1d60f43bd70a05b7ed82f551df983
--- /dev/null
+++ b/docs/explanation/plugin_system.md
@@ -0,0 +1,102 @@
+# NOMAD plugin system
+
+## Introduction
+
+NOMAD is used by many research communities with their specific data, workflows, and analysis tools. NOMAD plugins are key
+to adopt NOMAD to these heterogeneous environments.
+You can think of plugins as “add-ons†that provide additional capabilities.
+Each plugin is a small independant software project that integrates with the core NOMAD and provides features without modifictions to the core NOMAD itself.
+Some key advantages of using plugins:
+
+- **Modularity**: You can pick and choose which features or functions to add, rather than having everything baked into the core NOMAD.
+
+- **Customizability**: Users can add their own plugins to address specific use cases, without changing the official NOMAD software.
+
+- **Easy updates**: If a feature needs to be updated or improved, it can be done at the plugin level, without having to release a new NOMAD version.
+
+- **Collaboration**: Since plugins are independent, multiple developers can work on different features in parallel and with different release cycles without interfering with each other.
+
+## Architecture
+
+There are three core components to the plugin system:
+
+- **Distributions** define lists of plugins and their version. A distribution is a small
+ Git and Python project that maintains a list of plugin dependencies in its `pyproject.toml`.
+- **Plugins** are Git and Python projects that contain one or many _entry points_.
+ We provide a [template repository](https://github.com/FAIRmat-NFDI/nomad-plugin-template)
+ for a quick start into plugin development.
+- **Entry points** are individual contributions (e.g. parsers, schemas, or apps)
+ which are defined using a feature of Python called [_entry points_](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
+
+<figure markdown style="width: 100%">
+ ``` mermaid
+ %%{init:{'flowchart':{'nodeSpacing': 25, 'subGraphTitleMargin': {'top': 5, 'bottom': 10}, 'padding': 10}}}%%
+ graph LR
+ subgraph NOMAD Distribution
+ subgraph NOMAD Plugin C
+ ro7(Entry point: Schema 1)
+ ro8(Entry point: Schema 2)
+ end
+ subgraph NOMAD Plugin B
+ ro4(Entry point: Schema)
+ ro5(Entry point: App 1)
+ ro6(Entry point: App 2)
+ end
+ subgraph NOMAD Plugin A
+ ro1(Entry point: Schema)
+ ro2(Entry point: Parser 1)
+ ro3(Entry point: Parser 2)
+ end
+ end
+ ```
+ <figcaption>Relation between NOMAD distributions, plugins and entry points.</figcaption>
+</figure>
+
+This architecture allows plugin developers to freely choose a suitable granularity for their use case: they may create a single plugin package that contains everything that e.g. a certain lab needs: schemas, parsers and apps. Alternatively they may also develop multiple plugins, each containing a single entry point. Reduction in package scope can help in developing different parts indepedently and also allows plugin users to choose only the parts that they need.
+
+## Entry point discovery
+
+Entry points are like pre-defined connectors or hooks that allow the main system to recognize and load the code from plugins without needing to hard-code them directly into the platform. This mechanism enables the automatic discovery of plugin code. Plugin entry points represent different types of customizations that can be added to a NOMAD installation. The following plugin entry point types are currently supported:
+
+- [Apps](../howto/plugins/apps.md)
+- [Example uploads](../howto/plugins/example_uploads.md)
+- [Normalizers](../howto/plugins/parsers.md)
+- [Parsers](../howto/plugins/parsers.md)
+- [Schema packages](../howto/plugins/schema_packages.md)
+
+## Loading plugins
+
+Entry points contain **configuration** (the entry point), but also a separate **resource** (the implementation). This split enables lazy-loading: the configuration can be loaded immediately, while the resource is loaded later when/if it is required. This can significantly improve startup times, as long as all time-consuming initializations are performed only when loading the resource. This split also helps to avoid cyclical imports between the plugin code and the `nomad-lab` package. The following diagram illustrates how NOMAD interacts with the entry points in a plugin:
+
+<figure markdown style="width: 100%">
+ ``` mermaid
+ %%{init:{'sequence':{'mirrorActors': false}}}%%
+ sequenceDiagram
+ autonumber
+ rect
+ NOMAD->>Plugin: Request entry points matching the nomad.plugin group
+ Plugin-->>NOMAD: Return all entry point configurations
+ end
+ Note over NOMAD: Other tasks
+ rect
+ NOMAD->>Plugin: Request a specific entry point resource
+ opt
+ Plugin->>NOMAD: Request configuration overrides from nomad.yaml
+ NOMAD-->>Plugin: Return final configuration for entry point
+ end
+ Plugin-->>NOMAD: Return fully initialized entry point resource
+ end
+ ```
+ <figcaption>NOMAD interaction with a plugin.</figcaption>
+</figure>
+
+1. When NOMAD starts, it scans for plugin entry points defined under the `nomad.plugin` group in all of the Python packages that have been installed.
+2. The plugin returns all entry points that it has registered in `pyproject.toml` under the `nomad.plugin` group. This only loads the configuration, but does not yet load the resource, i.e. main Python implementation.
+3. When NOMAD needs to load the actual resource for an entry point (e.g. a parser), loads it by using the configuration instance.
+4. When the resource is being loaded, the entry point may ask for any configuration overrides that may have been set in `nomad.yaml`.
+5. NOMAD will return the final validated configuration that contains the default values and possible overrides.
+6. The plugin loads and returns the resource using the final configuration. This typically involves creating an instance of a specific class, e.g. `Parser` in the case of parser entry points.
+
+## Learn how to write plugins
+
+You can learn more about plugin development in the [Get started with plugins](../howto/plugins/plugins.md) -page.
diff --git a/docs/howto/plugins/plugins.md b/docs/howto/plugins/plugins.md
index 41b23bae1ac1cf1893155600f99824ce5a2bdec6..3d70f6f1c1fc7a43acc9fa4a4c2ae0bb251a05d1 100644
--- a/docs/howto/plugins/plugins.md
+++ b/docs/howto/plugins/plugins.md
@@ -1,6 +1,6 @@
# Get started with plugins
-The main way to customize a NOMAD installation is through the use of **plugins**. A NOMAD Plugin is a Git repository that contains a Python package that an administrator can install into a NOMAD deployment to add custom features. This page contains the basics of how to create, develop and publish a NOMAD Plugin.
+The main way to customize a NOMAD installation is through the use of **plugins**. A NOMAD plugin is a Python package that an administrator can install into a NOMAD distribution to add custom features. This page contains shows you how to create, develop and publish a NOMAD plugin. For a high-level overview of the plugin mechanism, see the [NOMAD plugin system](../../explanation/plugin_system.md) -page.
## Plugin anatomy
@@ -40,17 +40,9 @@ In the folder structure you can see that a single plugin can contain multiple ty
## Plugin entry points
-Plugin entry points represent different types of customizations that can be added to a NOMAD installation. The following plugin entry point types are currently supported:
+Plugin entry points represent different types of customizations that can be added to a NOMAD installation. Entry points contain **configuration**, but also a **resource**, which lives in a separate Python module. This split enables lazy-loading: the configuration can be loaded immediately, while the resource is loaded later when/if it is required. This can significantly improve startup times, as long as all time-consuming initializations are performed only when loading the resource. This split also helps to avoid cyclical imports between the plugin code and the `nomad-lab` package.
- - [Apps](./apps.md)
- - [Example uploads](./example_uploads.md)
- - [Normalizers](./parsers.md)
- - [Parsers](./parsers.md)
- - [Schema packages](./schema_packages.md)
-
-Entry points contain **configuration**, but also a separate **resource**, which should live in a separate Python module. This split enables lazy-loading: the configuration can be loaded immediately, while the resource is loaded later when/if it is required. This can significantly improve startup times, as long as all time-consuming initializations are performed only when loading the resource. This split also helps to avoid cyclical imports between the plugin code and the `nomad-lab` package.
-
-For example the entry point instance for a parser is contained in `.../parsers/__init__.py` and it contains e.g. the name, version and any additional entry point-specific parameters that control its behaviour. The entry point has a `load` method than can be called lazily to return the resource, which is a `Parser` instance defined in `.../parsers/myparser.py`.
+For example the entry point configuration for a parser is contained in `.../parsers/__init__.py` and it contains e.g. the name, version and any additional entry point-specific parameters that control its behaviour. The entry point has a `load` method than can be called lazily to return the resource, which is a `Parser` instance defined in `.../parsers/myparser.py`.
In `pyproject.toml` you can expose plugin entry points for automatic discovery. E.g. to expose an app and a package, you would add the following to `pyproject.toml`:
@@ -64,28 +56,11 @@ Here it is important to use the `nomad.plugin` group name in the `project.entry-
You can read more about how to write different types of entry points in their dedicated documentation pages or learn more about the [Python entry point mechanism](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
-### Controlling loading of plugin entry points
-
-By default, plugin entry points are automatically loaded, and as an administrator you only need to install the Python package. You can, however, control which entry points to load by explicitly including/excluding them in your `nomad.yaml`. For example, if a plugin has the following `pyproject.toml`:
+### Plugin configuration
-```toml
-[project.entry-points.'nomad.plugin']
-myparser = "nomad_example.parsers:myparser"
-```
+The plugin entry point configuration is an instance of a [`pydantic`](https://docs.pydantic.dev/1.10/) model. This base model may already contain entry point-specific fields (such as the file extensions that a parser plugin will match) but it is also possible to extend this model to define additional fields that control your plugin behaviour.
-You could disable the parser entry point in your `nomad.yaml` with:
-
-```yaml
-plugins:
- entry_points:
- exclude: ["nomad_plugin.parsers:myparser"]
-```
-
-### Extending and using the entry point
-
-The plugin entry point is an instance of a [`pydantic`](https://docs.pydantic.dev/1.10/) model. This base model may already contain entry point-specific fields (such as the file extensions that a parser plugin will match) but it is also possible to extend this model to define additional fields that control your plugin behaviour.
-
-To specify new configuration options, you can add new `pydantic` fields to the subclass. For example, if we wanted to add a new configuration option for a parser, we could do the following:
+Here is an example of a new plugin entry point configuration class and instance for a parser, that has a new custom `parameter` configuration added as a `pydantic` `Field`:
```python
from pydantic import Field
@@ -94,9 +69,15 @@ from nomad.config.models.plugins import ParserEntryPoint
class MyParserEntryPoint(ParserEntryPoint):
parameter: int = Field(0, description='Config parameter for this parser.')
+
+myparser = MyParserEntryPoint(
+ name = 'MyParser',
+ description = 'My custom parser.',
+ mainfile_name_re = '.*\.myparser',
+)
```
-where we have defined a new subclass of `ParserEntryPoint` and added a new configuration field `parameter`. The plugin users can then control these settings in their `nomad.yaml` using `plugins.entry_points.options`:
+The plugin entry point behaviour can be controlled in `nomad.yaml` using `plugins.entry_points.options`:
```yaml
plugins:
@@ -108,7 +89,20 @@ plugins:
Note that the model will also validate the values coming from `nomad.yaml`, and you should utilize the validation mechanisms of `pydantic` to provide users with helpful messages about invalid configuration.
-In your code, you can then access the whole entry point by loading it with `config.get_plugin_entry_point`:
+### Plugin resource
+
+The configuration class has a `load` method that returns the entry point resource. This is typically an instance of a class, e.g. `Parser` instance in the case of a parser entry point. Here is an example of a `load` method for a parser:
+
+```python
+class MyParserEntryPoint(ParserEntryPoint):
+
+ def load(self):
+ from nomad_example.parsers.myparser import MyParser
+
+ return MyParser(**self.dict())
+```
+
+Often when loading the resource, you will need access to the final entry point configuration defined in `nomad.yaml`. This way also any overrides to the plugin configuration are correctly taken into account. You can get the final configuration using the `get_plugin_entry_point` function and the plugin name as defined in `pyproject.toml` as an argument:
```python
from nomad.config import config
@@ -117,6 +111,23 @@ configuration = config.get_plugin_entry_point('nomad_example.parsers:myparser')
print(f'The parser parameter is: {configuration.parameter}')
```
+### Controlling loading of plugin entry points
+
+By default, plugin entry points are automatically loaded, and as an administrator you only need to install the Python package. You can, however, control which entry points to load by explicitly including/excluding them in your `nomad.yaml`. For example, if a plugin has the following `pyproject.toml`:
+
+```toml
+[project.entry-points.'nomad.plugin']
+myparser = "nomad_example.parsers:myparser"
+```
+
+You could disable the parser entry point in your `nomad.yaml` with:
+
+```yaml
+plugins:
+ entry_points:
+ exclude: ["nomad_plugin.parsers:myparser"]
+```
+
## Plugin development guidelines
### Linting and formatting
diff --git a/mkdocs.yml b/mkdocs.yml
index 04aede9dfcdccfb3b1f77e1523ffc29650868ae9..13bc1d39cc34960163f093714320c9c8814bfce9 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -3,103 +3,104 @@ site_description: |
The documentation for NOMAD v1.
site_author: The NOMAD Authors
repo_url: https://github.com/nomad-coe/nomad
-edit_uri: ''
+edit_uri: ""
nav:
- Home: index.md
- Tutorial:
- - Navigating to NOMAD: tutorial/nomad_repo.md
- - Uploading and publishing data: tutorial/upload_publish.md
- - Exploring data: tutorial/explore.md
- - Access data via API: tutorial/access_api.md
- - Schemas and plugins: tutorial/custom.md
- - Developing a NOMAD plugin: tutorial/develop_plugin.md
+ - Navigating to NOMAD: tutorial/nomad_repo.md
+ - Uploading and publishing data: tutorial/upload_publish.md
+ - Exploring data: tutorial/explore.md
+ - Access data via API: tutorial/access_api.md
+ - Schemas and plugins: tutorial/custom.md
+ - Developing a NOMAD plugin: tutorial/develop_plugin.md
- How-to guides:
- - Overview: howto/overview.md
- - Manage and find data:
- - Upload and publish data for supported formats: howto/manage/upload.md
- - Use ELNs: howto/manage/eln.md
- - Explore data: howto/manage/explore.md
- - Use NORTH: howto/manage/north.md
- - Programmatic use:
- - Use the API: howto/programmatic/api.md # TODO separate into How-to and Explanation/Reference
- - Download data: howto/programmatic/download.md
- - Publish data using python: howto/programmatic/publish_python.md
- - Install nomad-lab: howto/programmatic/pythonlib.md
- - Access processed data: howto/programmatic/archive_query.md
- - Run a parser: howto/programmatic/local_parsers.md
- - NOMAD Oasis:
- - Install an Oasis: howto/oasis/install.md
- - Customize an Oasis: howto/oasis/customize.md
- - Install plugins: howto/oasis/plugins_install.md
- - Migrate Oasis versions: howto/oasis/migrate.md
- - Perform admin tasks: howto/oasis/admin.md
- - Plugins:
- - Get started with plugins: howto/plugins/plugins.md
- - Write an app: howto/plugins/apps.md
- - Write an example upload: howto/plugins/example_uploads.md
- - Write a normalizer: howto/plugins/normalizers.md
- - Write a parser: howto/plugins/parsers.md
- - Write a schema package: howto/plugins/schema_packages.md
- - Customization:
- - Write a YAML schema package: howto/customization/basics.md
- - Define ELNs: howto/customization/elns.md
- - Use base sections: howto/customization/base_sections.md
- - Parse tabular data: howto/customization/tabular.md
- - Define workflows: howto/customization/workflows.md
- - Work with units: howto/customization/units.md
- - Use HDF5 to handle large quantities: howto/customization/hdf5.md
- - Development:
- - Get started: howto/develop/setup.md
- - Navigate the code: howto/develop/code.md
- - Contribute: howto/develop/contrib.md
- - Extend the search: howto/develop/search.md
- - Migrate to autoformatted code: howto/develop/migrate-to-autoformatter.md
- - Make a release: howto/develop/release.md
+ - Overview: howto/overview.md
+ - Manage and find data:
+ - Upload and publish data for supported formats: howto/manage/upload.md
+ - Use ELNs: howto/manage/eln.md
+ - Explore data: howto/manage/explore.md
+ - Use NORTH: howto/manage/north.md
+ - Programmatic use:
+ - Use the API: howto/programmatic/api.md # TODO separate into How-to and Explanation/Reference
+ - Download data: howto/programmatic/download.md
+ - Publish data using python: howto/programmatic/publish_python.md
+ - Install nomad-lab: howto/programmatic/pythonlib.md
+ - Access processed data: howto/programmatic/archive_query.md
+ - Run a parser: howto/programmatic/local_parsers.md
+ - NOMAD Oasis:
+ - Install an Oasis: howto/oasis/install.md
+ - Customize an Oasis: howto/oasis/customize.md
+ - Install plugins: howto/oasis/plugins_install.md
+ - Migrate Oasis versions: howto/oasis/migrate.md
+ - Perform admin tasks: howto/oasis/admin.md
+ - Plugins:
+ - Get started with plugins: howto/plugins/plugins.md
+ - Write an app: howto/plugins/apps.md
+ - Write an example upload: howto/plugins/example_uploads.md
+ - Write a normalizer: howto/plugins/normalizers.md
+ - Write a parser: howto/plugins/parsers.md
+ - Write a schema package: howto/plugins/schema_packages.md
+ - Customization:
+ - Write a YAML schema package: howto/customization/basics.md
+ - Define ELNs: howto/customization/elns.md
+ - Use base sections: howto/customization/base_sections.md
+ - Parse tabular data: howto/customization/tabular.md
+ - Define workflows: howto/customization/workflows.md
+ - Work with units: howto/customization/units.md
+ - Use HDF5 to handle large quantities: howto/customization/hdf5.md
+ - Development:
+ - Get started: howto/develop/setup.md
+ - Navigate the code: howto/develop/code.md
+ - Contribute: howto/develop/contrib.md
+ - Extend the search: howto/develop/search.md
+ - Migrate to autoformatted code: howto/develop/migrate-to-autoformatter.md
+ - Make a release: howto/develop/release.md
- Domain-specific examples:
- - Overview: examples/overview.md
- - Computational data:
- - Quick start: examples/computational_data/uploading.md
- - Workflows: examples/computational_data/workflows.md
- - MetaInfo: examples/computational_data/metainfo.md
- - Schema plugins: examples/computational_data/schema_plugins.md
- - Parser plugins: examples/computational_data/parser_plugins.md
- - H5MD schema - Howto: examples/computational_data/h5md_howto.md
- - H5MD schema - Explanation: examples/computational_data/h5md_expl.md
- - H5MD - Reference: examples/computational_data/h5md_ref.md
- - Experimental data:
- - NeXus:
- - NeXus semantic file format: examples/experiment_data/nexus.md
- - Guide to parser library pynxtools: examples/experiment_data/pynxtools.md
- - Electron microscopy: examples/experiment_data/em.md
- - Photoemission spectroscopy: examples/experiment_data/mpes.md
- - X-ray photoemission spectroscopy: examples/experiment_data/xps.md
- - Optical spectroscopy: examples/experiment_data/opt.md
- - Atom probe tomography: examples/experiment_data/apm.md
- - Scanning tunneling spectroscopy: examples/experiment_data/sts.md
- # - NOMAD-supported plugins:
- # - Databases for specific research topics:
+ - Overview: examples/overview.md
+ - Computational data:
+ - Quick start: examples/computational_data/uploading.md
+ - Workflows: examples/computational_data/workflows.md
+ - MetaInfo: examples/computational_data/metainfo.md
+ - Schema plugins: examples/computational_data/schema_plugins.md
+ - Parser plugins: examples/computational_data/parser_plugins.md
+ - H5MD schema - Howto: examples/computational_data/h5md_howto.md
+ - H5MD schema - Explanation: examples/computational_data/h5md_expl.md
+ - H5MD - Reference: examples/computational_data/h5md_ref.md
+ - Experimental data:
+ - NeXus:
+ - NeXus semantic file format: examples/experiment_data/nexus.md
+ - Guide to parser library pynxtools: examples/experiment_data/pynxtools.md
+ - Electron microscopy: examples/experiment_data/em.md
+ - Photoemission spectroscopy: examples/experiment_data/mpes.md
+ - X-ray photoemission spectroscopy: examples/experiment_data/xps.md
+ - Optical spectroscopy: examples/experiment_data/opt.md
+ - Atom probe tomography: examples/experiment_data/apm.md
+ - Scanning tunneling spectroscopy: examples/experiment_data/sts.md
+ # - NOMAD-supported plugins:
+ # - Databases for specific research topics:
- Explanation:
- - From files to data: explanation/basics.md
- - Data structure: explanation/data.md
- - Processing: explanation/processing.md
- - Architecture: explanation/architecture.md
- - Federation and Oasis: explanation/oasis.md
+ - From files to data: explanation/basics.md
+ - Data structure: explanation/data.md
+ - Processing: explanation/processing.md
+ - Architecture: explanation/architecture.md
+ - Plugins: explanation/plugin_system.md
+ - Federation and Oasis: explanation/oasis.md
- Reference:
- - reference/config.md
- - reference/annotations.md
- - reference/cli.md
- - reference/plugins.md
- - reference/parsers.md
- - reference/code_guidelines.md
- - reference/glossary.md
- - reference/tutorials.md
+ - reference/config.md
+ - reference/annotations.md
+ - reference/cli.md
+ - reference/plugins.md
+ - reference/parsers.md
+ - reference/code_guidelines.md
+ - reference/glossary.md
+ - reference/tutorials.md
theme:
name: material
palette:
- primary: '#2A4CDF'
- accent: '#008A67'
+ primary: "#2A4CDF"
+ accent: "#008A67"
font:
- text: 'Titillium Web'
+ text: "Titillium Web"
logo: assets/nomad-logo.png
favicon: assets/favicon.png
features:
@@ -138,19 +139,19 @@ extra:
homepage: https://nomad-lab.eu
use_directory_urls: false
plugins:
- - search
- - macros:
- module_name: nomad/mkdocs
- - redirects:
- redirect_maps:
- 'pythonlib.md': 'howto/programmatic/pythonlib.md'
- 'oasis.md': 'howto/oasis/install.md'
- 'develop/gitlab.md': 'howto/develop/contrib.md'
- - glightbox
- # - git-revision-date-localized
- # TODO Fix error in pipeline when this plugin is included
- # (with 'mkdocs-git-revision-date-localized-plugin==1.2.1' in pyproject.toml)
- # see pipeline error here: https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/2473217
+ - search
+ - macros:
+ module_name: nomad/mkdocs
+ - redirects:
+ redirect_maps:
+ "pythonlib.md": "howto/programmatic/pythonlib.md"
+ "oasis.md": "howto/oasis/install.md"
+ "develop/gitlab.md": "howto/develop/contrib.md"
+ - glightbox
+ # - git-revision-date-localized
+ # TODO Fix error in pipeline when this plugin is included
+ # (with 'mkdocs-git-revision-date-localized-plugin==1.2.1' in pyproject.toml)
+ # see pipeline error here: https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/jobs/2473217
extra_css:
- stylesheets/extra.css
extra_javascript: