diff --git a/docs/explanation/plugin_system.md b/docs/explanation/plugin_system.md
index d34f43db2ce1d60f43bd70a05b7ed82f551df983..3584be37771c7fb0ecf0b1a248fdfc7235934549 100644
--- a/docs/explanation/plugin_system.md
+++ b/docs/explanation/plugin_system.md
@@ -21,7 +21,8 @@ Some key advantages of using plugins:
There are three core components to the plugin system:
- **Distributions** define lists of plugins and their version. A distribution is a small
- Git and Python project that maintains a list of plugin dependencies in its `pyproject.toml`.
+ Git and Python project that maintains a list of plugin dependencies in its `pyproject.toml`. We provide a [template repository](https://github.com/FAIRmat-NFDI/nomad-distro-template)
+ for a quick start into creating distributions.
- **Plugins** are Git and Python projects that contain one or many _entry points_.
We provide a [template repository](https://github.com/FAIRmat-NFDI/nomad-plugin-template)
for a quick start into plugin development.
@@ -54,19 +55,92 @@ There are three core components to the plugin system:
This architecture allows plugin developers to freely choose a suitable granularity for their use case: they may create a single plugin package that contains everything that e.g. a certain lab needs: schemas, parsers and apps. Alternatively they may also develop multiple plugins, each containing a single entry point. Reduction in package scope can help in developing different parts indepedently and also allows plugin users to choose only the parts that they need.
-## Entry point discovery
+## Plugin entry points
-Entry points are like pre-defined connectors or hooks that allow the main system to recognize and load the code from plugins without needing to hard-code them directly into the platform. This mechanism enables the automatic discovery of plugin code. Plugin entry points represent different types of customizations that can be added to a NOMAD installation. The following plugin entry point types are currently supported:
+Plugin entry points represent different types of customizations that can be added to a NOMAD installation. The following plugin entry point types are currently supported:
+- [APIs](../howto/plugins/apis.md)
- [Apps](../howto/plugins/apps.md)
- [Example uploads](../howto/plugins/example_uploads.md)
- [Normalizers](../howto/plugins/parsers.md)
- [Parsers](../howto/plugins/parsers.md)
- [Schema packages](../howto/plugins/schema_packages.md)
-## Loading plugins
-Entry points contain **configuration** (the entry point), but also a separate **resource** (the implementation). This split enables lazy-loading: the configuration can be loaded immediately, while the resource is loaded later when/if it is required. This can significantly improve startup times, as long as all time-consuming initializations are performed only when loading the resource. This split also helps to avoid cyclical imports between the plugin code and the `nomad-lab` package. The following diagram illustrates how NOMAD interacts with the entry points in a plugin:
+Entry points contain **configuration**, but also a **resource**, which lives in a separate Python module. This split enables lazy-loading: the configuration can be loaded immediately, while the resource is loaded later when/if it is required. This can significantly improve startup times, as long as all time-consuming initializations are performed only when loading the resource. This split also helps to avoid cyclical imports between the plugin code and the `nomad-lab` package.
+
+For example the entry point configuration for a parser is contained in `.../parsers/__init__.py` and it contains e.g. the name, version and any additional entry point-specific parameters that control its behaviour. The entry point has a `load` method than can be called lazily to return the resource, which is a `Parser` instance defined in `.../parsers/myparser.py`.
+
+In `pyproject.toml` you can expose plugin entry points for automatic discovery. E.g. to expose an app and a package, you would add the following to `pyproject.toml`:
+
+```toml
+[project.entry-points.'nomad.plugin']
+myapp = "nomad_example.parsers:myapp"
+mypackage = "nomad_example.schema_packages:mypackage"
+```
+
+Here it is important to use the `nomad.plugin` group name in the `project.entry-points` header. The value on the right side (`"nomad_example.schema_packages:mypackage"`) must be a path pointing to a plugin entry point instance inside the python code. This unique key will be used to identify the plugin entry point when e.g. accessing it to read some of it's configuration values. The name on the left side (`mypackage`) can be set freely.
+
+You can read more about how to write different types of entry points in their dedicated documentation pages or learn more about the [Python entry point mechanism](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
+
+### Plugin configuration
+
+The plugin entry point configuration is an instance of a [`pydantic`](https://docs.pydantic.dev/latest/) model. This base model may already contain entry point-specific fields (such as the file extensions that a parser plugin will match) but it is also possible to extend this model to define additional fields that control your plugin behaviour.
+
+Here is an example of a new plugin entry point configuration class and instance for a parser, that has a new custom `parameter` configuration added as a `pydantic` `Field`:
+
+```python
+from pydantic import Field
+from nomad.config.models.plugins import ParserEntryPoint
+
+
+class MyParserEntryPoint(ParserEntryPoint):
+ parameter: int = Field(0, description='Config parameter for this parser.')
+
+myparser = MyParserEntryPoint(
+ name = 'MyParser',
+ description = 'My custom parser.',
+ mainfile_name_re = '.*\.myparser',
+)
+```
+
+The plugin entry point behaviour can be controlled in `nomad.yaml` using `plugins.entry_points.options`:
+
+```yaml
+plugins:
+ entry_points:
+ options:
+ "nomad_example.parsers:myparser":
+ parameter: 47
+```
+
+Note that the model will also validate the values coming from `nomad.yaml`, and you should utilize the validation mechanisms of `pydantic` to provide users with helpful messages about invalid configuration.
+
+### Plugin resource
+
+The configuration class has a `load` method that returns the entry point resource. This is typically an instance of a class, e.g. `Parser` instance in the case of a parser entry point. Here is an example of a `load` method for a parser:
+
+```python
+class MyParserEntryPoint(ParserEntryPoint):
+
+ def load(self):
+ from nomad_example.parsers.myparser import MyParser
+
+ return MyParser(**self.dict())
+```
+
+Often when loading the resource, you will need access to the final entry point configuration defined in `nomad.yaml`. This way also any overrides to the plugin configuration are correctly taken into account. You can get the final configuration using the `get_plugin_entry_point` function and the plugin name as defined in `pyproject.toml` as an argument:
+
+```python
+from nomad.config import config
+
+configuration = config.get_plugin_entry_point('nomad_example.parsers:myparser')
+print(f'The parser parameter is: {configuration.parameter}')
+```
+
+## Entry point discovery
+
+Entry points are like pre-defined connectors or hooks that allow the main system to recognize and load the code from plugins without needing to hard-code them directly into the platform. This mechanism enables the automatic discovery of plugin code. The following diagram illustrates how NOMAD interacts with the entry points in a plugin:
<figure markdown style="width: 100%">
``` mermaid
@@ -99,4 +173,4 @@ Entry points contain **configuration** (the entry point), but also a separate **
## Learn how to write plugins
-You can learn more about plugin development in the [Get started with plugins](../howto/plugins/plugins.md) -page.
+You can learn more about plugin development in the [introduction to plugins](../howto/plugins/plugins.md) -page.
diff --git a/docs/howto/overview.md b/docs/howto/overview.md
index 7a1bcb634ff11325d9e33084526d3e98a10c4cc6..fd890d870fd7cc3c1aa63aadd62e41ba6b14568e 100644
--- a/docs/howto/overview.md
+++ b/docs/howto/overview.md
@@ -58,20 +58,30 @@ Host NOMAD for your lab or institution.
</div>
<div markdown="block">
+### Plugins
+
+Learn how to write NOMAD plugins.
+
+- [Introduction to plugins](plugins/plugins.md)
+- [Write an API](plugins/apis.md)
+- [Write an app](plugins/apps.md)
+- [Write an example upload](plugins/example_uploads.md)
+- [Write a normalizer](plugins/normalizers.md)
+- [Write a parser](plugins/parsers.md)
+- [Write a schema packages](plugins/schema_packages.md)
+
+</div>
+<div markdown="block">
+
### Customization
-Customize NOMAD, write plugins, and tailor NOMAD Oasis.
+Customize NOMAD and tailor NOMAD Oasis.
- [Write a schema](customization/basics.md)
- [Define ELNs](customization/elns.md)
- [Use base sections](customization/base_sections.md)
- [Parse tabular data](customization/tabular.md)
- [Define workflows](customization/workflows.md)
-- [Write plugins](plugins/plugins.md)
-- [Write an app](plugins/apps.md)
-- [Write a normalizer](plugins/normalizers.md)
-- [Write a parser](plugins/parsers.md)
-- [Write a schema packages](plugins/schema_packages.md)
- [Work with units](customization/units.md)
- [Use HDF5 to handle large quantities](customization/hdf5.md)
- [Use Mapping parser to write data on archive](customization/mapping_parser.md)
diff --git a/docs/howto/plugins/apis.md b/docs/howto/plugins/apis.md
index 3f87105fe9b5563b7cc9d0436e640862df360f5d..b665f91940a9068766c53a3a39e4043a6562a86f 100644
--- a/docs/howto/plugins/apis.md
+++ b/docs/howto/plugins/apis.md
@@ -5,7 +5,7 @@ a [FastAPI](https://fastapi.tiangolo.com) apps that can be mounted into the main
such as `/api/v1`, `/optimade`, etc.
This documentation shows you how to write a plugin entry point for an API.
-You should read the [documentation on getting started with plugins](./plugins.md)
+You should read the [introduction to plugins](./plugins.md)
to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
## Getting started
diff --git a/docs/howto/plugins/apps.md b/docs/howto/plugins/apps.md
index 2e043d8aaf4988825029c057858ada9c62640263..3dda6cc58485759a8c73e40e1e97bda9279c630c 100644
--- a/docs/howto/plugins/apps.md
+++ b/docs/howto/plugins/apps.md
@@ -4,7 +4,7 @@ Apps provide customized views of data in the GUI, making it easier for the users
Apps only affect the way data is *displayed* for the user: if you wish to affect the underlying data structure, you will need to write a [Python schema package](./schema_packages.md) or a [YAML schema package](../customization/basics.md).
-This documentation shows you how to write an plugin entry point for an app. You should read the [documentation on getting started with plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
+This documentation shows you how to write an plugin entry point for an app. You should read the [introduction to plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
## Getting started
@@ -29,6 +29,7 @@ The entry point defines basic information about your app and is used to automati
```python
from nomad.config.models.plugins import AppEntryPoint
+from nomad.config.models.ui import App
myapp = AppEntryPoint(
name = 'MyApp',
@@ -51,105 +52,9 @@ myapp = "nomad_example.apps:myapp"
The definition fo the actual app is given as an instance of the `App` class specified as part of the entry point. A full breakdown of the model is given below in the [app reference](#app-reference), but here is a small example:
```python
-from nomad.config.models.plugins import AppEntryPoint
-from nomad.config.models.ui import App, Column, Menu, MenuItemPeriodicTable, MenuItemHistogram, MenuItemTerms, SearchQuantities
-
-schema = 'nomad_example.schema_packages.mypackage.MySchema'
-myapp = AppEntryPoint(
- name='MyApp',
- description='App defined using the new plugin mechanism.',
- app = App(
- # Label of the App
- label='My App',
- # Path used in the URL, must be unique
- path='myapp',
- # Used to categorize apps in the explore menu
- category='Theory',
- # Brief description used in the app menu
- description='An app customized for me.',
- # Longer description that can also use markdown
- readme='Here is a much longer description of this app.',
- # If you want to use quantities from a custom schema, you need to load
- # the search quantities from it first here. Note that you can use a glob
- # syntax to load the entire package, or just a single schema from a
- # package.
- search_quantities=SearchQuantities(
- include=['*#nomad_example.schema_packages.mypackage.MySchema'],
- ),
- # Controls which columns are shown in the results table
- columns=[
- Column(quantity='entry_id', selected=True),
- Column(
- quantity=f'data.section.myquantity#{schema}',
- selected=True
- ),
- Column(
- quantity=f'data.my_repeated_section[*].myquantity#{schema}',
- selected=True
- )
- Column(quantity='upload_create_time')
- ],
- # Dictionary of search filters that are always enabled for queries made
- # within this app. This is especially important to narrow down the
- # results to the wanted subset. Any available search filter can be
- # targeted here. This example makes sure that only entries that use
- # MySchema are included.
- filters_locked={
- "section_defs.definition_qualified_name": [schema]
- },
- # Controls the menu shown on the left
- menu = Menu(
- title='Material',
- items=[
- Menu(
- title='elements',
- items=[
- MenuItemPeriodicTable(
- quantity='results.material.elements',
- ),
- MenuItemTerms(
- quantity='results.material.chemical_formula_hill',
- width=6,
- options=0,
- ),
- MenuItemTerms(
- quantity='results.material.chemical_formula_iupac',
- width=6,
- options=0,
- ),
- MenuItemHistogram(
- x='results.material.n_elements',
- )
- ]
- )
- ]
- )
- # Controls the default dashboard shown in the search interface
- dashboard={
- 'widgets': [
- {
- 'type': 'histogram',
- 'show_input': False,
- 'autorange': True,
- 'nbins': 30,
- 'scale': 'linear',
- 'quantity': f'data.mysection.myquantity#{schema}',
- 'layout': {
- 'lg': {
- 'minH': 3,
- 'minW': 3,
- 'h': 4,
- 'w': 12,
- 'y': 0,
- 'x': 0
- }
- }
- }
- ]
- }
- )
-)
+--8<-- "examples/plugins/app.py"
```
+
!!! tip
If you want to load an app definition from a YAML file, this can be easily done with the pydantic `parse_obj` function:
@@ -178,7 +83,7 @@ By default, quantities from custom schemas are not available in an app, and they
!!! important
- Note that not all of the quantities from a custom schema can be loaded into the search. At the moment we only support loading **scalar** quantities from custom schemas.
+ Note that not all of the quantities from a custom schema can be loaded into the search. At the moment, we only support loading **scalar** quantities from custom schemas.
Each schema has a unique name within the NOMAD ecosystem, which is needed to target them in the configuration. The name depends on the resource in which the schema is defined in:
@@ -192,10 +97,9 @@ if you have uploaded a schema YAML file containing a section definition called
`MySchema`, and it has been assigned an `entry_id`, the schema name will be
`entry_id:<entry_id>.MySchema`.
-The quantities from schemas may be included or excluded as filter by using the
-[`filters`](#filters) field in the app config. This option supports a
-wildcard/glob syntax for including/excluding certain filters. For example, to
-include all filters from the Python schema defined in the class
+The quantities from schemas may be included or excluded by using the [`SearchQuantities`](#app-reference)
+field in the app config. This option supports a wildcard/glob syntax for including/excluding certain
+search quantities. For example, to include all search quantities from the Python schema defined in the class
`nomad_example.schema_packages.mypackage.MySchema`, you could use:
```python
@@ -212,18 +116,9 @@ search_quantities=SearchQuantities(
)
```
-Once search quantities are loaded, they can be targeted in the rest of the app. The app configuration often refers to specific search quantities to configure parts of the user interface. For example, one could configure the results table to show a new column using one of the search quantities with:
+### Using loaded search quantity definitions
-```python
-columns=[
- Column(quantity='entry_id', selected=True),
- Column(
- quantity='data.mysection.myquantity#nomad_example.schema_packages.mypackage.MySchema',
- selected=True
- ),
- Column(quantity='upload_create_time')
-]
-```
+Once search quantities are loaded, they can be targeted in the rest of the app. The app configuration often refers to specific search quantities to configure parts of the user interface.
The syntax for targeting quantities depends on the resource:
@@ -234,56 +129,62 @@ by a hashtag (#), for example `data.mysection.myquantity#entry_id:<entry_id>.MyS
- Quantities that are common for all NOMAD entries can be targeted by using only
the path without the need for specifying a schema, e.g. `results.material.symmetry.space_group`.
+For example, one could configure the results table to show a new column using one of the search quantities with:
+
+```python
+--8<-- "examples/plugins/columns.py"
+```
+
+### Narrowing down search results in the app
+
+The search results that will show up in the app can be narrowed down by passing
+a dictionary to the `filters_locked` option. In the example app, only
+entries that use `MySchema` are included.
+
+```python
+filters_locked={
+ "section_defs.definition_qualified_name": [schema]
+}
+```
+
+It is also possible to filter by quantities defined in the [`results`](../../reference/glossary.md#results-section-results) section. For example, if you want to limit your app to entries that have the property `catalytic` filled in the `results` section:
+
+```python
+filters_locked={
+ "quantities": ["results.properties.catalytic"]
+}
+```
+
### Menu
-The `menu` field controls the structure of the menu shown on the left side of the search interface. Menus have a controllable width, and they contains items that are displayed on a 12-based grid. You can also nest menus within each other. For example, this defines a menu with two levels:
+The `menu` field controls the structure of the menu shown on the left side of the search interface. Menus have a controllable width, and they contain items that are displayed on a 12-based grid. You can also nest menus within each other. For example, this defines a menu with two levels:
```python
-# This is a top level menu that is always visible. It shows two items: a terms
-# item and a submenu beneath it.
-menu = Menu(
- size='sm',
- items=[
- MenuItemTerms(
- search_quantity='authors.name',
- options=5
- ),
- # This is a submenu whose items become visible once selected. It
- # contains three items: one full-width histogram and two terms items
- # which are displayed side-by-side.
- Menu(
- title='Submenu'
- size='md',
- items=[
- MenuItemHistogram(
- search_quantity='upload_create_time'
- ),
- # These items target data from a custom schema
- MenuItemTerms(
- width=6,
- search_quantity='data.quantity1#nomad_example.schema_packages.mypackage.MySchema'
- ),
- MenuItemTerms(
- width=6,
- search_quantity='data.quantity2#nomad_example.schema_packages.mypackage.MySchema'
- )
- ]
- )
- ]
-)
+--8<-- "examples/plugins/menu.py"
```
The following items are supported in menus, and you can read more about them in the App reference:
- - [`Menu`](#menu_1): Defines a nested submenu.
- - [`MenuItemTerms`](#menuitemterms): Used to display a set of possible text options.
- - [`MenuItemHistogram`](#menuitemhistogram): Histogram of numerical values.
- - [`MenuItemPeriodictable`](#menuitemperiodictable): Displays a periodic table.
- - [`MenuItemOptimade`](#menuitemoptimade): OPTIMADE query field.
- - [`MenuItemVisibility`](#menuitemvisibility): Controls for the query visibility.
- - [`MenuItemDefinitions`](#menuitemdefinitions): Shows a tree of available definitions from which items can be selected for the query.
- - [`MenuItemCustomQuantities`](#menuitemcustomquantities): Form for querying custom quantities coming from any schema.
- - [`MenuItemNestedObject`](#menuitemnestedobject): Used to group together menu items so that their query is performed using an Elasticsearch nested query. Note that you cannot yet use nested queries for search quantities originating from custom schemas.
+ - [`Menu`](#app-reference): Defines a nested submenu.
+ - `MenuItemTerms`: Used to display a set of possible text options.
+ - `MenuItemHistogram`: Histogram of numerical values.
+ - `MenuItemPeriodictable`: Displays a periodic table.
+ - `MenuItemOptimade`: Field for entering OPTIMADE queries.
+ - `MenuItemVisibility`: Controls for the query visibility.
+ - `MenuItemDefinitions`: Shows a tree of available definitions from which items can be selected for the query.
+ - `MenuItemCustomQuantities`: Form for querying custom quantities coming from any schema.
+ - `MenuItemNestedObject`: Used to group together menu items so that their query is performed using an Elasticsearch nested query. Note that you cannot yet use nested queries for search quantities originating from custom schemas.
+
+### Dashboard
+ The Dashboard field controls the content of the main search interface window. Different widgets can be added which contain terms or numerical information and can be controlled in size and position. There are 4 different types of Widgets:
+ `WidgetTerms`,
+ `WidgetHistogram`,
+ `WidgetScatterplot` and
+ `WidgetPeriodicTable`
+
+```python
+--8<-- "examples/plugins/dashboard.py:13:"
+```
## App reference
diff --git a/docs/howto/plugins/example_uploads.md b/docs/howto/plugins/example_uploads.md
index 3243d093ccef49ad32c1ebc25aa4fca2b105bb0e..fe3cc97a41f7a7822fd3b043f0731c27cd911053 100644
--- a/docs/howto/plugins/example_uploads.md
+++ b/docs/howto/plugins/example_uploads.md
@@ -2,7 +2,7 @@
Example uploads can be used to add representative collections of data for your plugin. Example uploads are available for end-users in the *Uploads*-page under the *Add example uploads*-button. There, users can instantiate an example upload with a click. This can be very useful for educational or demonstration purposes but also for testing.
-This documentation shows you how to write a plugin entry point for an example upload. You should read the [documentation on getting started with plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
+This documentation shows you how to write a plugin entry point for an example upload. You should read the [introduction to plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
## Getting started
@@ -25,7 +25,7 @@ See the documentation on [plugin development guidelines](./plugins.md#plugin-dev
## Example upload entry point
-The entry point is an instance of a `ExampleUploadEntryPoint` or its subclass. It defines basic information about your example upload and is used to automatically load the associated data into a NOMAD distribution. The entry point should be defined in `*/example_uploads/__init__.py` like this:
+The entry point is an instance of an `ExampleUploadEntryPoint` or its subclass. It defines basic information about your example upload and is used to automatically load the associated data into a NOMAD distribution. The entry point should be defined in `*/example_uploads/__init__.py` like this:
```python
from nomad.config.models.plugins import ExampleUploadEntryPoint
@@ -38,7 +38,7 @@ myexampleupload = ExampleUploadEntryPoint(
)
```
-The `resources` field can contain one or several data resources that can be provided directly in the Python package or stored online. You can learn more about different data loading options in the next section. In the reference you can also see all of the available [configuration options for a `ExampleUploadEntryPoint`](../../reference/plugins.md#exampleuploadentrypoint).
+The `resources` field can contain one or several data resources that can be provided directly in the Python package or stored online. You can learn more about different data loading options in the next section. In the reference you can also see all of the available [configuration options for an `ExampleUploadEntryPoint`](../../reference/plugins.md#exampleuploadentrypoint).
The entry point instance should then be added to the `[project.entry-points.'nomad.plugin']` table in `pyproject.toml` in order for the example upload to be automatically detected:
diff --git a/docs/howto/plugins/normalizers.md b/docs/howto/plugins/normalizers.md
index c939b63a92386e3a66076332f07aefb30857857d..e65ee182429b29ce0e588efe5c9f34eed9133071 100644
--- a/docs/howto/plugins/normalizers.md
+++ b/docs/howto/plugins/normalizers.md
@@ -2,7 +2,7 @@
A normalizer takes the archive of an entry as input and manipulates (usually expands) the given archive. This way, a normalizer can add additional sections and quantities based on the information already available in the archive. All normalizers are executed in the order [determined by their `level`](#control-normalizer-execution-order) after parsing, but the normalizer may decide to not do anything based on the entry contents.
-This documentation shows you how to write a plugin entry point for a normaliser. You should read the [documentation on getting started with plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
+This documentation shows you how to write a plugin entry point for a normaliser. You should read the [introduction to plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
## Getting started
@@ -27,7 +27,6 @@ See the documentation on [plugin development guidelines](./plugins.md#plugin-dev
The entry point defines basic information about your normalizer and is used to automatically load the normalizer code into a NOMAD distribution. It is an instance of a `NormalizerEntryPoint` or its subclass and it contains a `load` method which returns a `nomad.normalizing.Normalizer` instance that will perform the actual normalization. You will learn more about the `Normalizer` class in the next sections. The entry point should be defined in `*/normalizers/__init__.py` like this:
```python
-from pydantic import Field
from nomad.config.models.plugins import NormalizerEntryPoint
@@ -62,11 +61,13 @@ mynormalizer = "nomad_example.normalizers:mynormalizer"
The resource returned by a normalizer entry point must be an instance of a `nomad.normalizing.Normalizer` class. This normalizer definition should be contained in a separate file (e.g. `*/normalizer/mynormalizer.py`) and could look like this:
```python
-from typing import Dict
+from typing import TYPE_CHECKING
-from nomad.datamodel import EntryArchive
from nomad.normalizing import Normalizer
+if TYPE_CHECKING:
+ from nomad.datamodel import EntryArchive
+
class MyNormalizer(Normalizer):
def normalize(
@@ -82,13 +83,13 @@ The minimal requirement is that your class has a `normalize` function, which as
- `archive`: The [`EntryArchive` object](../../reference/glossary.md#archive) in which the normalization results will be stored
- `logger`: Logger that you can use to log normalization events into
-## `SystemBasedNormalizer` class
+### `SystemBasedNormalizer` class
`SystemBasedNormalizer` is a special base class for normalizing systems that allows to run the normalization on all (or only the resulting) `representative` systems:
```python
-from nomad.normalizing import SystemBasedNormalizer
from nomad.atomutils import get_volume
+from nomad.normalizing import SystemBasedNormalizer
class UnitCellVolumeNormalizer(SystemBasedNormalizer):
def _normalize_system(self, system, is_representative):
@@ -142,9 +143,9 @@ from nomad.datamodel import EntryArchive
from nomad_example.normalizers.mynormalizer import MyNormalizer
import logging
-p = MyNormalizer()
-a = EntryArchive()
-p.normalize(a, logger=logging.getLogger())
+normalizer = MyNormalizer()
+entry_archive = EntryArchive()
+normalizer.normalize(entry_archive, logger=logging.getLogger())
-print(a.m_to_dict())
-```
\ No newline at end of file
+print(entry_archive.m_to_dict())
+```
diff --git a/docs/howto/plugins/parsers.md b/docs/howto/plugins/parsers.md
index fa3b44cb838cd13bdfaa394306a74270dc44f743..fb7d450b52a4d84c39a423b3ac78dc0464bc7d42 100644
--- a/docs/howto/plugins/parsers.md
+++ b/docs/howto/plugins/parsers.md
@@ -2,7 +2,7 @@
NOMAD uses parsers to automatically extract information from raw files and output that information into structured [archives](../../reference/glossary.md#archive). Parsers can decide which files act upon based on the filename, mime type or file contents and can also decide into which schema the information should be populated into.
-This documentation shows you how to write a plugin entry point for a parser. You should read the [documentation on getting started with plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
+This documentation shows you how to write a plugin entry point for a parser. You should read the [introduction to plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
## Getting started
@@ -162,7 +162,7 @@ import sys
from nomad.client import parse, normalize_all
# Match and run the parser
-archives = parse('path/to/you/file')
+archives = parse('path/to/your/file')
# Run all normalizers
for archive in archives:
normalize_all(archive)
@@ -406,7 +406,7 @@ argument `skip-normalizers`.
## Extending the Metainfo
There are several built-in schemas NOMAD (`nomad.datamodel.metainfo`).
<!-- ? What about restructuring this part into the idea of "NOMAD has some predefined section and quantities ... please, check HERE... and HERE... for more information and details"? -->
-In the example above, we have made use of the base section for workflow and extended
+In the example below, we have made use of the base section for workflow and extended
it to include a code-specific quantity `x_example_magic_value`.
```python
# We extend the existing common definition of section Workflow
diff --git a/docs/howto/plugins/plugins.md b/docs/howto/plugins/plugins.md
index 47be45319fc74afac064283893c0c647e14a59e8..810e308a1c3a4e9ac08f78dbe9786a2552bea27d 100644
--- a/docs/howto/plugins/plugins.md
+++ b/docs/howto/plugins/plugins.md
@@ -1,16 +1,21 @@
-# Get started with plugins
+# Introduction to plugins
-The main way to customize a NOMAD installation is through the use of **plugins**. A NOMAD plugin is a Python package that an administrator can install into a NOMAD distribution to add custom features. This page contains shows you how to create, develop and publish a NOMAD plugin. For a high-level overview of the plugin mechanism, see the [NOMAD plugin system](../../explanation/plugin_system.md) -page.
+The main way to customize a NOMAD installation is through the use of **plugins**. A NOMAD plugin is a Python package that an administrator can install into a NOMAD distribution to add custom features. This page helps you develop and publish a NOMAD plugin. For a high-level overview of the plugin mechanism, see the [NOMAD plugin system](../../explanation/plugin_system.md) -page.
-See the [FAIRmat-NFDI GitHub organization page](https://github.com/FAIRmat-NFDI) for a list of plugins developed by FAIRmat. You can also see the list of activated plugins and plugin entry points at the bottom of the _Information page_ (`about/information`) of any NOMAD installation, for example check out the [central NOMAD installation](https://nomad-lab.eu/prod/v1/gui/about/information).
+A single Python plugin package can contain multiple [plugin entry points](../../explanation/plugin_system.md#plugin-entry-points). These entry points represent different types of customizations including:
-## Plugin anatomy
+- [APIs](./apis.md)
+- [Apps](./apps.md)
+- [Example uploads](./example_uploads.md)
+- [Normalizers](./parsers.md)
+- [Parsers](./parsers.md)
+- [Schema packages](./schema_packages.md)
-!!! tip
+See the [FAIRmat-NFDI GitHub organization page](https://github.com/FAIRmat-NFDI) for a list of plugins developed by FAIRmat. You can also see the list of activated plugins and plugin entry points at the bottom of the _Information page_ (`about/information`) of any NOMAD installation, for example check out the [central NOMAD installation](https://nomad-lab.eu/prod/v1/gui/about/information).
- We provide a [template repository](https://github.com/FAIRmat-NFDI/nomad-plugin-template) which you can use to create the initial plugin layout for you.
+## Plugin anatomy
-Plugin Git repositories should roughly follow this layout:
+We provide a [template repository](https://github.com/FAIRmat-NFDI/nomad-plugin-template) which you can use to create the initial plugin repository layout for you. The repository layout as generated by the template looks like this:
```txt
├── nomad-example
@@ -39,80 +44,6 @@ We suggest using the following convention for naming the repository name and the
- repository name: `nomad-<plugin name>`
- package name: `nomad_<plugin name>`
-In the folder structure you can see that a single plugin can contain multiple types of customizations: apps, parsers, schema packages and normalizers. These are called a **plugin entry points** and you will learn more about them next.
-
-## Plugin entry points
-
-Plugin entry points represent different types of customizations that can be added to a NOMAD installation. Entry points contain **configuration**, but also a **resource**, which lives in a separate Python module. This split enables lazy-loading: the configuration can be loaded immediately, while the resource is loaded later when/if it is required. This can significantly improve startup times, as long as all time-consuming initializations are performed only when loading the resource. This split also helps to avoid cyclical imports between the plugin code and the `nomad-lab` package.
-
-For example the entry point configuration for a parser is contained in `.../parsers/__init__.py` and it contains e.g. the name, version and any additional entry point-specific parameters that control its behaviour. The entry point has a `load` method than can be called lazily to return the resource, which is a `Parser` instance defined in `.../parsers/myparser.py`.
-
-In `pyproject.toml` you can expose plugin entry points for automatic discovery. E.g. to expose an app and a package, you would add the following to `pyproject.toml`:
-
-```toml
-[project.entry-points.'nomad.plugin']
-myapp = "nomad_example.parsers:myapp"
-mypackage = "nomad_example.schema_packages:mypackage"
-```
-
-Here it is important to use the `nomad.plugin` group name in the `project.entry-points` header. The value on the right side (`"nomad_example.schema_packages:mypackage"`) must be a path pointing to a plugin entry point instance inside the python code. This unique key will be used to identify the plugin entry point when e.g. accessing it to read some of it's configuration values. The name on the left side (`mypackage`) can be set freely.
-
-You can read more about how to write different types of entry points in their dedicated documentation pages or learn more about the [Python entry point mechanism](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
-
-### Plugin configuration
-
-The plugin entry point configuration is an instance of a [`pydantic`](https://docs.pydantic.dev/1.10/) model. This base model may already contain entry point-specific fields (such as the file extensions that a parser plugin will match) but it is also possible to extend this model to define additional fields that control your plugin behaviour.
-
-Here is an example of a new plugin entry point configuration class and instance for a parser, that has a new custom `parameter` configuration added as a `pydantic` `Field`:
-
-```python
-from pydantic import Field
-from nomad.config.models.plugins import ParserEntryPoint
-
-
-class MyParserEntryPoint(ParserEntryPoint):
- parameter: int = Field(0, description='Config parameter for this parser.')
-
-myparser = MyParserEntryPoint(
- name = 'MyParser',
- description = 'My custom parser.',
- mainfile_name_re = '.*\.myparser',
-)
-```
-
-The plugin entry point behaviour can be controlled in `nomad.yaml` using `plugins.entry_points.options`:
-
-```yaml
-plugins:
- entry_points:
- options:
- "nomad_example.parsers:myparser":
- parameter: 47
-```
-
-Note that the model will also validate the values coming from `nomad.yaml`, and you should utilize the validation mechanisms of `pydantic` to provide users with helpful messages about invalid configuration.
-
-### Plugin resource
-
-The configuration class has a `load` method that returns the entry point resource. This is typically an instance of a class, e.g. `Parser` instance in the case of a parser entry point. Here is an example of a `load` method for a parser:
-
-```python
-class MyParserEntryPoint(ParserEntryPoint):
-
- def load(self):
- from nomad_example.parsers.myparser import MyParser
-
- return MyParser(**self.dict())
-```
-
-Often when loading the resource, you will need access to the final entry point configuration defined in `nomad.yaml`. This way also any overrides to the plugin configuration are correctly taken into account. You can get the final configuration using the `get_plugin_entry_point` function and the plugin name as defined in `pyproject.toml` as an argument:
-
-```python
-from nomad.config import config
-
-configuration = config.get_plugin_entry_point('nomad_example.parsers:myparser')
-print(f'The parser parameter is: {configuration.parameter}')
-```
### Controlling loading of plugin entry points
diff --git a/docs/howto/plugins/schema_packages.md b/docs/howto/plugins/schema_packages.md
index fbe1ea0863ae1e66542ea37ef2f459fef6580ded..6d0f716d2b9f7e70cd8dac1b221d1fc2a21c3d4e 100644
--- a/docs/howto/plugins/schema_packages.md
+++ b/docs/howto/plugins/schema_packages.md
@@ -1,8 +1,8 @@
# How to write a schema package
-Schema packages are used to define and distribute custom data definitions that can be used within NOMAD. These schema packages typically contain [schemas](../../reference/glossary.md#schema) that users can select to instantiate manually filled entries using our ELN functionality, or that parsers when organizing data they extract from files. Schema packages may also contain more abstract base classes that other schema packages use.
+Schema packages are used to define and distribute custom data definitions that can be used within NOMAD. These schema packages typically contain [schemas](../../reference/glossary.md#schema) that users can select to instantiate manually filled entries using our ELN functionality, or that parsers select when organizing data they extract from files. Schema packages may also contain more abstract base classes that other schema packages use.
-This documentation shows you how to write a plugin entry point for a schema package. You should read the [documentation on getting started with plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
+This documentation shows you how to write a plugin entry point for a schema package. You should read the [introduction to plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
## Getting started
@@ -45,7 +45,7 @@ mypackage = MySchemaPackageEntryPoint(
)
```
-Here you can see that a new subclass of `SchemaPackageEntryPoint` was defined. In this new class you can override the `load` method to determine how the `SchemaPackage` class is loaded, but you can also extend the `SchemaPackageEntryPoint` model to add new configurable parameters for this schema package as explained [here](./plugins.md#plugin-configuration).
+Here you can see that a new subclass of `SchemaPackageEntryPoint` was defined. In this new class you can override the `load` method to determine how the `SchemaPackage` class is loaded, but you can also extend the `SchemaPackageEntryPoint` model to add new configurable parameters for this schema package as explained [here](../../explanation/plugin_system.md#plugin-configuration).
We also instantiate an object `mypackage` from the new subclass. This is the final entry point instance in which you specify the default parameterization and other details about the schema package. In the reference you can see all of the available [configuration options for a `SchemaPackageEntryPoint`](../../reference/plugins.md#schemapackageentrypoint).
@@ -468,6 +468,7 @@ NOMAD. It therefore defines the top level sections:
- `results`: a summary with copies and references to data from method specific sections. This also
presents the [searchable metadata](../develop/search.md).
- `workflows`: all workflow metadata
+- `data`: contains all data from method specific sections by default.
- Method-specific subsections: e.g. `run`. This is were all parsers are supposed to
add the parsed data.
<!-- TODO Update!!! -->
@@ -790,3 +791,4 @@ The following is a list of plugins containing schema packages developed by FAIRm
| synthesis | <https://github.com/FAIRmat-NFDI/AreaA-data_modeling_and_schemas.git> |
| material processing | <https://github.com/FAIRmat-NFDI/nomad-material-processing.git> |
| measurements | <https://github.com/FAIRmat-NFDI/nomad-measurements.git> |
+| catalysis | <https://github.com/FAIRmat-NFDI/nomad-catalysis-plugin.git> |
diff --git a/docs/index.md b/docs/index.md
index b2d1a0f2db56c0629cae537a40d83396e73d80ab..93be0698b8e6a688d5280180f9c81ab6b39c4493 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -43,6 +43,7 @@ How-to guides provide step-by-step instructions for a wide range of tasks, with
- Manage and find data
- Programmatic data access
- Oasis
+- Plugins
- Customization
- Development
diff --git a/docs/reference/glossary.md b/docs/reference/glossary.md
index 1448c98b11d5ae0780d5fd15035565a8de19a895..c919bfb22b32f124a436a6632a02e3e50d947240 100644
--- a/docs/reference/glossary.md
+++ b/docs/reference/glossary.md
@@ -8,7 +8,6 @@ TODO consider the following items:
- Oasis
- Package (metainfo)
- Workflow
-- App (maybe too soon to add since we don't know yet what this really entails)
-->
This is a list of terms that have a specific meaning for NOMAD and are used through
@@ -42,6 +41,17 @@ An *author* is typically a natural person that has uploaded a piece of data into
has authorship over it. Often *authors* are [users](#user), but not always.
Therefore, we have to distinguish between authors and users.
+### Dataset
+
+Users can organize [entries](#entry) into *datasets*. Datasets are not created automatically,
+don't confuse them with [uploads](#upload). Datasets can be compared to albums, labels, or tags
+on other platforms. Datasets are used to reference a collection of data and users can get a DOI for their
+datasets.
+
+### Distribution / distro
+
+NOMAD *Distribution* is a Git repository containing the configuration for instantiating a customized NOMAD instance. Distributions define the plugins that should be installed, the configurations files (e.g. `nomad.yaml`) to use, CI pipeline steps for building final Docker images and a `docker-compose.yaml` file that can be used to launch the instance.
+
### ELN
Electronic Lab Notebooks (*ELNs*) are a specific kind of [entry](#entry) in NOMAD. These
@@ -58,12 +68,9 @@ associated with [raw files](#raw-file), where one of these files is the [mainfil
Raw files are processed to create the [processed data](#processed-data) (or the [archive](#archive))
for an entry.
-### Dataset
+### Example upload
-Users can organize [entries](#entry) into *datasets*. Datasets are not created automatically,
-don't confuse them with [uploads](#upload). Datasets can be compared to albums, labels, or tags
-on other platforms. Datasets are used to reference a collection of data and users can get a DOI for their
-datasets.
+*Example uploads* are pre-prepared uploads containing data that typically showcases certain features of a plugin. The contents of example uploads can be fixed, created programmatically or fetched from online sources. Example uploads can be instantiated by using the "Example uploads" -button in the "Uploads" -page of the GUI. Example uploads can be defined by creating an example upload [plugin entry point](#plugin-entry-point).
### Mainfile
diff --git a/docs/reference/plugins.md b/docs/reference/plugins.md
index 5b7324de2491114493f1c15610e4277111e053f2..60acaf88a0d7410a1a49690dfc26874e8dda9531 100644
--- a/docs/reference/plugins.md
+++ b/docs/reference/plugins.md
@@ -10,12 +10,12 @@ Plugins allow one to add Python-based functionality to NOMAD without a custom NO
This is a list of the available plugin entry point configuration models.
+{{ pydantic_model('nomad.config.models.plugins.APIEntryPoint') }}
{{ pydantic_model('nomad.config.models.plugins.AppEntryPoint') }}
{{ pydantic_model('nomad.config.models.plugins.ExampleUploadEntryPoint') }}
{{ pydantic_model('nomad.config.models.plugins.NormalizerEntryPoint') }}
{{ pydantic_model('nomad.config.models.plugins.ParserEntryPoint') }}
{{ pydantic_model('nomad.config.models.plugins.SchemaPackageEntryPoint') }}
-{{ pydantic_model('nomad.config.models.plugins.APIEntryPoint') }}
## Default plugin entry points
diff --git a/examples/metainfo/use_metainfo.py b/examples/metainfo/use_metainfo.py
deleted file mode 100644
index 637b980d993ab701db336ae1856581f670855724..0000000000000000000000000000000000000000
--- a/examples/metainfo/use_metainfo.py
+++ /dev/null
@@ -1,13 +0,0 @@
-import json
-
-from nomad.datamodel.metainfo import public
-
-# A simple example that demonstrates how to set references
-run = public.section_run()
-scc = run.m_create(public.section_single_configuration_calculation)
-system = run.m_create(public.section_system)
-scc.single_configuration_calculation_to_system_ref = system
-
-assert scc.single_configuration_calculation_to_system_ref == system
-
-print(json.dumps(run.m_to_dict(), indent=2))
diff --git a/examples/plugins/app.py b/examples/plugins/app.py
new file mode 100644
index 0000000000000000000000000000000000000000..f9494d7e4f8e519146f9d816904b389e2ac5ce2f
--- /dev/null
+++ b/examples/plugins/app.py
@@ -0,0 +1,96 @@
+from nomad.config.models.plugins import AppEntryPoint
+from nomad.config.models.ui import (
+ App,
+ Axis,
+ Column,
+ Dashboard,
+ Layout,
+ Menu,
+ MenuItemPeriodicTable,
+ MenuItemHistogram,
+ MenuItemTerms,
+ SearchQuantities,
+ WidgetHistogram,
+)
+
+schema = 'nomad_example.schema_packages.mypackage.MySchema'
+myapp = AppEntryPoint(
+ name='MyApp',
+ description='App defined using the new plugin mechanism.',
+ app=App(
+ # Label of the App
+ label='My App',
+ # Path used in the URL, must be unique
+ path='myapp',
+ # Used to categorize apps in the explore menu
+ category='Theory',
+ # Brief description used in the app menu
+ description='An app customized for me.',
+ # Longer description that can also use markdown
+ readme='Here is a much longer description of this app.',
+ # If you want to use quantities from a custom schema, you need to load
+ # the search quantities from it first here. Note that you can use a glob
+ # syntax to load the entire package, or just a single schema from a
+ # package.
+ search_quantities=SearchQuantities(
+ include=['*#nomad_example.schema_packages.mypackage.MySchema'],
+ ),
+ # Controls which columns are shown in the results table
+ columns=[
+ Column(quantity='entry_id', selected=True),
+ Column(quantity=f'data.section.myquantity#{schema}', selected=True),
+ Column(
+ quantity=f'data.my_repeated_section[*].myquantity#{schema}',
+ selected=True,
+ ),
+ Column(quantity='upload_create_time'),
+ ],
+ # Dictionary of search filters that are always enabled for queries made
+ # within this app. This is especially important to narrow down the
+ # results to the wanted subset. Any available search filter can be
+ # targeted here. This example makes sure that only entries that use
+ # MySchema are included.
+ filters_locked={'section_defs.definition_qualified_name': [schema]},
+ # Controls the menu shown on the left
+ menu=Menu(
+ title='Material',
+ items=[
+ Menu(
+ title='elements',
+ items=[
+ MenuItemPeriodicTable(
+ quantity='results.material.elements',
+ ),
+ MenuItemTerms(
+ quantity='results.material.chemical_formula_hill',
+ width=6,
+ options=0,
+ ),
+ MenuItemTerms(
+ quantity='results.material.chemical_formula_iupac',
+ width=6,
+ options=0,
+ ),
+ MenuItemHistogram(
+ x='results.material.n_elements',
+ ),
+ ],
+ )
+ ],
+ ),
+ # Controls the default dashboard shown in the search interface
+ dashboard=Dashboard(
+ widgets=[
+ WidgetHistogram(
+ title='Histogram Title',
+ show_input=False,
+ autorange=True,
+ nbins=30,
+ scale='linear',
+ x=Axis(search_quantity=f'data.mysection.myquantity#{schema}'),
+ layout={'lg': Layout(minH=3, minW=3, h=4, w=12, y=0, x=0)},
+ )
+ ]
+ ),
+ ),
+)
diff --git a/examples/plugins/columns.py b/examples/plugins/columns.py
new file mode 100644
index 0000000000000000000000000000000000000000..dec8f3ceb615a2fc4d8a9d31a1d0d3f834d8ff78
--- /dev/null
+++ b/examples/plugins/columns.py
@@ -0,0 +1,11 @@
+from nomad.config.models.ui import Column
+
+columns = [
+ Column(quantity='entry_id', selected=True),
+ Column(
+ quantity='data.mysection.myquantity#nomad_example.schema_packages.mypackage.MySchema',
+ label='My Quantity Name',
+ selected=True,
+ ),
+ Column(quantity='upload_create_time'),
+]
diff --git a/examples/plugins/dashboard.py b/examples/plugins/dashboard.py
new file mode 100644
index 0000000000000000000000000000000000000000..41085b15d62179c97a7709a883b27108349f1fe9
--- /dev/null
+++ b/examples/plugins/dashboard.py
@@ -0,0 +1,56 @@
+from nomad.config.models.ui import (
+ Axis,
+ Dashboard,
+ Layout,
+ WidgetPeriodicTable,
+ WidgetTerms,
+ WidgetHistogram,
+ WidgetScatterPlot,
+)
+
+schema = 'nomad_example.schema_packages.mypackage.MySchema'
+
+dashboard = Dashboard(
+ widgets=[
+ WidgetPeriodicTable(
+ title='Elements of the material',
+ layout={
+ 'lg': Layout(h=8, minH=8, minW=12, w=12, x=0, y=0),
+ },
+ search_quantity='results.material.elements',
+ scale='linear',
+ ),
+ WidgetTerms(
+ title='Widget Terms Title',
+ layout={
+ 'lg': Layout(h=8, minH=3, minW=3, w=6, x=12, y=0),
+ },
+ search_quantity=f'data.mysection.myquantity#{schema}',
+ showinput=True,
+ scale='linear',
+ ),
+ WidgetHistogram(
+ title='Histogram Title',
+ show_input=False,
+ autorange=True,
+ nbins=30,
+ scale='linear',
+ x=Axis(search_quantity=f'data.mysection.myquantity#{schema}'),
+ layout={'lg': Layout(minH=3, minW=3, h=8, w=12, y=8, x=0)},
+ ),
+ WidgetScatterPlot(
+ title='Scatterplot title',
+ autorange=True,
+ layout={
+ 'lg': Layout(h=8, minH=3, minW=8, w=12, x=12, y=8),
+ },
+ x=Axis(
+ search_quantity=f'data.mysection.mynumericalquantity#{schema}',
+ title='quantity x',
+ ),
+ y=Axis(search_quantity=f'data.mysection.myothernumericalquantity#{schema}'),
+ color=f'data.mysection.myquantity#{schema}', # optional, if set has to be scalar value
+ size=1000, # maximum number of entries loaded
+ ),
+ ]
+)
diff --git a/examples/plugins/menu.py b/examples/plugins/menu.py
new file mode 100644
index 0000000000000000000000000000000000000000..95968246d4fa8d3416f111876e7dc24097303e8f
--- /dev/null
+++ b/examples/plugins/menu.py
@@ -0,0 +1,29 @@
+from nomad.config.models.ui import Axis, Menu, MenuItemTerms, MenuItemHistogram
+
+# This is a top level menu that is always visible. It shows two items: a terms
+# item and a submenu beneath it.
+menu = Menu(
+ size='sm',
+ items=[
+ MenuItemTerms(search_quantity='authors.name', options=5),
+ # This is a submenu whose items become visible once selected. It
+ # contains three items: one full-width histogram and two terms items
+ # which are displayed side-by-side.
+ Menu(
+ title='Submenu',
+ size='md',
+ items=[
+ MenuItemHistogram(x=Axis(search_quantity='upload_create_time')),
+ # These items target data from a custom schema
+ MenuItemTerms(
+ width=6,
+ search_quantity='data.quantity1#nomad_example.schema_packages.mypackage.MySchema',
+ ),
+ MenuItemTerms(
+ width=6,
+ search_quantity='data.quantity2#nomad_example.schema_packages.mypackage.MySchema',
+ ),
+ ],
+ ),
+ ],
+)
diff --git a/mkdocs.yml b/mkdocs.yml
index bbb8a6363b34e159fe26e9f29244577404279f8b..f48d5876f037d359e0985a85d1053f94c7c63cee 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -32,13 +32,13 @@ nav:
- Update an Oasis: howto/oasis/update.md
- Perform admin tasks: howto/oasis/admin.md
- Plugins:
- - Get started with plugins: howto/plugins/plugins.md
+ - Introduction to plugins: howto/plugins/plugins.md
+ - Write an API: howto/plugins/apis.md
- Write an app: howto/plugins/apps.md
- Write an example upload: howto/plugins/example_uploads.md
- Write a normalizer: howto/plugins/normalizers.md
- Write a parser: howto/plugins/parsers.md
- Write a schema package: howto/plugins/schema_packages.md
- - Write an API: howto/plugins/apis.md
- Customization:
- Write a YAML schema package: howto/customization/basics.md
- Define ELNs: howto/customization/elns.md
@@ -107,6 +107,8 @@ theme:
- navigation.path
- navigation.footer
- navigation.top
+ - content.code.copy
+ - content.code.annotate
icon:
repo: fontawesome/brands/github
# repo_url: https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/
diff --git a/nomad/config/models/plugins.py b/nomad/config/models/plugins.py
index a5fc95dfd94bf5ae63ea2cff7db35513d5cb3a96..9db928ae7f2e40a2929605a3e8d7b1878d8c4ae8 100644
--- a/nomad/config/models/plugins.py
+++ b/nomad/config/models/plugins.py
@@ -46,15 +46,20 @@ class EntryPoint(BaseModel):
id: str | None = Field(
None,
description='Unique identifier corresponding to the entry point name. Automatically set to the plugin entry point name in pyproject.toml.',
- )
- entry_point_type: str = Field(description='Determines the entry point type.')
+ hidden=True,
+ ) # type: ignore[call-overload]
+ entry_point_type: str = Field(
+ description='Determines the entry point type.', hidden=True
+ ) # type: ignore[call-overload]
name: str | None = Field(None, description='Name of the plugin entry point.')
description: str | None = Field(
None, description='A human readable description of the plugin entry point.'
)
plugin_package: str | None = Field(
- None, description='The plugin package from which this entry points comes from.'
- )
+ None,
+ description='The plugin package from which this entry points comes from.',
+ hidden=True,
+ ) # type: ignore[call-overload]
def dict_safe(self):
"""Used to serialize the non-confidential parts of a plugin model. This
@@ -69,8 +74,8 @@ class AppEntryPoint(EntryPoint):
"""Base model for app plugin entry points."""
entry_point_type: Literal['app'] = Field(
- 'app', description='Determines the entry point type.'
- )
+ 'app', description='Determines the entry point type.', hidden=True
+ ) # type: ignore[call-overload]
app: App = Field(description='The app configuration.')
def dict_safe(self):
@@ -83,8 +88,8 @@ class SchemaPackageEntryPoint(EntryPoint, metaclass=ABCMeta):
"""Base model for schema package plugin entry points."""
entry_point_type: Literal['schema_package'] = Field(
- 'schema_package', description='Specifies the entry point type.'
- )
+ 'schema_package', description='Specifies the entry point type.', hidden=True
+ ) # type: ignore[call-overload]
@abstractmethod
def load(self) -> 'SchemaPackage':
@@ -98,8 +103,8 @@ class NormalizerEntryPoint(EntryPoint, metaclass=ABCMeta):
"""Base model for normalizer plugin entry points."""
entry_point_type: Literal['normalizer'] = Field(
- 'normalizer', description='Determines the entry point type.'
- )
+ 'normalizer', description='Determines the entry point type.', hidden=True
+ ) # type: ignore[call-overload]
level: int = Field(
0,
description="""
@@ -121,8 +126,8 @@ class ParserEntryPoint(EntryPoint, metaclass=ABCMeta):
"""Base model for parser plugin entry points."""
entry_point_type: Literal['parser'] = Field(
- 'parser', description='Determines the entry point type.'
- )
+ 'parser', description='Determines the entry point type.', hidden=True
+ ) # type: ignore[call-overload]
level: int = Field(
0,
description="""
@@ -229,8 +234,8 @@ class ExampleUploadEntryPoint(EntryPoint):
"""Base model for example upload plugin entry points."""
entry_point_type: Literal['example_upload'] = Field(
- 'example_upload', description='Determines the entry point type.'
- )
+ 'example_upload', description='Determines the entry point type.', hidden=True
+ ) # type: ignore[call-overload]
category: str | None = Field(description='Category for the example upload.')
title: str | None = Field(description='Title of the example upload.')
description: str | None = Field(
@@ -253,7 +258,8 @@ class ExampleUploadEntryPoint(EntryPoint):
from_examples_directory: bool = Field(
False,
description='Whether this example upload should be read from the "examples" directory.',
- )
+ hidden=True,
+ ) # type: ignore[call-overload]
def get_package_path(self):
"""Once all built-in example uploads have been removed, this function
@@ -408,8 +414,8 @@ class APIEntryPoint(EntryPoint, metaclass=ABCMeta):
"""Base model for API plugin entry points."""
entry_point_type: Literal['api'] = Field(
- 'api', description='Specifies the entry point type.'
- )
+ 'api', description='Specifies the entry point type.', hidden=True
+ ) # type: ignore[call-overload]
prefix: str = Field(
None,
diff --git a/nomad/config/models/ui.py b/nomad/config/models/ui.py
index 5d70e7332a668c936d7766768dd83ed0177861c0..345ead0c6d53a160561c17d0f60c183e7d3ea545 100644
--- a/nomad/config/models/ui.py
+++ b/nomad/config/models/ui.py
@@ -362,8 +362,8 @@ class RowActionURL(RowAction):
description="""JMESPath pointing to a path in the archive that contains the URL."""
)
type: Literal['url'] = Field(
- 'url', description='Set as `url` to get this widget type.'
- )
+ 'url', description='Set as `url` to get this action type.', hidden=True
+ ) # type: ignore[call-overload]
@model_validator(mode='before')
@classmethod
@@ -382,8 +382,8 @@ class RowActionNorth(RowAction):
)
tool_name: str = Field(description="""Name of the NORTH tool to open.""")
type: Literal['north'] = Field(
- 'north', description='Set as `north` to get this widget type.'
- )
+ 'north', description='Set as `north` to get this action type.', hidden=True
+ ) # type: ignore[call-overload]
@model_validator(mode='before')
@classmethod
@@ -583,8 +583,8 @@ class TermsBase(ConfigBaseModel):
)
search_quantity: str = Field(description='The targeted search quantity.')
type: Literal['terms'] = Field(
- description='Set as `terms` to get this type.',
- )
+ description='Set as `terms` to get this type.', hidden=True
+ ) # type: ignore[call-overload]
scale: ScaleEnum = Field(ScaleEnum.LINEAR, description='Statistics scaling.')
show_input: bool = Field(True, description='Whether to show text input field.')
showinput: bool | None = Field(
@@ -618,8 +618,8 @@ class HistogramBase(ConfigBaseModel):
"""Base model for configuring histogram components."""
type: Literal['histogram'] = Field(
- description='Set as `histogram` to get this widget type.'
- )
+ description='Set as `histogram` to get this widget type.', hidden=True
+ ) # type: ignore[call-overload]
quantity: str | None = Field(
None,
deprecated='The "quantity" field is deprecated, use "x.search_quantity" instead.',
@@ -698,8 +698,8 @@ class PeriodicTableBase(ConfigBaseModel):
"""Base model for configuring periodic table components."""
type: Literal['periodic_table'] = Field(
- description='Set as `periodic_table` to get this widget type.'
- )
+ description='Set as `periodic_table` to get this widget type.', hidden=True
+ ) # type: ignore[call-overload]
quantity: str | None = Field(
None,
deprecated='The "quantity" field is deprecated, use "search_quantity" instead.',
@@ -813,8 +813,8 @@ class MenuItemVisibility(MenuItem):
"""Menu item that shows a radio button that can be used to change the visiblity."""
type: Literal['visibility'] = Field(
- description='Set as `visibility` to get this menu item type.',
- )
+ description='Set as `visibility` to get this menu item type.', hidden=True
+ ) # type: ignore[call-overload]
@model_validator(mode='before')
@classmethod
@@ -829,8 +829,8 @@ class MenuItemDefinitions(MenuItem):
"""Menu item that shows a tree for filtering data by the presence of definitions."""
type: Literal['definitions'] = Field(
- description='Set as `definitions` to get this menu item type.',
- )
+ description='Set as `definitions` to get this menu item type.', hidden=True
+ ) # type: ignore[call-overload]
@model_validator(mode='before')
@classmethod
@@ -845,8 +845,8 @@ class MenuItemOptimade(MenuItem):
"""Menu item that shows a dialog for entering OPTIMADE queries."""
type: Literal['optimade'] = Field(
- description='Set as `optimade` to get this menu item type.',
- )
+ description='Set as `optimade` to get this menu item type.', hidden=True
+ ) # type: ignore[call-overload]
@model_validator(mode='before')
@classmethod
@@ -865,7 +865,8 @@ class MenuItemCustomQuantities(MenuItem):
type: Literal['custom_quantities'] = Field(
description='Set as `custom_quantities` to get this menu item type.',
- )
+ hidden=True,
+ ) # type: ignore[call-overload]
@model_validator(mode='before')
@classmethod
@@ -900,8 +901,8 @@ class MenuItemNestedObject(MenuItem):
"""
type: Literal['nested_object'] = Field(
- description='Set as `nested_object` to get this menu item type.',
- )
+ description='Set as `nested_object` to get this menu item type.', hidden=True
+ ) # type: ignore[call-overload]
path: str = Field(
description='Path of the nested object. Typically a section name.'
)
@@ -944,8 +945,8 @@ class Menu(MenuItem):
"""
type: Literal['menu'] = Field(
- description='Set as `nested_object` to get this menu item type.',
- )
+ description='Set as `nested_object` to get this menu item type.', hidden=True
+ ) # type: ignore[call-overload]
size: MenuSizeEnum | str | None = Field(
MenuSizeEnum.SM,
description="""
@@ -1063,32 +1064,32 @@ class WidgetTerms(Widget, TermsBase):
"""Terms widget configuration."""
type: Literal['terms'] = Field(
- description='Set as `terms` to get this type.',
- )
+ description='Set as `terms` to get this widget type.', hidden=True
+ ) # type: ignore[call-overload]
class WidgetHistogram(Widget, HistogramBase):
"""Histogram widget configuration."""
type: Literal['histogram'] = Field(
- description='Set as `histogram` to get this type.',
- )
+ description='Set as `histogram` to get this widget type.', hidden=True
+ ) # type: ignore[call-overload]
class WidgetPeriodicTable(Widget, PeriodicTableBase):
"""Periodic table widget configuration."""
type: Literal['periodic_table'] = Field(
- description='Set as `periodic_table` to get this type.',
- )
+ description='Set as `periodic_table` to get this widget type.', hidden=True
+ ) # type: ignore[call-overload]
class WidgetPeriodicTableDeprecated(WidgetPeriodicTable):
"""Deprecated copy of WidgetPeriodicTable with a misspelled type."""
type: Literal['periodictable'] = Field( # type: ignore[assignment]
- description='Set as `periodictable` to get this widget type.'
- )
+ description='Set as `periodictable` to get this widget type.', hidden=True
+ ) # type: ignore[call-overload]
@model_validator(mode='before')
@classmethod
@@ -1117,8 +1118,8 @@ class WidgetScatterPlot(Widget):
"""Scatter plot widget configuration."""
type: Literal['scatter_plot'] = Field(
- description='Set as `scatter_plot` to get this widget type.'
- )
+ description='Set as `scatter_plot` to get this widget type.', hidden=True
+ ) # type: ignore[call-overload]
x: AxisLimitedScale | str = Field(
description='Configures the information source and display options for the x-axis.'
)
@@ -1182,8 +1183,8 @@ class WidgetScatterPlotDeprecated(WidgetScatterPlot):
"""Deprecated copy of WidgetScatterPlot with a misspelled type."""
type: Literal['scatterplot'] = Field( # type: ignore[assignment]
- description='Set as `scatterplot` to get this widget type.'
- )
+ description='Set as `scatterplot` to get this type.', hidden=True
+ ) # type: ignore[call-overload]
@model_validator(mode='before')
@classmethod
diff --git a/nomad/mkdocs/__init__.py b/nomad/mkdocs/__init__.py
index e0fbf44fac3c772faa9a1a8aff5545b0e29e720a..7318ff5a494cf3a0f68f98bef81501372ad9d418 100644
--- a/nomad/mkdocs/__init__.py
+++ b/nomad/mkdocs/__init__.py
@@ -26,7 +26,7 @@ import yaml
import json
import os.path
-from typing import get_args
+from typing import get_args, cast
from inspect import isclass
@@ -197,6 +197,12 @@ def define_env(env):
return '</br>'.join(result)
def field_row(name: str, field: FieldInfo):
+ # The field is not shown in the docs if it has the 'hidden' flag set to True
+ if (
+ field.json_schema_extra
+ and cast(dict, field.json_schema_extra).get('hidden', False) is True
+ ):
+ return ''
if name.startswith('m_') or field is None:
return ''
type_name, classes = get_field_type_info(field)
@@ -218,7 +224,7 @@ def define_env(env):
result += '|----|----|-|\n'
if isinstance(fields, tuple):
# handling union types
- results = []
+ results: list[str] = []
for field in fields:
if hasattr(field, 'model_fields'):
# if the field is a pydantic model, generate the documentation for that model
@@ -226,15 +232,14 @@ def define_env(env):
elif "<class 'NoneType'>" not in str(field):
# the check is a bit awkward but checking for None directly falls through
results.append(field_row(name, field))
- result = ''.join(results)
else:
- result += ''.join(
- [
- field_row(name, field)
- for name, field in fields.items()
- if name not in hide
- ]
- )
+ results = [
+ field_row(name, field)
+ for name, field in fields.items()
+ if name not in hide
+ ]
+ results = sorted(results, key=lambda x: 'None' in x)
+ result += ''.join(results)
for required_model in required_models:
if required_model.__name__ not in exported_config_models:
diff --git a/tests/examples/test_examples.py b/tests/examples/test_examples.py
new file mode 100644
index 0000000000000000000000000000000000000000..c29e72dfc9c935c56e4b926ffbb63de12ace9df3
--- /dev/null
+++ b/tests/examples/test_examples.py
@@ -0,0 +1,46 @@
+import os
+import runpy
+
+import pytest
+
+
+@pytest.mark.parametrize(
+ 'path',
+ [
+ 'examples/metainfo/data_frames.py',
+ 'examples/plugins',
+ ],
+)
+def test_metainfo(path):
+ """Runs the python files(s) in the given path."""
+ abs_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '../../', path))
+ if os.path.isdir(abs_path):
+ files = find_py_files(abs_path)
+ else:
+ files = [abs_path]
+ for file in files:
+ runpy.run_path(file)
+
+
+def find_py_files(directory):
+ """
+ Recursively traverses the given directory and returns a list of absolute paths for all .py files.
+
+ Args:
+ directory (str): The path of the directory to traverse.
+
+ Returns:
+ list: A list of absolute paths to .py files.
+ """
+ # Iterate over all entries in the current directory
+ py_files = []
+ with os.scandir(directory) as entries:
+ for entry in entries:
+ # If it's a file and ends with '.py', add its absolute path
+ if entry.is_file() and entry.name.endswith('.py'):
+ py_files.append(os.path.abspath(entry.path))
+ # If it's a directory, recursively search it
+ elif entry.is_dir():
+ py_files.extend(find_py_files(entry.path))
+
+ return py_files
diff --git a/tests/examples/test_metainfo.py b/tests/examples/test_metainfo.py
deleted file mode 100644
index 739280867837e9d5cf4ef9ccf9d6fb1844f29468..0000000000000000000000000000000000000000
--- a/tests/examples/test_metainfo.py
+++ /dev/null
@@ -1,16 +0,0 @@
-import os
-import runpy
-
-import pytest
-
-prefix = os.path.join(__file__, '../../../examples/metainfo')
-
-
-@pytest.mark.parametrize(
- 'file',
- [
- f'{prefix}/data_frames.py',
- ],
-)
-def test_metainfo(file):
- runpy.run_path(file)