diff --git a/docs/howto/plugins/example_uploads.md b/docs/howto/plugins/example_uploads.md
new file mode 100644
index 0000000000000000000000000000000000000000..41b441f23826249442f68c517576214184aaa4ad
--- /dev/null
+++ b/docs/howto/plugins/example_uploads.md
@@ -0,0 +1,91 @@
+# How to write an example upload
+
+Example uploads can be used to add representative collections of data for your plugin. Example uploads are available for end-users in the *Uploads*-page under the *Add example uploads*-button. There users can instantiate an example upload with a click. This can be very useful for educational or demonstration purposes but also for testing.
+
+This documentation shows you how to write a plugin entry point for an example upload. You should read the [documentation on getting started with plugins](./plugins.md) to have a basic understanding of how plugins and plugin entry points work in the NOMAD ecosystem.
+
+## Getting started
+
+You can use our [template repository](https://github.com/FAIRmat-NFDI/nomad-plugin-template) to create an initial structure for a plugin containing an app. The relevant part of the repository layout will look something like this:
+
+```txt
+nomad-example
+ ├── src
+ │ ├── nomad_example
+ │ │ ├── example_uploads
+ │ │ │ ├── getting_started
+ │ │ │ ├── __init__.py
+ ├── LICENSE.txt
+ ├── README.md
+ ├── MANIFEST.in
+ └── pyproject.toml
+```
+
+See the documentation on [plugin development guidelines](./plugins.md#plugin-development-guidelines) for more details on the best development practices for plugins, including linting, testing and documenting.
+
+## Example upload entry point
+
+The entry point defines basic information about your example upload and is used to automatically load the associated data into a NOMAD distribution. It is an instance of a `ExampleUploadEntryPoint` or its subclass and it contains a `load` method which will prepare the data that should be contained in the upload. The entry point should be defined in `*/example_uploads/__init__.py` like this:
+
+```python
+from nomad.config.models.plugins import ExampleUploadEntryPoint
+
+myexampleupload = ExampleUploadEntryPoint(
+ title = 'My Example Upload',
+ category = 'Examples',
+ description = 'Description of this example upload.',
+ path='example_uploads/getting_started
+)
+```
+
+Here we instantiate an object `myexampleupload`. This is the final entry point instance in which you specify the default parameterization and other details about the example upload. By default, it is assumed that you include the upload data in the plugin repository and use the `path` field to specify the location with respect to the package root directory. You can learn more about different data loading options in the next section. In the reference you can also see all of the available [configuration options for a `ExampleUploadEntryPoint`](../../reference/plugins.md#exampleuploadentrypoint).
+
+The entry point instance should then be added to the `[project.entry-points.'nomad.plugin']` table in `pyproject.toml` in order for the example upload to be automatically detected:
+
+```toml
+[project.entry-points.'nomad.plugin']
+myexampleupload = "nomad_example.example_uploads:myexampleupload"
+```
+
+## Including data in an example upload
+
+There are three main ways to include data in an example upload:
+
+1. Data stored directly in the plugin package using `path`:
+
+ This is the default method that assumes you simply store the data under a path in the plugin source code. This is very convenient if you have relative small example data and wish to track this in version control. The path should be given relative to the package installation location (`src/<package-name>`), and you should ensure that the data is distributed with your Python package. Distribution of additional data files in Python packages is controlled with the `MANIFEST.in` file. If you create a plugin with our [template](https://github.com/FAIRmat-NFDI/nomad-plugin-template), the `src/<package-name>/example_uploads` folder is included automatically in `MANIFEST.in`. If you later add an example upload entry point to your plugin, remember to include the folder by adding the following line to `MANIFEST.in`:
+
+ ```sh
+ graft src/<package-name>/<path>
+ ```
+
+2. Data retrieved online during app startup using `url`:
+
+ If your example uploads are very large (>100MB), storing them in Git may become unpractical. In order to deal with larger uploads, they can be stored in a separate online service. To load such external resoruces, you can instead of path specify an `url` parameter of to activate online data retrieval. This will retrieve the large online file *once* upon the first app launch and then store it for later use:
+
+ ```python
+ from nomad.config.models.plugins import ExampleUloadEntryPoint
+
+ myexampleupload = ExampleUploadEntryPoint(
+ name = 'MyExampleUpload',
+ description = 'My custom example upload.',
+ url='http://my_large_file_address.zip
+ )
+ ```
+
+3. Data retrieved with a custom method: If the above options do not suite your use case, you can also override the `load`-method of `ExampleUploadEntryPoint` to perform completely custom data loading logic.
+
+ ```python
+ from pydantic import Field
+ from nomad.config.models.plugins import ExampleUploadEntryPoint
+
+
+ class MyExampleUploadEntryPoint(ExampleUploadEntryPoint):
+
+ def load(self):
+ """Add your custom loading logic here. Note that the loaded data
+ should be saved in the package installation directory in order to be
+ accessible. Check the default `load` function for more details.
+ """
+ ...
+ ```
\ No newline at end of file
diff --git a/docs/howto/plugins/plugins.md b/docs/howto/plugins/plugins.md
index e9bdb7190b2467e0b4a919c29a901035050de30b..964ab90f43534fd40e0475de40ad841bfdd58345 100644
--- a/docs/howto/plugins/plugins.md
+++ b/docs/howto/plugins/plugins.md
@@ -43,6 +43,7 @@ In the folder structure you can see that a single plugin can contain multiple ty
Plugin entry points represent different types of customizations that can be added to a NOMAD installation. The following plugin entry point types are currently supported:
- [Apps](./apps.md)
+ - [Example uploads](./example_uploads.md)
- [Normalizers](./parsers.md)
- [Parsers](./parsers.md)
- [Schema packages](./schema_packages.md)
@@ -59,7 +60,7 @@ myapp = "nomad_example.parsers:myapp"
mypackage = "nomad_example.schema_packages:mypackage"
```
-Here it is important to use the `nomad.plugin` group name in the `project.entry-points` header. The plugin name used on the left side (`mypackage`) can be arbitrary, what matters is that the key (`"nomad_example.schema_packages:mypackage"`) is a path pointing to a plugin entry point instance inside the python code. This unique key will be used to identify the plugin entry point when e.g. accessing it to read some of it's configuration values.
+Here it is important to use the `nomad.plugin` group name in the `project.entry-points` header. The value on the right side (`"nomad_example.schema_packages:mypackage"`) must be a path pointing to a plugin entry point instance inside the python code. This unique key will be used to identify the plugin entry point when e.g. accessing it to read some of it's configuration values. The name on the left side (`mypackage`) can be set freely.
You can read more about how to write different types of entry points in their dedicated documentation pages or learn more about the [Python entry point mechanism](https://setuptools.pypa.io/en/latest/userguide/entry_point.html).
diff --git a/docs/reference/plugins.md b/docs/reference/plugins.md
index 3b5ad54f4a349e9ee2d885e34e431156caa7616b..1178e2eec3fb6b925f73ba50791d5a9a6fd5413c 100644
--- a/docs/reference/plugins.md
+++ b/docs/reference/plugins.md
@@ -11,6 +11,7 @@ Plugins allow one to add Python-based functionality to NOMAD without a custom NO
This is a list of the available plugin entry point configuration models.
{{ pydantic_model('nomad.config.models.plugins.AppEntryPoint') }}
+{{ pydantic_model('nomad.config.models.plugins.ExampleUploadEntryPoint') }}
{{ pydantic_model('nomad.config.models.plugins.NormalizerEntryPoint') }}
{{ pydantic_model('nomad.config.models.plugins.ParserEntryPoint') }}
{{ pydantic_model('nomad.config.models.plugins.SchemaPackageEntryPoint') }}
diff --git a/examples/data/cow_tutorial/nomad-countries/pyproject.toml b/examples/data/cow_tutorial/nomad-countries/pyproject.toml
index b750f3df27fd2cc6e8da005d94cac214e7de891f..e60ef9d1814c231b620d4aae4ef859dafd68f0f4 100644
--- a/examples/data/cow_tutorial/nomad-countries/pyproject.toml
+++ b/examples/data/cow_tutorial/nomad-countries/pyproject.toml
@@ -74,7 +74,7 @@ select = [
"UP",
# isort
"I",
- # pylint
+ # pylint
"PL",
]
diff --git a/gui/src/components/uploads/ExampleUploadButton.js b/gui/src/components/uploads/ExampleUploadButton.js
index c1bb13a877374843f5ea34079db5e45d27ad6f78..6f58a809aef51fedf0cf0b30dd91ee09b62e2844 100644
--- a/gui/src/components/uploads/ExampleUploadButton.js
+++ b/gui/src/components/uploads/ExampleUploadButton.js
@@ -94,7 +94,7 @@ export default function ExampleUploadButton(props) {
const handleClose = () => setOpen(false)
const handleSelect = (exampleUpload) => {
- api.post(`/uploads?local_path=${exampleUpload.path}&upload_name=${exampleUpload.title}`)
+ api.post(`/uploads?local_path=${exampleUpload.local_path || exampleUpload.path}&upload_name=${exampleUpload.title}`)
.then((data) => {
history.push(`/user/uploads/upload/id/${data.upload_id}`)
})
diff --git a/gui/src/config.js b/gui/src/config.js
index 5341ef93a320207558000f73dad59b7e1c6ca044..bc517a67f39137df7caae7d078bf37e265803700 100644
--- a/gui/src/config.js
+++ b/gui/src/config.js
@@ -122,7 +122,12 @@ export const searchQuantities = window.nomadArtifacts.searchQuantities
export const metainfo = window.nomadArtifacts.metainfo
export const parserMetadata = window.nomadArtifacts.parserMetadata
export const toolkitMetadata = window.nomadArtifacts.toolkitMetadata
-export const exampleUploads = window.nomadArtifacts.exampleUploads
+export const exampleUploads = window.nomadArtifacts.exampleUploads || {}
+Object.values(entry_points?.options || [])
+ .filter(entry_point => entry_point.entry_point_type === 'example_upload')
+ .forEach(entry_point => {
+ exampleUploads[entry_point.category] = {...exampleUploads[entry_point.category], [entry_point.id]: entry_point}
+ })
export const northTools = window.nomadArtifacts.northTools
export const unitList = window.nomadArtifacts.unitList
export const unitPrefixes = window.nomadArtifacts.unitPrefixes
diff --git a/mkdocs.yml b/mkdocs.yml
index 3df5e7bbd8ee018147020e63942690d554c4e1da..b88cc64e9cc4b8176642556557f2a7fc59c895aa 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -36,6 +36,7 @@ nav:
- Plugins:
- Get started with plugins: howto/plugins/plugins.md
- Write an app: howto/plugins/apps.md
+ - Write an example upload: howto/plugins/example_uploads.md
- Write a normalizer: howto/plugins/normalizers.md
- Write a parser: howto/plugins/parsers.md
- Write a schema package: howto/plugins/schema_packages.md
diff --git a/nomad/app/v1/routers/uploads.py b/nomad/app/v1/routers/uploads.py
index 482171938e1e7e934dfd494ff01e36bf78f2aefb..de1deabdbf690694f884d3b9ec49662947c26edd 100644
--- a/nomad/app/v1/routers/uploads.py
+++ b/nomad/app/v1/routers/uploads.py
@@ -41,10 +41,12 @@ from fastapi.exceptions import RequestValidationError
from nomad import utils, files
from nomad.config import config
+from nomad.config.models.plugins import example_upload_path_prefix
from nomad.files import (
StagingUploadFiles,
is_safe_relative_path,
is_safe_basename,
+ is_safe_path,
PublicUploadFiles,
)
from nomad.bundles import BundleExporter, BundleImporter
@@ -58,6 +60,7 @@ from nomad.processing import (
MetadataEditRequestHandler,
)
from nomad.utils import strip
+from nomad.common import get_package_path
from nomad.search import (
search,
search_iterator,
@@ -1373,6 +1376,13 @@ async def put_upload_raw_path(
upload = _get_upload_with_write_access(upload_id, user, include_published=False)
+ if local_path:
+ if not os.path.isfile(local_path):
+ raise HTTPException(
+ status_code=status.HTTP_400_BAD_REQUEST,
+ detail='Uploading folders with local_path is not yet supported.',
+ )
+
if not is_safe_relative_path(path):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST, detail='Bad path provided.'
@@ -1407,7 +1417,7 @@ async def put_upload_raw_path(
detail='Bad source path provided.',
)
- upload_paths, method = await _get_files_if_provided(
+ upload_paths, _, method = await _get_files_if_provided(
upload_id, request, file, local_path, file_name, user
)
@@ -1842,7 +1852,7 @@ async def post_upload(
upload_id = utils.create_uuid()
- upload_paths, method = await _get_files_if_provided(
+ upload_paths, upload_folders, method = await _get_files_if_provided(
upload_id, request, file, local_path, file_name, user
)
@@ -1870,8 +1880,13 @@ async def post_upload(
if upload_paths:
upload.process_upload(
file_operations=[
- dict(op='ADD', path=upload_path, target_dir='', temporary=(method != 0))
- for upload_path in upload_paths
+ dict(
+ op='ADD',
+ path=upload_path,
+ target_dir=upload_folders[i_path],
+ temporary=(method != 0),
+ )
+ for i_path, upload_path in enumerate(upload_paths)
]
)
@@ -2439,11 +2454,18 @@ async def post_upload_bundle(
bundle_path: str = None
method = None
+ if local_path:
+ if not os.path.isfile(local_path):
+ raise HTTPException(
+ status_code=status.HTTP_400_BAD_REQUEST,
+ detail='You can only target a single bundle file using local_path.',
+ )
+
try:
bundle_importer = BundleImporter(user, import_settings)
bundle_importer.check_api_permissions()
- bundle_paths, method = await _get_files_if_provided(
+ bundle_paths, _, method = await _get_files_if_provided(
tmp_dir_prefix='bundle',
request=request,
file=file,
@@ -2497,7 +2519,7 @@ async def _get_files_if_provided(
local_path: str,
file_name: str,
user: User,
-) -> Tuple[List[str], Union[None, int]]:
+) -> Tuple[List[str], List[str], Union[None, int]]:
"""
If the user provides one or more files with the api call, load and save them to a temporary
folder (or, if method 0 is used, just "forward" the file path). The method thus needs to identify
@@ -2509,22 +2531,38 @@ async def _get_files_if_provided(
# Determine the source data stream
sources: List[Tuple[Any, str]] = [] # List of tuples (source, filename)
if local_path:
- # Method 0: Local file - only for local path under /tests/data/proc/
- if not local_path.startswith('examples') and not user.is_admin:
- raise HTTPException(
- status_code=status.HTTP_401_UNAUTHORIZED,
- detail=strip(
- """
- You are not authorized to access this path."""
- ),
- )
- if not os.path.exists(local_path) or not os.path.isfile(local_path):
+ # Method 0: Local file - only for local path under examples/data, an
+ # example upload stored within a plugin package or or admin use.
+
+ # The legacy example uploads are stored as zip files under
+ # examples/data.
+ safe_path = None
+ if local_path.startswith('examples'):
+ safe_path = os.path.abspath('examples/data')
+ # The example uploads distributed through plugins are accessible
+ # through the use of a special path prefix.
+ elif local_path.startswith(example_upload_path_prefix):
+ parts = local_path.split('/')
+ package_name = parts[1]
+ safe_path = get_package_path(package_name)
+ local_path = os.path.join(safe_path, '/'.join(parts[2:]))
+
+ # Check if local path points to a safe location
+ if not user.is_admin:
+ if not safe_path or not is_safe_path(local_path, safe_path):
+ raise HTTPException(
+ status_code=status.HTTP_401_UNAUTHORIZED,
+ detail=strip("""
+ You are not authorized to access this path.
+ """),
+ )
+
+ if not os.path.exists(local_path):
raise HTTPException(
status_code=status.HTTP_400_BAD_REQUEST,
- detail=strip(
- """
- The specified local_path cannot be found or is not a file."""
- ),
+ detail=strip("""
+ The specified local_path cannot be found.
+ """),
)
method = 0
elif file:
@@ -2553,15 +2591,28 @@ async def _get_files_if_provided(
# Forward the file path (if method == 0) or save the file(s)
if method == 0:
- tmp_dir = files.create_tmp_dir(tmp_dir_prefix)
- # copy provided path to a temp directory
- shutil.copy(local_path, tmp_dir)
- upload_paths = [os.path.join(tmp_dir, os.path.basename(local_path))]
- uploaded_bytes = os.path.getsize(local_path)
+ is_file = os.path.isfile(local_path)
+ # Single file
+ if is_file:
+ upload_paths = [local_path]
+ upload_folders = ['']
+ # Folder
+ else:
+ upload_paths = []
+ upload_folders = []
+ for root, _, filepaths in os.walk(local_path):
+ for uploaded_file in filepaths:
+ file_path = os.path.abspath(os.path.join(root, uploaded_file))
+ folder = os.path.relpath(root, local_path)
+ if folder == '.':
+ folder = ''
+ upload_paths.append(file_path)
+ upload_folders.append(folder)
else:
tmp_dir = files.create_tmp_dir(tmp_dir_prefix)
upload_paths = []
uploaded_bytes = 0
+ upload_folders = []
for source_stream, source_file_name in sources:
upload_path = os.path.join(tmp_dir, source_file_name)
try:
@@ -2593,11 +2644,12 @@ async def _get_files_if_provided(
detail='Some IO went wrong, upload probably aborted/disrupted.',
)
upload_paths.append(upload_path)
+ upload_folders.append('')
if not uploaded_bytes and method == 2:
# No data was provided
shutil.rmtree(tmp_dir)
- return [], None
+ return [], [], None
logger.info(f'received uploaded file(s)')
if method == 2 and no_file_name_info_provided:
@@ -2620,8 +2672,9 @@ async def _get_files_if_provided(
# Add the correct extension
shutil.move(upload_path, upload_path + ext)
upload_paths = [upload_path + ext]
+ upload_folders = ['']
- return upload_paths, method
+ return upload_paths, upload_folders, method
async def _asyncronous_file_reader(f):
diff --git a/nomad/cli/dev.py b/nomad/cli/dev.py
index 9aa482ec6ff2270e204b97b9f95618970aa8b830..61cab37299f54d3da8bee7c628b06dc4a585a503 100644
--- a/nomad/cli/dev.py
+++ b/nomad/cli/dev.py
@@ -22,6 +22,7 @@ import os
import click
from nomad.config import config
+from nomad.config.models.plugins import ExampleUploadEntryPoint
from nomad.metainfo.elasticsearch_extension import schema_separator
from .cli import cli
@@ -209,14 +210,13 @@ def parser_metadata():
def get_gui_config() -> str:
"""Create a simplified and stripped version of the nomad.yaml contents that
is used by the GUI.
-
- Args:
- proxy: Whether the build is using a proxy. Affects whether calls to different
- services use an explicit host+port+path as configured in the config, or whether
- they simply use a relative path that a proxy can resolve.
"""
from nomad.config import config
+ config.load_plugins()
+
+ # We need to sort the plugin entry point information, because otherwise the
+ # artifacts tests will fail.
def _sort_dict(d):
if isinstance(d, dict):
return {k: _sort_dict(v) for k, v in sorted(d.items())}
@@ -224,8 +224,15 @@ def get_gui_config() -> str:
return [_sort_dict(v) for v in d]
return d
- config.load_plugins()
plugins = _sort_dict(config.plugins.dict(exclude_unset=True))
+
+ # We need to load the example upload entry points to get the final upload
+ # path
+ enabled_entry_points = config.plugins.entry_points.filtered_values()
+ for entry_point in enabled_entry_points:
+ if isinstance(entry_point, ExampleUploadEntryPoint):
+ entry_point.load()
+
for key in plugins['entry_points']['options'].keys():
plugins['entry_points']['options'][key] = config.plugins.entry_points.options[
key
diff --git a/nomad/common.py b/nomad/common.py
new file mode 100644
index 0000000000000000000000000000000000000000..f4aa922eeab63604d269c6602586af0530e55ac4
--- /dev/null
+++ b/nomad/common.py
@@ -0,0 +1,45 @@
+#
+# Copyright The NOMAD Authors.
+#
+# This file is part of NOMAD. See https://nomad-lab.eu for further info.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import pkgutil
+
+
+def get_package_path(package_name: str) -> str:
+ """Given a python package name, returns the filepath of the package root folder."""
+ package_path = None
+ try:
+ # We try to deduce the package path from the top-level package
+ package_path_segments = package_name.split('.')
+ root_package = package_path_segments[0]
+ package_dirs = package_path_segments[1:]
+ package_path = os.path.join(
+ os.path.dirname(
+ pkgutil.get_loader(root_package).get_filename() # type: ignore
+ ),
+ *package_dirs,
+ )
+ if not os.path.isdir(package_path):
+ # We could not find it this way. Let's try to official way
+ package_path = os.path.dirname(
+ pkgutil.get_loader(package_name).get_filename() # type: ignore
+ )
+ except Exception as e:
+ raise ValueError(f'The python package {package_name} cannot be loaded.', e)
+
+ return package_path
diff --git a/nomad/config/models/config.py b/nomad/config/models/config.py
index 21c8e0c74848e2af3c45fafa25a461cfaa573684..7da72463c310aedd6248022e7167fe762c17e575 100644
--- a/nomad/config/models/config.py
+++ b/nomad/config/models/config.py
@@ -18,7 +18,6 @@
import logging
import os
-import pkgutil
import sys
import warnings
from importlib.metadata import metadata, version
@@ -46,6 +45,7 @@ from .common import (
from .north import NORTH
from .plugins import EntryPointType, PluginPackage, Plugins
from .ui import UI
+from nomad.common import get_package_path
warnings.filterwarnings('ignore', message='numpy.dtype size changed')
warnings.filterwarnings('ignore', message='numpy.ufunc size changed')
@@ -1084,7 +1084,7 @@ class Config(ConfigBaseModel):
package_metadata = metadata(package_name)
url_list = package_metadata.get_all('Project-URL')
url_dict = {}
- for url in url_list:
+ for url in url_list or []:
name, value = url.split(',')
url_dict[name.lower()] = value.strip()
if package_name not in plugin_packages:
@@ -1154,28 +1154,3 @@ class Config(ConfigBaseModel):
)
self.plugins = Plugins.parse_obj(_plugins)
-
-
-def get_package_path(package_name: str) -> str:
- """Given a python package name, returns the filepath of the package root folder."""
- package_path = None
- try:
- # We try to deduce the package path from the top-level package
- package_path_segments = package_name.split('.')
- root_package = package_path_segments[0]
- package_dirs = package_path_segments[1:]
- package_path = os.path.join(
- os.path.dirname(
- pkgutil.get_loader(root_package).get_filename() # type: ignore
- ),
- *package_dirs,
- )
- if not os.path.isdir(package_path):
- # We could not find it this way. Let's try to official way
- package_path = os.path.dirname(
- pkgutil.get_loader(package_name).get_filename() # type: ignore
- )
- except Exception as e:
- raise ValueError(f'The python package {package_name} cannot be loaded.', e)
-
- return package_path
diff --git a/nomad/config/models/plugins.py b/nomad/config/models/plugins.py
index af199bddb751d73225b4ea8863dc075aa2eb4e0e..fb8e94e7dc60a5315c61489e1575d5ed54a439e6 100644
--- a/nomad/config/models/plugins.py
+++ b/nomad/config/models/plugins.py
@@ -16,15 +16,23 @@
# limitations under the License.
#
+import os
import sys
+import shutil
+from tempfile import TemporaryDirectory
from abc import ABCMeta, abstractmethod
import importlib
from typing import Optional, Dict, Union, List, Literal, TYPE_CHECKING
-from pydantic import BaseModel, Field
+import requests
+from pydantic import BaseModel, Field, root_validator
+
+from nomad.common import get_package_path
from .common import Options
from .ui import App
+example_upload_path_prefix = '__example_uploads__'
+
if TYPE_CHECKING:
from nomad.metainfo import SchemaPackage
from nomad.normalizing import Normalizer as NormalizerBaseClass
@@ -54,7 +62,7 @@ class EntryPoint(BaseModel):
class AppEntryPoint(EntryPoint):
- """Base model for a app plugin entry points."""
+ """Base model for app plugin entry points."""
entry_point_type: Literal['app'] = Field(
'app', description='Determines the entry point type.'
@@ -183,6 +191,103 @@ class ParserEntryPoint(EntryPoint, metaclass=ABCMeta):
return self.dict(include=ParserEntryPoint.__fields__.keys(), exclude_none=True)
+class ExampleUploadEntryPoint(EntryPoint):
+ """Base model for example upload plugin entry points."""
+
+ entry_point_type: Literal['example_upload'] = Field(
+ 'example_upload', description='Determines the entry point type.'
+ )
+ category: str = Field(description='Category for the example upload.')
+ title: str = Field(description='Title of the example upload.')
+ description: str = Field(description='Longer description of the example upload.')
+ path: Optional[str] = Field(
+ description="""
+ Path to the example upload contents folder. Should be a path that starts
+ from the package root folder, e.g. 'example_uploads/my_upload'.
+ """,
+ )
+ url: Optional[str] = Field(
+ description="""
+ URL that points to an online file. If you use this instead of 'path',
+ the file will be downloaded once upon app startup.
+ """,
+ )
+ local_path: Optional[str] = Field(
+ description="""
+ The final path to use when creating the upload. This field will be
+ automatically generated by the 'load' function.
+ """
+ )
+
+ @root_validator(pre=True)
+ def _validate(cls, values):
+ """Checks that only either path or url is given."""
+ path = values.get('path')
+ url = values.get('url')
+ if path and url:
+ raise ValueError('Provide only "path" or "url", not both.')
+ if not path and not url:
+ raise ValueError('Provide either "path" or "url".')
+
+ return values
+
+ def load(self) -> None:
+ """Used to initialize the example upload data and set the final upload
+ path.
+
+ By default this function will simply populate the 'upload_path' variable
+ based on 'path' or 'url', but you may also overload this function in
+ your plugin to perform completely custom data loading, after which you
+ set the 'upload_path'. Note that you should store any data you fetch
+ directly into your plugin installation folder (you can use
+ 'get_package_path' to fetch the location), because only that folder is
+ accessible by the upload API. This function is called once upon app
+ startup.
+ """
+ path = self.path
+ if not path and self.url:
+ final_folder = os.path.join(
+ get_package_path(self.plugin_package), 'example_uploads'
+ )
+ filename = self.url.rsplit('/')[-1]
+ final_filepath = os.path.join(final_folder, filename)
+
+ if not os.path.exists(final_filepath):
+ try:
+ with requests.get(self.url, stream=True) as response:
+ response.raise_for_status()
+ # Download into a temporary directory to ensure the integrity of
+ # the download
+ with TemporaryDirectory() as tmp_folder:
+ tmp_filepath = os.path.join(tmp_folder, filename)
+ with open(tmp_filepath, mode='wb') as file:
+ for chunk in response.iter_content(
+ chunk_size=10 * 1024
+ ):
+ file.write(chunk)
+ # If download has succeeeded, copy the files over to
+ # final location
+ os.makedirs(final_folder, exist_ok=True)
+ shutil.copy(tmp_filepath, final_filepath)
+ except requests.RequestException as e:
+ raise ValueError(
+ f'Could not fetch the example upload from URL: {self.url}'
+ ) from e
+
+ path = os.path.join('example_uploads', filename)
+
+ if not path:
+ raise ValueError('No valid path or URL provided for example upload.')
+
+ prefix = os.path.join(example_upload_path_prefix, self.plugin_package)
+ self.local_path = os.path.join(prefix, path)
+
+ def dict_safe(self):
+ return self.dict(
+ include=ExampleUploadEntryPoint.__fields__.keys(), exclude_none=True
+ )
+
+
class PluginBase(BaseModel):
"""
Base model for a NOMAD plugin.
@@ -420,6 +525,7 @@ EntryPointType = Union[
ParserEntryPoint,
NormalizerEntryPoint,
AppEntryPoint,
+ ExampleUploadEntryPoint,
]
diff --git a/nomad/config/models/ui.py b/nomad/config/models/ui.py
index 705ebd18af024ffec6181aaed52aad25a8063605..1955350c4d37e559c80fe2ce2d9b04af2ee76db6 100644
--- a/nomad/config/models/ui.py
+++ b/nomad/config/models/ui.py
@@ -17,7 +17,6 @@
#
from enum import Enum
-import inspect
from typing import List, Dict, Union, Optional
from typing_extensions import Literal, Annotated
from pydantic import Field, root_validator
@@ -94,7 +93,7 @@ class UnitSystem(ConfigBaseModel):
)
@root_validator(pre=True)
- def __validate(cls, values): # pylint: disable=no-self-argument
+ def _validate(cls, values): # pylint: disable=no-self-argument
"""Adds SI defaults for dimensions that are missing a unit."""
units = values.get('units', {})
from nomad.units import ureg
@@ -558,7 +557,7 @@ class WidgetHistogram(Widget):
)
@root_validator(pre=True)
- def __validate(cls, values):
+ def _validate(cls, values):
"""Ensures backwards compatibility for quantity and scale."""
# X-axis
x = values.get('x', {})
@@ -628,7 +627,7 @@ class WidgetScatterPlot(Widget):
)
@root_validator(pre=True)
- def __validate(cls, values):
+ def _validate(cls, values):
"""Ensures backwards compatibility of x, y, and color."""
color = values.get('color')
if color is not None:
diff --git a/nomad/files.py b/nomad/files.py
index de4776bf1fed385536f8b5f0d56efae874662da5..5e8b0e988dd677341de43c03ee4f5c1ca5bc1ae5 100644
--- a/nomad/files.py
+++ b/nomad/files.py
@@ -173,6 +173,29 @@ def is_safe_basename(basename: str) -> bool:
return True
+def is_safe_path(path: str, safe_path: str, is_directory=True) -> bool:
+ """Returns whether the given path ultimately points to a known safe
+ location. Can be used to prevent path traversal attacks, such as relative
+ paths or symlinks.
+
+ Args:
+ path: The path to check
+ safe_path: A safe location. Can be a folder or a file.
+ is_directory: Whether the safe path is a directory or not. If True,
+ a trailing slash is added and only the common prefix is tested.
+ If False, the whole path must match. Otherwise users may access
+ other locations with the same name prefix (e.g. /safe2 when
+ safe_path was /safe).
+ """
+ real_path = os.path.realpath(path)
+ if is_directory:
+ if not safe_path.endswith(os.path.sep):
+ safe_path += os.path.sep
+ return os.path.commonprefix((real_path, safe_path)) == safe_path
+
+ return real_path == safe_path
+
+
def is_safe_relative_path(path: str) -> bool:
"""
Checks if path is a *safe* relative path. We consider it safe if it does not start with
diff --git a/tests/app/v1/routers/uploads/test_basic_uploads.py b/tests/app/v1/routers/uploads/test_basic_uploads.py
index 49a28800ff0828538e477089c4837b7d447b5e88..3a2f96867344e2f659223f7269a934141d06b401 100644
--- a/tests/app/v1/routers/uploads/test_basic_uploads.py
+++ b/tests/app/v1/routers/uploads/test_basic_uploads.py
@@ -46,6 +46,7 @@ from tests.processing.test_edit_metadata import (
from tests.test_files import (
assert_upload_files,
empty_file,
+ example_directory,
example_file_aux,
example_file_corrupt_zip,
example_file_mainfile_different_atoms,
@@ -231,6 +232,18 @@ def assert_file_upload_and_processing(
target_path_full = os.path.join(target_path, path)
assert upload_files.raw_path_exists(target_path_full)
assert upload_files.raw_path_is_file(target_path_full)
+ elif os.path.isdir(source_path):
+ for root, _, filepaths in os.walk(source_path):
+ for filepath in filepaths:
+ rel_dir = os.path.relpath(root, source_path)
+ path = (
+ filepath
+ if rel_dir == '.'
+ else os.path.join(rel_dir, filepath)
+ )
+ target_path_full = os.path.join(target_path, path)
+ assert upload_files.raw_path_exists(target_path_full)
+ assert upload_files.raw_path_is_file(target_path_full)
else:
if mode == 'stream':
# Must specify file_name
@@ -2984,7 +2997,18 @@ def test_post_upload_edit(
False,
True,
200,
- id='local_path',
+ id='local_path_file',
+ ),
+ pytest.param(
+ 'local_path',
+ 'tests/data/proc/example_upload',
+ dict(upload_name='test_name'),
+ 'user0',
+ False,
+ False,
+ True,
+ 200,
+ id='local_path_folder',
),
pytest.param(
'local_path',
diff --git a/tests/config/test_models.py b/tests/config/models/test_common.py
similarity index 100%
rename from tests/config/test_models.py
rename to tests/config/models/test_common.py
diff --git a/tests/config/models/test_plugins.py b/tests/config/models/test_plugins.py
new file mode 100644
index 0000000000000000000000000000000000000000..c96f2e1c8e1f4f576b2ed2683c29c2858835047c
--- /dev/null
+++ b/tests/config/models/test_plugins.py
@@ -0,0 +1,119 @@
+#
+# Copyright The NOMAD Authors.
+#
+# This file is part of NOMAD. See https://nomad-lab.eu for further info.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import tempfile
+import pytest
+
+from nomad.config.models.plugins import (
+ ExampleUploadEntryPoint,
+ example_upload_path_prefix,
+)
+
+
+def mock_plugin_package(monkeypatch, directory):
+ """Used for mocking the presence of a plugin package installation
+ location."""
+
+ def mock_get_package_path(package_name):
+ return directory
+
+ monkeypatch.setattr(
+ 'nomad.config.models.plugins.get_package_path', mock_get_package_path
+ )
+
+
+@pytest.mark.parametrize(
+ 'config, expected_local_path',
+ [
+ pytest.param(
+ {
+ 'title': 'test',
+ 'description': 'test',
+ 'category': 'test',
+ 'path': 'example_uploads/getting_started',
+ },
+ f'{example_upload_path_prefix}/nomad_test_plugin/example_uploads/getting_started',
+ id='load with path',
+ ),
+ pytest.param(
+ {
+ 'title': 'test',
+ 'description': 'test',
+ 'category': 'test',
+ 'url': 'https://nomad-lab.eu/prod/v1/docs/assets/nomad-oasis.zip',
+ },
+ f'{example_upload_path_prefix}/nomad_test_plugin/example_uploads/nomad-oasis.zip',
+ id='load with url',
+ ),
+ ],
+)
+def test_example_upload_entry_point_valid(config, expected_local_path, monkeypatch):
+ # Create tmp directory that will be used as a mocked package location.
+ with tempfile.TemporaryDirectory() as tmp_dir_path:
+ mock_plugin_package(monkeypatch, tmp_dir_path)
+ config['plugin_package'] = 'nomad_test_plugin'
+ entry_point = ExampleUploadEntryPoint(**config)
+ entry_point.load()
+ assert entry_point.local_path == expected_local_path
+
+
+@pytest.mark.parametrize(
+ 'config, error',
+ [
+ pytest.param(
+ {
+ 'title': 'test',
+ 'description': 'test',
+ 'category': 'test',
+ },
+ 'Provide either "path" or "url".',
+ id='no path or url given',
+ ),
+ pytest.param(
+ {
+ 'title': 'test',
+ 'description': 'test',
+ 'category': 'test',
+ 'path': 'example_uploads/getting_started',
+ 'url': 'https://test.zip',
+ },
+ 'Provide only "path" or "url", not both.',
+ id='path and url both given',
+ ),
+ pytest.param(
+ {
+ 'title': 'test',
+ 'description': 'test',
+ 'category': 'test',
+ 'url': 'https://test.zip',
+ },
+ 'Could not fetch the example upload from URL: https://test.zip',
+ id='cannot find url',
+ ),
+ ],
+)
+def test_example_upload_entry_point_invalid(config, error, monkeypatch):
+ # Create tmp directory that will be used as a mocked package location.
+ with tempfile.TemporaryDirectory() as tmp_dir_path:
+ mock_plugin_package(monkeypatch, tmp_dir_path)
+ config['plugin_package'] = 'nomad_test_plugin'
+ with pytest.raises(Exception) as exc_info:
+ entry_point = ExampleUploadEntryPoint(**config)
+ entry_point.load()
+
+ assert exc_info.match(error)
diff --git a/tests/data/proc/example_upload/a.txt b/tests/data/proc/example_upload/a.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/tests/data/proc/example_upload/a/b.txt b/tests/data/proc/example_upload/a/b.txt
new file mode 100644
index 0000000000000000000000000000000000000000..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
diff --git a/tests/test_files.py b/tests/test_files.py
index 0609b9c05829184185939dcb08f09862bfcac166..f862d87bfa04951b396f8a7a405f8c7f22a113c4 100644
--- a/tests/test_files.py
+++ b/tests/test_files.py
@@ -35,6 +35,7 @@ from nomad.files import (
PathObject,
empty_zip_file_size,
empty_archive_file_size,
+ is_safe_path,
)
from nomad.files import StagingUploadFiles, PublicUploadFiles, UploadFiles
from nomad.processing import Upload
@@ -796,3 +797,52 @@ def test_test_upload_files(raw_files_infra):
finally:
if upload_files.exists():
upload_files.delete()
+
+
+@pytest.mark.parametrize(
+ 'path, safe_path, is_directory, is_safe',
+ [
+ pytest.param(
+ '/safe/a', '/safe/', True, True, id='safe absolute path to folder'
+ ),
+ pytest.param(
+ '/safe/a/../b', '/safe/', True, True, id='safe relative path to folder'
+ ),
+ pytest.param(
+ '/unsafe/../c', '/safe/', True, False, id='unsafe absolute path to folder'
+ ),
+ pytest.param(
+ '/safe/../unsafe',
+ '/safe/',
+ True,
+ False,
+ id='unsafe relative path to folder',
+ ),
+ pytest.param(
+ '/safe2/',
+ '/safe',
+ True,
+ False,
+ id='unsafe absolute path to folder with same prefix',
+ ),
+ pytest.param(
+ '/safe/safe_file.zip', '/safe/safe_file.zip', False, True, id='safe file'
+ ),
+ pytest.param(
+ '/safe2/unsafe_file.zip',
+ '/safe/safe_file.zip',
+ False,
+ False,
+ id='unsafe file',
+ ),
+ pytest.param(
+ '/safe2/safe_file.zip2',
+ '/safe/safe_file.zip',
+ False,
+ False,
+ id='unsafe file with same prefix',
+ ),
+ ],
+)
+def test_is_safe_path(path, safe_path, is_directory, is_safe):
+ assert is_safe_path(path, safe_path, is_directory) == is_safe