Commit 09afff8d authored by Lauri Himanen's avatar Lauri Himanen
Browse files

Merge tag 'v0.8.2' into encyclopedia-api

parents c6b1c77d f522e4b6
Pipeline #78764 passed with stages
in 29 minutes and 51 seconds
[submodule "dependencies/nomad-meta-info"]
path = dependencies/nomad-meta-info
url = https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-meta-info.git
branch = nomad-fair
[submodule "dependencies/python_common"]
path = dependencies/python_common
url = https://gitlab.mpcdf.mpg.de/nomad-lab/python-common.git
......
......@@ -106,26 +106,23 @@ WORKDIR /app
# transfer installed packages from dependency stage
COPY --from=build /usr/local/lib/python3.7/site-packages /usr/local/lib/python3.7/site-packages
RUN echo "copy 1"
# copy the meta-info, since it files are loaded via relative paths. TODO that should change.
COPY --from=build /install/dependencies/nomad-meta-info /app/dependencies/nomad-meta-info
RUN echo "copy 2"
# copy the documentation, its files will be served by the API
COPY --from=build /install/docs/.build /app/docs/.build
RUN echo "copy 3"
RUN echo "copy 2"
# copy the source distribution, its files will be served by the API
COPY --from=build /install/dist /app/dist
RUN echo "copy 4"
RUN echo "copy 3"
# copy the nomad command
COPY --from=build /usr/local/bin/nomad /usr/bin/nomad
RUN echo "copy 5"
RUN echo "copy 4"
# copy the gui
RUN mkdir -p /app/gui
COPY --from=gui_build /app/build /app/gui/build
RUN echo "copy 6"
RUN echo "copy 5"
# copy the encyclopedia gui production code
COPY --from=gui_build /encyclopedia /app/dependencies/encyclopedia-gui/client
RUN rm -rf /app/dependencies/encyclopedia-gui/client/src
RUN echo "copy 7"
RUN echo "copy 6"
RUN mkdir -p /app/.volumes/fs
RUN useradd -ms /bin/bash nomad
......
Subproject commit a2df84b2dfc2c96344e7261580a40d655c904047
Subproject commit 45daba83219defb908044193214acce48baf76e9
Subproject commit d394f7ae375cd2355127abc67e07b1592f18d4f4
Subproject commit 38ed55429bd8c2289471c5dbc162bcdd80b7bb94
Subproject commit 6f0e7ec897284382d8ef30b0ee0372f010ce468f
Subproject commit 124b42b9851329682e11c2f1f3e00056d215bd3b
Subproject commit aa701e8f2780419e7911972c8e2a16d188008902
Subproject commit 4c83e96bcbf9080d431b7b22f1ecabd4b64dec27
Subproject commit cf843eda3e4650bc240eb0d1a1ab5b0cea6dda96
API(s) Documentation
API Reference
====================
This is just a brief summary of all API endpoints of the NOMAD API. For a more compelling documention
......
......@@ -69,10 +69,24 @@ identified an entry (given via a `upload_id`/`calc_id`, see the query output), a
you want to download it:
```
curl "http://repository.nomad-coe.eu/app/api/raw/calc/f0KQE2aiSz2KRE47QtoZtw/6xe9fZ9xoxBYZOq5lTt8JMgPa3gX/*" -o download.zip
curl "http://repository.nomad-coe.eu/app/api/raw/calc/JvdvikbhQp673R4ucwQgiA/k-ckeQ73sflE6GDA80L132VCWp1z/*" -o download.zip
```
This basically requests all the files (`*`) that belong to this entry. If you have a query
With `*` you basically requests all the files under an entry or path..
If you need a specific file (that you already know) of that calculation:
```
curl "http://repository.nomad-coe.eu/app/api/raw/calc/JvdvikbhQp673R4ucwQgiA/k-ckeQ73sflE6GDA80L132VCWp1z/INFO.OUT"
```
You can also download a specific file from the upload (given a `upload_id`), if you know
the path of that file:
```
curl "http://repository.nomad-coe.eu/app/api/raw/JvdvikbhQp673R4ucwQgiA/exciting_basis_set_error_study/monomers_expanded_k8_rgkmax_080_PBE/72_Hf/INFO.OUT"
```
If you have a query
that is more selective, you can also download all results. Here all compounds that only
consist of Si, O, bulk material simulations of cubic systems (currently ~100 entries):
......
.. _access-the-archive-label:
Accessing the Archive
=====================
Using the NOMAD Archive
=======================
Of course, you can access the NOMAD Archive directly via the NOMAD API (see the `API tutorial <api_tutorial.html>`_
and `API reference <api.html>`_). But, it is more effective and convenient to use NOMAD's Python client
......
docs/assets/components.png

107 KB | W: | H:

docs/assets/components.png

87.9 KB | W: | H:

docs/assets/components.png
docs/assets/components.png
docs/assets/components.png
docs/assets/components.png
  • 2-up
  • Swipe
  • Onion skin
......@@ -25,8 +25,8 @@ sys.path.insert(0, os.path.abspath('..'))
# -- Project information -----------------------------------------------------
project = 'nomad-FAIRDI'
copyright = '2018, FAIRDI e.V.'
project = 'NOMAD Repository and Archive'
copyright = '2014-2020, FAIRDI e.V.'
author = 'FAIRDI e.V.'
# The short X.Y version
......
# How to write a parser for nomad@FAIRDI
# How to write a parser
## The parser project
......@@ -313,7 +313,7 @@ is stored in a database for analysis. Do not abuse the logging.
## Testing and Debugging
You a writing a python program. You know what to do.
## Added the parser to nomad@FAIRDI
## Adding the parser to nomad@FAIRDI
First, you add your parser to the dependencies. Put it into the dependencies folder, then:
```
......@@ -343,3 +343,71 @@ parser_examples = [
('parsers/vaspoutcar', 'tests/data/parsers/vasp_outcar/OUTCAR'),
]
```
## FAIRDI parsers
The new fairdi parsers avoid the use of a backend and instead make use of the new metainfo
sections. The project structure is the same as above with the addition of a `metainfo`
folder
```
myparser/myparser/metainfo
```
This contains a file containing the definitions and an `__init__.py`. One should refer
to `nomad.metainfo.example.py` for a guide in writing the metainfo definitions.
Consequently, the parser class implementation is modified as in the following example.
```python
import json
from .metainfo import m_env
from nomad.parsing.parser import MatchingParser
from nomad.datamodel.metainfo.general_experimental import section_experiment as msection_experiment
from nomad.datamodel.metainfo.general_experimental import section_data as msection_data
from nomad.datamodel.metainfo.general_experimental_method import section_method as msection_method
from nomad.datamodel.metainfo.general_experimental_sample import section_sample as msection_sample
class ExampleParser(MatchingParser):
def __init__(self):
super().__init__(
name='parsers/example', code_name='example', code_homepage='https://github.com/example/example',
domain='ems', mainfile_mime_re=r'(application/json)|(text/.*)', mainfile_name_re=(r'.*.example')
)
def run(self, filepath, logger=None):
self._metainfo_env = m_env
with open(filepath, 'rt') as f:
data = json.load(f)
section_experiment = msection_experiment()
# Read general tool environment details
section_experiment.experiment_location = data.get('experiment_location')
section_experiment.experiment_facility_institution = data.get('experiment_facility_institution')
# Read data parameters
section_data = section_experiment.m_create(msection_data)
section_data.data_repository_name = data.get('data_repository_name')
section_data.data_preview_url = data.get('data_repository_url')
# Read parameters related to method
section_method = section_experiment.m_create(msection_method)
section_method.experiment_method_name = data.get('experiment_method')
section_method.probing_method = 'electric pulsing'
# Read parameters related to sample
section_sample = section_experiment.m_create(msection_sample)
section_sample.sample_description = data.get('specimen_description')
section_sample.sample_microstructure = data.get('specimen_microstructure')
section_sample.sample_constituents = data.get('specimen_constitution')
return section_experiment
```
The parser extends the ``MatchingParser`` class which already implements the determination
of the necessary file for parsing. The main difference to the old framework is the absense
the opening and closing of sections. One only needs to create a section which can be
accessed at any point in the code. The ``run`` method should return the root section.
Lastly, one should add an instance of the parser class in the list of parsers at
``nomad.parsing``.
\ No newline at end of file
nomad@FAIRDI
============
NOMAD Repository and Archive
============================
This project is a prototype for the continuation of the original NOMAD-coe software
and infrastructure with a simplyfied architecture and consolidated code base.
......@@ -10,12 +10,12 @@ and infrastructure with a simplyfied architecture and consolidated code base.
introduction.md
upload.rst
api_tutorial.md
api.rst
archive.rst
metainfo.rst
client/client.rst
metainfo.rst
archive.rst
ops/ops.rst
dev/setup.md
dev/dev_guidelines.rst
parser_tutorial.md
dev/parser_tutorial.md
api.rst
reference.rst
.. _metainfo-label:
Metainfo
========
NOMAD Metainfo
==============
Introduction
------------
The NOMAD Metainfo stores descriptive and structured information about materials-science
data contained in the NOMAD Archive. The Metainfo can be understood as the schema of
the Archive. The NOMAD Archive data is
structured to be independent of the electronic-structure theory code or molecular-simulation,
(or beyond). The NOMAD Metainfo can be browsed as part of the `NOMAD Repository and Archive web application <https://repository.nomad-coe.eu/app/gui/metainfo>`_.
Typically (meta-)data definitions are generated only for a predesigned and specific scientific field,
application or code. In contrast, the NOMAD Metainfo considers all pertinent information
in the input and output files of the supported electronic-structure theory, quantum chemistry,
and molecular-dynamics (force-field) codes. This ensures a complete coverage of all
material and molecule properties, even though some properties might not be as important as
others, or are missing in some input/output files of electronic-structure programs.
.. image:: assets/metainfo_example.png
NOMAD Metainfo is kept independent of the actual storage format and is not bound to any
specific storage method. In our practical implementation, we use a binary form of JSON,
called `msgpack <https://msgpack.org/>`_ on our servers and provide Archive data as JSON via
our API. For NOMAD end-users the internal storage format is of little relevance, because
the archive data is solely served by NOMAD's API.
The NOMAD Metainfo started within the `NOMAD Laboratory <https://nomad-lab.eu>`_. It was discussed at the
`CECAM workshop Towards a Common Format for Computational Materials Science Data <https://th.fhi-berlin.mpg.de/meetings/FCMSD2016/>`_
and is open to external contributions and extensions. More information can be found in:
- `Towards a Common Format for Computational Materials Science Data (Psi-K 2016 Highlight) <http://th.fhi-berlin.mpg.de/site/uploads/Publications/Psik_Highlight_131-2016.pdf>`_ provides a description on how to establish code-independent formats in detail and presents the challenges and practical strategies for achieving a common format for the representation of computational material-science data.
- `The Novel Materials Discovery Laboratory - Data formats and compression, D1.1 <https://www.nomad-coe.eu/uploads/outreach/Public%20Deliverables/NOMAD%20D1.1%20public%20KO20June2016.pdf>`_ outlines possible data formats, concepts, and compression techniques used to build a homogeneous (code-independent) data archive, called the NOMA
Metainfo Python Interface
-------------------------
.. automodule:: nomad.metainfo
......@@ -34,6 +69,9 @@ Python modules:
from nomad.datamodel.metainfo.public import section_run
my_run = section_run()
Many more examples about how to read the NOMAD Metainfo programmatically can be found
`here <https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/tree/master/examples/access_metainfo.py>`_.
API
___
......
Reference
=========
Python Reference
================
nomad.metainfo
--------------
......
==============
Uploading Data
==============
======================================
Uploading Data to the NOMAD Repository
======================================
To contribute your data to the repository, please, login to our `upload page <../uploads>`_ (you need to register first, if you do not have a NOMAD account yet).
......
# pylint: skip-file
# type: ignore
from nomad import metainfo
from nomad.datamodel.metainfo import public, common
# Access the quantities of a section definition
......@@ -40,3 +41,53 @@ import nomad.datamodel.datamodel # noqa
print(json.dumps(nomad.datamodel.datamodel.m_package.m_to_dict(), indent=2))
print(json.dumps(public.m_package.m_to_dict(), indent=2))
# Using an environment that manages multiple packages and provides utility functions
# to find definitions by name.
from nomad.datamodel.metainfo import m_env # noqa, contains all common, public, general metainfo
from vaspparser.metainfo import m_env as vasp_m_env # noqa, contains also the vasp specific definitions
print(m_env.packages)
# Resolve definition by name
print(m_env.resolve_definitions('number_of_atoms', metainfo.Quantity))
# Traverse all definitions:
for definition in m_env.m_all_contents():
print(definition)
# Dimensions are either numbers or rangens (e.g. 3, 1..3, 0..*) or references to
# shapeless, unitless, integer quantities (usually) of the same section.
# These quantities are not specifically designated as dimensions, because they represent
# quantities in their own right and are often used on their own.
# Dimensions of a specific quantity:
quantity = public.section_system.atom_labels
for dim in quantity.shape:
if isinstance(dim, str):
section = quantity.m_parent
print('%s[%s]: %s' % (quantity.name, dim, m_env.resolve_definitions(dim, metainfo.Quantity)))
# All quantities used as dimensions in a package:
for definition in public.m_package.m_all_contents():
if definition.m_def == metainfo.Quantity.m_def:
for dim in definition.shape:
if isinstance(dim, str) and '..' not in dim:
print('%s[%s]: %s' % (quantity.name, dim, m_env.resolve_definitions(dim, metainfo.Quantity)))
# Categories are special classes, similar to sections and they Python definition is a
# subclass of MCategory or MSection:
print(public.atom_forces_type, issubclass(public.atom_forces_type, metainfo.MCategory))
print(public.section_system, issubclass(public.section_system, metainfo.MSection))
# Or the definition of the definition is Category or Section respectively:
print(public.atom_forces_type, public.atom_forces_type.m_def == metainfo.Category.m_def)
print(public.section_system, public.section_system.m_def == metainfo.Section.m_def)
# Get all sections and categories definitions in a package:
print(public.m_package.category_definitions)
print(public.m_package.section_definitions)
# Access the categories of a metainfo definition, e.g. quantity
print(public.section_single_configuration_calculation.energy_total.categories)
print(m_env.resolve_definition('EntryMetadata', metainfo.Section).all_quantities)
print(m_env.resolve_definition('Bulk', metainfo.Section).all_quantities)
print(m_env.resolve_definition('OptimadeEntry', metainfo.Section).all_quantities)
\ No newline at end of file
{
"name": "nomad-fair-gui",
"version": "0.8.1",
"version": "0.8.2",
"commit": "e98694e",
"private": true,
"dependencies": {
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment