Commit f522e4b6 authored by Markus Scheidgen's avatar Markus Scheidgen

Merge branch 'v0.8.2' into 'master'

0.8.2 release

See merge request !134
parents 04f1385a e02dd4cf
Pipeline #78750 passed with stages
in 10 minutes and 9 seconds
......@@ -143,19 +143,20 @@ release:
- docker pull $TEST_IMAGE
- docker tag $TEST_IMAGE $LATEST_IMAGE
- docker push $LATEST_IMAGE
except:
- /^dev-.*$/
when: manual
only:
- branches
release_version:
stage: release
script:
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab-registry.mpcdf.mpg.de
- docker pull $LATEST_IMAGE
- docker tag $LATEST_IMAGE $CI_COMMIT_TAG
- docker tag $LATEST_IMAGE $STABLE_IMAGE
- docker pull $TEST_IMAGE
- docker tag $TEST_IMAGE $LATEST_IMAGE
- docker push $LATEST_IMAGE
- docker tag $TEST_IMAGE $STABLE_IMAGE
- docker push $STABLE_IMAGE
- docker push $CI_COMMIT_TAG
only:
- tags
[submodule "dependencies/nomad-meta-info"]
path = dependencies/nomad-meta-info
url = https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-meta-info.git
branch = nomad-fair
[submodule "dependencies/python_common"]
path = dependencies/python_common
url = https://gitlab.mpcdf.mpg.de/nomad-lab/python-common.git
......@@ -12,7 +8,7 @@
branch = nomad-fair
[submodule "dependencies/parsers/exciting"]
path = dependencies/parsers/exciting
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-exciting.git
url = https://gitlab.mpcdf.mpg.de/nomad-lab/parser-exciting-fairdi.git
branch = nomad-fair
[submodule "dependencies/parsers/fhi-aims"]
path = dependencies/parsers/fhi-aims
......
......@@ -106,26 +106,23 @@ WORKDIR /app
# transfer installed packages from dependency stage
COPY --from=build /usr/local/lib/python3.7/site-packages /usr/local/lib/python3.7/site-packages
RUN echo "copy 1"
# copy the meta-info, since it files are loaded via relative paths. TODO that should change.
COPY --from=build /install/dependencies/nomad-meta-info /app/dependencies/nomad-meta-info
RUN echo "copy 2"
# copy the documentation, its files will be served by the API
COPY --from=build /install/docs/.build /app/docs/.build
RUN echo "copy 3"
RUN echo "copy 2"
# copy the source distribution, its files will be served by the API
COPY --from=build /install/dist /app/dist
RUN echo "copy 4"
RUN echo "copy 3"
# copy the nomad command
COPY --from=build /usr/local/bin/nomad /usr/bin/nomad
RUN echo "copy 5"
RUN echo "copy 4"
# copy the gui
RUN mkdir -p /app/gui
COPY --from=gui_build /app/build /app/gui/build
RUN echo "copy 6"
RUN echo "copy 5"
# copy the encyclopedia gui production code
COPY --from=gui_build /encyclopedia /app/dependencies/encyclopedia-gui/client
RUN rm -rf /app/dependencies/encyclopedia-gui/client/src
RUN echo "copy 7"
RUN echo "copy 6"
RUN mkdir -p /app/.volumes/fs
RUN useradd -ms /bin/bash nomad
......
......@@ -37,7 +37,7 @@ nomad parse --show-backend <your-file-to-parse>
### For NOMAD developer
Read the [docs](https://repository.nomad-coe.eu/app/docs). The documentation is also part
Read the [docs](https://repository.nomad-coe.eu/app/docs/index.html). The documentation is also part
of the source code. It covers aspects like introduction, architecture, development setup/deployment,
contributing, and API reference.
......
Subproject commit bb633623789ee9850873ba0b16a32c66c8265558
Subproject commit 562ae7d26fa108c7c31ebe38775c767aba1e642c
Subproject commit a2df84b2dfc2c96344e7261580a40d655c904047
Subproject commit 45daba83219defb908044193214acce48baf76e9
Subproject commit d394f7ae375cd2355127abc67e07b1592f18d4f4
Subproject commit 38ed55429bd8c2289471c5dbc162bcdd80b7bb94
Subproject commit 6f0e7ec897284382d8ef30b0ee0372f010ce468f
Subproject commit 2849061c5212de8e2845ec0c3ed96b7a7f4e67d9
Subproject commit 0edade50d6be30bc83dca35d382bb6d462ea6f91
Subproject commit ca7513c0e23b20d738f65b081b482e74a8d0527d
Subproject commit d342d8a14dc49d528fe7ca867c3f5891d8b73077
Subproject commit 124b42b9851329682e11c2f1f3e00056d215bd3b
Subproject commit aa701e8f2780419e7911972c8e2a16d188008902
Subproject commit ba060d7c8cb902d16e47ca382ef986662a5f3fa4
Subproject commit 1a0c8cb0801375a78134c67f7a6d31319a338503
Subproject commit 4c83e96bcbf9080d431b7b22f1ecabd4b64dec27
Subproject commit cf843eda3e4650bc240eb0d1a1ab5b0cea6dda96
API(s) Documentation
API Reference
====================
This is just a brief summary of all API endpoints of the NOMAD API. For a more compelling documention
......
......@@ -69,10 +69,24 @@ identified an entry (given via a `upload_id`/`calc_id`, see the query output), a
you want to download it:
```
curl "http://repository.nomad-coe.eu/app/api/raw/calc/f0KQE2aiSz2KRE47QtoZtw/6xe9fZ9xoxBYZOq5lTt8JMgPa3gX/*" -o download.zip
curl "http://repository.nomad-coe.eu/app/api/raw/calc/JvdvikbhQp673R4ucwQgiA/k-ckeQ73sflE6GDA80L132VCWp1z/*" -o download.zip
```
This basically requests all the files (`*`) that belong to this entry. If you have a query
With `*` you basically requests all the files under an entry or path..
If you need a specific file (that you already know) of that calculation:
```
curl "http://repository.nomad-coe.eu/app/api/raw/calc/JvdvikbhQp673R4ucwQgiA/k-ckeQ73sflE6GDA80L132VCWp1z/INFO.OUT"
```
You can also download a specific file from the upload (given a `upload_id`), if you know
the path of that file:
```
curl "http://repository.nomad-coe.eu/app/api/raw/JvdvikbhQp673R4ucwQgiA/exciting_basis_set_error_study/monomers_expanded_k8_rgkmax_080_PBE/72_Hf/INFO.OUT"
```
If you have a query
that is more selective, you can also download all results. Here all compounds that only
consist of Si, O, bulk material simulations of cubic systems (currently ~100 entries):
......
.. _access-the-archive-label:
Accessing the Archive
=====================
Using the NOMAD Archive
=======================
Of course, you can access the NOMAD Archive directly via the NOMAD API (see the `API tutorial <api_tutorial.html>`_
and `API reference <api.html>`_). But, it is more effective and convenient to use NOMAD's Python client
......
docs/assets/components.png

107 KB | W: | H:

docs/assets/components.png

87.9 KB | W: | H:

docs/assets/components.png
docs/assets/components.png
docs/assets/components.png
docs/assets/components.png
  • 2-up
  • Swipe
  • Onion skin
......@@ -5,14 +5,16 @@ We release the NOMAD client library as a Python `distutils <https://docs.python.
You can download and install it the usual way using *pip* (or *conda*).
Install from pypi
.. code-block:: sh
pip install nomad-lab
Download and install latest release from nomad
.. code-block:: sh
curl https://repository.nomad-coe.eu/v0.8/dist/nomad-lab.tar.gz -o nomad-lab.tar.gz
curl https://repository.nomad-coe.eu/app/dist/nomad-lab.tar.gz -o nomad-lab.tar.gz
pip install ./nomad-lab.tar.gz
There are different layers of dependencies that you have to install, in order to use
......
......@@ -25,8 +25,8 @@ sys.path.insert(0, os.path.abspath('..'))
# -- Project information -----------------------------------------------------
project = 'nomad-FAIRDI'
copyright = '2018, FAIRDI e.V.'
project = 'NOMAD Repository and Archive'
copyright = '2014-2020, FAIRDI e.V.'
author = 'FAIRDI e.V.'
# The short X.Y version
......@@ -200,4 +200,4 @@ def setup(app):
# }, True)
# app.add_transform(AutoStructify)
extlinks = {'api': ('https://repository.nomad-coe.eu/v0.8/api/%s', 'NOMAD API ')}
\ No newline at end of file
extlinks = {'api': ('https://repository.nomad-coe.eu/app/api/%s', 'NOMAD API ')}
\ No newline at end of file
# How to write a parser for nomad@FAIRDI
# How to write a parser
## The parser project
......@@ -313,7 +313,7 @@ is stored in a database for analysis. Do not abuse the logging.
## Testing and Debugging
You a writing a python program. You know what to do.
## Added the parser to nomad@FAIRDI
## Adding the parser to nomad@FAIRDI
First, you add your parser to the dependencies. Put it into the dependencies folder, then:
```
......@@ -343,3 +343,71 @@ parser_examples = [
('parsers/vaspoutcar', 'tests/data/parsers/vasp_outcar/OUTCAR'),
]
```
## FAIRDI parsers
The new fairdi parsers avoid the use of a backend and instead make use of the new metainfo
sections. The project structure is the same as above with the addition of a `metainfo`
folder
```
myparser/myparser/metainfo
```
This contains a file containing the definitions and an `__init__.py`. One should refer
to `nomad.metainfo.example.py` for a guide in writing the metainfo definitions.
Consequently, the parser class implementation is modified as in the following example.
```python
import json
from .metainfo import m_env
from nomad.parsing.parser import MatchingParser
from nomad.datamodel.metainfo.general_experimental import section_experiment as msection_experiment
from nomad.datamodel.metainfo.general_experimental import section_data as msection_data
from nomad.datamodel.metainfo.general_experimental_method import section_method as msection_method
from nomad.datamodel.metainfo.general_experimental_sample import section_sample as msection_sample
class ExampleParser(MatchingParser):
def __init__(self):
super().__init__(
name='parsers/example', code_name='example', code_homepage='https://github.com/example/example',
domain='ems', mainfile_mime_re=r'(application/json)|(text/.*)', mainfile_name_re=(r'.*.example')
)
def run(self, filepath, logger=None):
self._metainfo_env = m_env
with open(filepath, 'rt') as f:
data = json.load(f)
section_experiment = msection_experiment()
# Read general tool environment details
section_experiment.experiment_location = data.get('experiment_location')
section_experiment.experiment_facility_institution = data.get('experiment_facility_institution')
# Read data parameters
section_data = section_experiment.m_create(msection_data)
section_data.data_repository_name = data.get('data_repository_name')
section_data.data_preview_url = data.get('data_repository_url')
# Read parameters related to method
section_method = section_experiment.m_create(msection_method)
section_method.experiment_method_name = data.get('experiment_method')
section_method.probing_method = 'electric pulsing'
# Read parameters related to sample
section_sample = section_experiment.m_create(msection_sample)
section_sample.sample_description = data.get('specimen_description')
section_sample.sample_microstructure = data.get('specimen_microstructure')
section_sample.sample_constituents = data.get('specimen_constitution')
return section_experiment
```
The parser extends the ``MatchingParser`` class which already implements the determination
of the necessary file for parsing. The main difference to the old framework is the absense
the opening and closing of sections. One only needs to create a section which can be
accessed at any point in the code. The ``run`` method should return the root section.
Lastly, one should add an instance of the parser class in the list of parsers at
``nomad.parsing``.
\ No newline at end of file
......@@ -77,7 +77,7 @@ conda activate nomad_env
To install libmagick for conda, you can use (other channels might also work):
```
conda -c conda-forge install --name nomad_env libmagic
conda install -c conda-forge --name nomad_env libmagic
```
#### pip
......
nomad@FAIRDI
============
NOMAD Repository and Archive
============================
This project is a prototype for the continuation of the original NOMAD-coe software
and infrastructure with a simplyfied architecture and consolidated code base.
......@@ -10,12 +10,12 @@ and infrastructure with a simplyfied architecture and consolidated code base.
introduction.md
upload.rst
api_tutorial.md
api.rst
archive.rst
metainfo.rst
client/client.rst
metainfo.rst
archive.rst
ops/ops.rst
dev/setup.md
dev/dev_guidelines.rst
parser_tutorial.md
dev/parser_tutorial.md
api.rst
reference.rst
.. _metainfo-label:
Metainfo
========
NOMAD Metainfo
==============
Introduction
------------
The NOMAD Metainfo stores descriptive and structured information about materials-science
data contained in the NOMAD Archive. The Metainfo can be understood as the schema of
the Archive. The NOMAD Archive data is
structured to be independent of the electronic-structure theory code or molecular-simulation,
(or beyond). The NOMAD Metainfo can be browsed as part of the `NOMAD Repository and Archive web application <https://repository.nomad-coe.eu/app/gui/metainfo>`_.
Typically (meta-)data definitions are generated only for a predesigned and specific scientific field,
application or code. In contrast, the NOMAD Metainfo considers all pertinent information
in the input and output files of the supported electronic-structure theory, quantum chemistry,
and molecular-dynamics (force-field) codes. This ensures a complete coverage of all
material and molecule properties, even though some properties might not be as important as
others, or are missing in some input/output files of electronic-structure programs.
.. image:: assets/metainfo_example.png
NOMAD Metainfo is kept independent of the actual storage format and is not bound to any
specific storage method. In our practical implementation, we use a binary form of JSON,
called `msgpack <https://msgpack.org/>`_ on our servers and provide Archive data as JSON via
our API. For NOMAD end-users the internal storage format is of little relevance, because
the archive data is solely served by NOMAD's API.
The NOMAD Metainfo started within the `NOMAD Laboratory <https://nomad-lab.eu>`_. It was discussed at the
`CECAM workshop Towards a Common Format for Computational Materials Science Data <https://th.fhi-berlin.mpg.de/meetings/FCMSD2016/>`_
and is open to external contributions and extensions. More information can be found in:
- `Towards a Common Format for Computational Materials Science Data (Psi-K 2016 Highlight) <http://th.fhi-berlin.mpg.de/site/uploads/Publications/Psik_Highlight_131-2016.pdf>`_ provides a description on how to establish code-independent formats in detail and presents the challenges and practical strategies for achieving a common format for the representation of computational material-science data.
- `The Novel Materials Discovery Laboratory - Data formats and compression, D1.1 <https://www.nomad-coe.eu/uploads/outreach/Public%20Deliverables/NOMAD%20D1.1%20public%20KO20June2016.pdf>`_ outlines possible data formats, concepts, and compression techniques used to build a homogeneous (code-independent) data archive, called the NOMA
Metainfo Python Interface
-------------------------
.. automodule:: nomad.metainfo
......@@ -34,6 +69,9 @@ Python modules:
from nomad.datamodel.metainfo.public import section_run
my_run = section_run()
Many more examples about how to read the NOMAD Metainfo programmatically can be found
`here <https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/tree/master/examples/access_metainfo.py>`_.
API
___
......
Reference
=========
Python Reference
================
nomad.metainfo
--------------
......
==============
Uploading Data
==============
======================================
Uploading Data to the NOMAD Repository
======================================
To contribute your data to the repository, please, login to our `upload page <../uploads>`_ (you need to register first, if you do not have a NOMAD account yet).
......
# pylint: skip-file
# type: ignore
from nomad import metainfo
from nomad.datamodel.metainfo import public, common
# Access the quantities of a section definition
for quantity in public.section_method.m_def.quantities:
print(quantity.name, quantity.type, quantity.shape, quantity.unit)
# Access all the quantities of a section definition, including those added by other packages
import vaspparser.metainfo.vasp # noqa
for quantity in public.section_method.m_def.all_quantities.values():
print(quantity.name, quantity.type, quantity.shape, quantity.unit)
# Access sub-sections and their definitions
for sub_section in public.section_run.m_def.sub_sections:
print(sub_section.name) # access the name of the sub section definition
print(sub_section.sub_section.name) # access the name of the section definition that the sub section refers to
# Go through all sub sections recursively
def visit_section(section, indent=0):
print(' ' * indent + section.name)
for sub_section in section.all_sub_sections.values():
visit_section(sub_section.sub_section, indent + 2)
visit_section(public.section_run.m_def)
# Look at the EntryArchive, e.g. where section_metadata (and everything else) is a subsection
from nomad.datamodel import EntryArchive # noqa
visit_section(EntryArchive.m_def)
# To get everything within a metainfo package (i.e. what was former in a .nomadmetainfo.json file) as JSON/dict data:
import json # noqa
import nomad.datamodel.datamodel # noqa
print(json.dumps(nomad.datamodel.datamodel.m_package.m_to_dict(), indent=2))
print(json.dumps(public.m_package.m_to_dict(), indent=2))
# Using an environment that manages multiple packages and provides utility functions
# to find definitions by name.
from nomad.datamodel.metainfo import m_env # noqa, contains all common, public, general metainfo
from vaspparser.metainfo import m_env as vasp_m_env # noqa, contains also the vasp specific definitions
print(m_env.packages)
# Resolve definition by name
print(m_env.resolve_definitions('number_of_atoms', metainfo.Quantity))
# Traverse all definitions:
for definition in m_env.m_all_contents():
print(definition)
# Dimensions are either numbers or rangens (e.g. 3, 1..3, 0..*) or references to
# shapeless, unitless, integer quantities (usually) of the same section.
# These quantities are not specifically designated as dimensions, because they represent
# quantities in their own right and are often used on their own.
# Dimensions of a specific quantity:
quantity = public.section_system.atom_labels
for dim in quantity.shape:
if isinstance(dim, str):
section = quantity.m_parent
print('%s[%s]: %s' % (quantity.name, dim, m_env.resolve_definitions(dim, metainfo.Quantity)))
# All quantities used as dimensions in a package:
for definition in public.m_package.m_all_contents():
if definition.m_def == metainfo.Quantity.m_def:
for dim in definition.shape:
if isinstance(dim, str) and '..' not in dim:
print('%s[%s]: %s' % (quantity.name, dim, m_env.resolve_definitions(dim, metainfo.Quantity)))
# Categories are special classes, similar to sections and they Python definition is a
# subclass of MCategory or MSection:
print(public.atom_forces_type, issubclass(public.atom_forces_type, metainfo.MCategory))
print(public.section_system, issubclass(public.section_system, metainfo.MSection))
# Or the definition of the definition is Category or Section respectively:
print(public.atom_forces_type, public.atom_forces_type.m_def == metainfo.Category.m_def)
print(public.section_system, public.section_system.m_def == metainfo.Section.m_def)
# Get all sections and categories definitions in a package:
print(public.m_package.category_definitions)
print(public.m_package.section_definitions)
# Access the categories of a metainfo definition, e.g. quantity
print(public.section_single_configuration_calculation.energy_total.categories)
print(m_env.resolve_definition('EntryMetadata', metainfo.Section).all_quantities)
print(m_env.resolve_definition('Bulk', metainfo.Section).all_quantities)
print(m_env.resolve_definition('OptimadeEntry', metainfo.Section).all_quantities)
\ No newline at end of file
......@@ -2,8 +2,6 @@ from nomad import config
from nomad.client import ArchiveQuery
from nomad.metainfo import units
# this will not be necessary, once this is the official NOMAD version
config.client.url = 'http://labdev-nomad.esc.rzg.mpg.de/fairdi/nomad/testing-major/api'
query = ArchiveQuery(
query={
......
......@@ -3,7 +3,7 @@ from nomad import config, infrastructure, search
config.elastic.host = 'localhost'
config.elastic.port = 19202
config.elastic.index_name = 'fairdi_nomad_prod_v0_7'
config.elastic.index_name = 'fairdi_nomad_prod_v0_8'
infrastructure.setup_elastic()
......
{
"name": "nomad-fair-gui",
"version": "0.8.1",
"version": "0.8.2",
"commit": "e98694e",
"private": true,
"dependencies": {
......
......@@ -8,9 +8,9 @@ window.nomadEnv = {
'matomoUrl': 'https://repository.nomad-coe.eu/fairdi/stat',
'matomoSiteId': '2',
'version': {
"label": "0.8.1",
"isBeta": true,
"usesBetaData": true,
"label": "0.8.2",
"isBeta": false,
"usesBetaData": false,
"officialUrl": "https://repository.nomad-coe.eu/app/gui"
}
}
......@@ -248,7 +248,8 @@ class DataTableUnStyled extends React.Component {
whiteSpace: 'nowrap',
maxWidth: 200,
paddingLeft: theme.spacing(3),
paddingRight: theme.spacing(3)
paddingRight: theme.spacing(3),
height: theme.spacing.unit * 6
},
ellipsisFront: {
direction: 'rtl',
......@@ -293,14 +294,14 @@ class DataTableUnStyled extends React.Component {
super(props)
this.handleSelectAllClick = this.handleSelectAllClick.bind(this)
this.handleSelectedColumnsChanged = this.handleSelectedColumnsChanged.bind(this)
}
defaultSelectedColumns() {
let selectedColumns = this.props.selectedColumns || Object.keys(this.props.columns)
if (this.props.selectedColumnsKey) {
selectedColumns = globalSelectedColumns[this.props.selectedColumnsKey] || selectedColumns
}
this.state = {
...this.state,
selectedColumns: selectedColumns
}
return selectedColumns
}
state = {
......@@ -375,7 +376,8 @@ class DataTableUnStyled extends React.Component {
renderDetails(row) {
const { classes, entryDetails, id, entryActions } = this.props
const { selectedColumns, selectedEntry } = this.state
const { selectedEntry } = this.state
const selectedColumns = this.state.selectedColumns || this.defaultSelectedColumns()
if (entryDetails) {
return (
<tr>
......@@ -400,7 +402,8 @@ class DataTableUnStyled extends React.Component {
const {
classes, data, total, order, orderBy, id, rows, selectActions, actions,
entryDetails, entryActions, columns, entityLabels, pagination } = this.props
const { selectedColumns, selectedEntry } = this.state
const { selectedEntry } = this.state
const selectedColumns = this.state.selectedColumns || this.defaultSelectedColumns()
const totalNumber = total || 0
......
......@@ -75,6 +75,7 @@ export default function DatasetPage() {
query={{dataset_id: [datasetId]}}
ownerTypes={['all', 'public']}
initialResultTab="entries"
resultListProps={{showAccessColumn: true}}
availableResultTabs={['entries', 'groups', 'datasets']}
/>
</div>
......
......@@ -217,7 +217,7 @@ function Markdown(props) {
return (
<Typography variant="body1"
className={classes.root}
dangerouslySetInnerHTML={{__html: marked(content)}}
dangerouslySetInnerHTML={{__html: marked(content || '')}}
/>
)
}
......
......@@ -54,7 +54,7 @@ Once you assigned a DOI to a dataset, no entries can be removed or added to the
function UserdataPage() {
return <Search
ownerTypes={['user', 'staging']}
ownerTypes={['user', 'shared', 'staging']}
initialOwner="user"
initialRequest={{order_by: 'upload_time', uploads_grouped: true}}
initialResultTab="uploads"
......
......@@ -63,6 +63,10 @@ class RawFiles extends React.Component {
whiteSpace: 'nowrap',
direction: 'rtl',
textAlign: 'left'
},
mainfileLabel: {
fontSize: '10pt',
fontWeight: 'bold'
}
})