Commit f522e4b6 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'v0.8.2' into 'master'

0.8.2 release

See merge request !134
parents 04f1385a e02dd4cf
Pipeline #78750 passed with stages
in 10 minutes and 9 seconds
# How to write a parser for nomad@FAIRDI
# How to write a parser
## The parser project
......@@ -313,7 +313,7 @@ is stored in a database for analysis. Do not abuse the logging.
## Testing and Debugging
You a writing a python program. You know what to do.
## Added the parser to nomad@FAIRDI
## Adding the parser to nomad@FAIRDI
First, you add your parser to the dependencies. Put it into the dependencies folder, then:
```
......@@ -343,3 +343,71 @@ parser_examples = [
('parsers/vaspoutcar', 'tests/data/parsers/vasp_outcar/OUTCAR'),
]
```
## FAIRDI parsers
The new fairdi parsers avoid the use of a backend and instead make use of the new metainfo
sections. The project structure is the same as above with the addition of a `metainfo`
folder
```
myparser/myparser/metainfo
```
This contains a file containing the definitions and an `__init__.py`. One should refer
to `nomad.metainfo.example.py` for a guide in writing the metainfo definitions.
Consequently, the parser class implementation is modified as in the following example.
```python
import json
from .metainfo import m_env
from nomad.parsing.parser import MatchingParser
from nomad.datamodel.metainfo.general_experimental import section_experiment as msection_experiment
from nomad.datamodel.metainfo.general_experimental import section_data as msection_data
from nomad.datamodel.metainfo.general_experimental_method import section_method as msection_method
from nomad.datamodel.metainfo.general_experimental_sample import section_sample as msection_sample
class ExampleParser(MatchingParser):
def __init__(self):
super().__init__(
name='parsers/example', code_name='example', code_homepage='https://github.com/example/example',
domain='ems', mainfile_mime_re=r'(application/json)|(text/.*)', mainfile_name_re=(r'.*.example')
)
def run(self, filepath, logger=None):
self._metainfo_env = m_env
with open(filepath, 'rt') as f:
data = json.load(f)
section_experiment = msection_experiment()
# Read general tool environment details
section_experiment.experiment_location = data.get('experiment_location')
section_experiment.experiment_facility_institution = data.get('experiment_facility_institution')
# Read data parameters
section_data = section_experiment.m_create(msection_data)
section_data.data_repository_name = data.get('data_repository_name')
section_data.data_preview_url = data.get('data_repository_url')
# Read parameters related to method
section_method = section_experiment.m_create(msection_method)
section_method.experiment_method_name = data.get('experiment_method')
section_method.probing_method = 'electric pulsing'
# Read parameters related to sample
section_sample = section_experiment.m_create(msection_sample)
section_sample.sample_description = data.get('specimen_description')
section_sample.sample_microstructure = data.get('specimen_microstructure')
section_sample.sample_constituents = data.get('specimen_constitution')
return section_experiment
```
The parser extends the ``MatchingParser`` class which already implements the determination
of the necessary file for parsing. The main difference to the old framework is the absense
the opening and closing of sections. One only needs to create a section which can be
accessed at any point in the code. The ``run`` method should return the root section.
Lastly, one should add an instance of the parser class in the list of parsers at
``nomad.parsing``.
\ No newline at end of file
......@@ -77,7 +77,7 @@ conda activate nomad_env
To install libmagick for conda, you can use (other channels might also work):
```
conda -c conda-forge install --name nomad_env libmagic
conda install -c conda-forge --name nomad_env libmagic
```
#### pip
......
nomad@FAIRDI
============
NOMAD Repository and Archive
============================
This project is a prototype for the continuation of the original NOMAD-coe software
and infrastructure with a simplyfied architecture and consolidated code base.
......@@ -10,12 +10,12 @@ and infrastructure with a simplyfied architecture and consolidated code base.
introduction.md
upload.rst
api_tutorial.md
api.rst
archive.rst
metainfo.rst
client/client.rst
metainfo.rst
archive.rst
ops/ops.rst
dev/setup.md
dev/dev_guidelines.rst
parser_tutorial.md
dev/parser_tutorial.md
api.rst
reference.rst
.. _metainfo-label:
Metainfo
========
NOMAD Metainfo
==============
Introduction
------------
The NOMAD Metainfo stores descriptive and structured information about materials-science
data contained in the NOMAD Archive. The Metainfo can be understood as the schema of
the Archive. The NOMAD Archive data is
structured to be independent of the electronic-structure theory code or molecular-simulation,
(or beyond). The NOMAD Metainfo can be browsed as part of the `NOMAD Repository and Archive web application <https://repository.nomad-coe.eu/app/gui/metainfo>`_.
Typically (meta-)data definitions are generated only for a predesigned and specific scientific field,
application or code. In contrast, the NOMAD Metainfo considers all pertinent information
in the input and output files of the supported electronic-structure theory, quantum chemistry,
and molecular-dynamics (force-field) codes. This ensures a complete coverage of all
material and molecule properties, even though some properties might not be as important as
others, or are missing in some input/output files of electronic-structure programs.
.. image:: assets/metainfo_example.png
NOMAD Metainfo is kept independent of the actual storage format and is not bound to any
specific storage method. In our practical implementation, we use a binary form of JSON,
called `msgpack <https://msgpack.org/>`_ on our servers and provide Archive data as JSON via
our API. For NOMAD end-users the internal storage format is of little relevance, because
the archive data is solely served by NOMAD's API.
The NOMAD Metainfo started within the `NOMAD Laboratory <https://nomad-lab.eu>`_. It was discussed at the
`CECAM workshop Towards a Common Format for Computational Materials Science Data <https://th.fhi-berlin.mpg.de/meetings/FCMSD2016/>`_
and is open to external contributions and extensions. More information can be found in:
- `Towards a Common Format for Computational Materials Science Data (Psi-K 2016 Highlight) <http://th.fhi-berlin.mpg.de/site/uploads/Publications/Psik_Highlight_131-2016.pdf>`_ provides a description on how to establish code-independent formats in detail and presents the challenges and practical strategies for achieving a common format for the representation of computational material-science data.
- `The Novel Materials Discovery Laboratory - Data formats and compression, D1.1 <https://www.nomad-coe.eu/uploads/outreach/Public%20Deliverables/NOMAD%20D1.1%20public%20KO20June2016.pdf>`_ outlines possible data formats, concepts, and compression techniques used to build a homogeneous (code-independent) data archive, called the NOMA
Metainfo Python Interface
-------------------------
.. automodule:: nomad.metainfo
......@@ -34,6 +69,9 @@ Python modules:
from nomad.datamodel.metainfo.public import section_run
my_run = section_run()
Many more examples about how to read the NOMAD Metainfo programmatically can be found
`here <https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/tree/master/examples/access_metainfo.py>`_.
API
___
......
Reference
=========
Python Reference
================
nomad.metainfo
--------------
......
==============
Uploading Data
==============
======================================
Uploading Data to the NOMAD Repository
======================================
To contribute your data to the repository, please, login to our `upload page <../uploads>`_ (you need to register first, if you do not have a NOMAD account yet).
......
# pylint: skip-file
# type: ignore
from nomad import metainfo
from nomad.datamodel.metainfo import public, common
# Access the quantities of a section definition
for quantity in public.section_method.m_def.quantities:
print(quantity.name, quantity.type, quantity.shape, quantity.unit)
# Access all the quantities of a section definition, including those added by other packages
import vaspparser.metainfo.vasp # noqa
for quantity in public.section_method.m_def.all_quantities.values():
print(quantity.name, quantity.type, quantity.shape, quantity.unit)
# Access sub-sections and their definitions
for sub_section in public.section_run.m_def.sub_sections:
print(sub_section.name) # access the name of the sub section definition
print(sub_section.sub_section.name) # access the name of the section definition that the sub section refers to
# Go through all sub sections recursively
def visit_section(section, indent=0):
print(' ' * indent + section.name)
for sub_section in section.all_sub_sections.values():
visit_section(sub_section.sub_section, indent + 2)
visit_section(public.section_run.m_def)
# Look at the EntryArchive, e.g. where section_metadata (and everything else) is a subsection
from nomad.datamodel import EntryArchive # noqa
visit_section(EntryArchive.m_def)
# To get everything within a metainfo package (i.e. what was former in a .nomadmetainfo.json file) as JSON/dict data:
import json # noqa
import nomad.datamodel.datamodel # noqa
print(json.dumps(nomad.datamodel.datamodel.m_package.m_to_dict(), indent=2))
print(json.dumps(public.m_package.m_to_dict(), indent=2))
# Using an environment that manages multiple packages and provides utility functions
# to find definitions by name.
from nomad.datamodel.metainfo import m_env # noqa, contains all common, public, general metainfo
from vaspparser.metainfo import m_env as vasp_m_env # noqa, contains also the vasp specific definitions
print(m_env.packages)
# Resolve definition by name
print(m_env.resolve_definitions('number_of_atoms', metainfo.Quantity))
# Traverse all definitions:
for definition in m_env.m_all_contents():
print(definition)
# Dimensions are either numbers or rangens (e.g. 3, 1..3, 0..*) or references to
# shapeless, unitless, integer quantities (usually) of the same section.
# These quantities are not specifically designated as dimensions, because they represent
# quantities in their own right and are often used on their own.
# Dimensions of a specific quantity:
quantity = public.section_system.atom_labels
for dim in quantity.shape:
if isinstance(dim, str):
section = quantity.m_parent
print('%s[%s]: %s' % (quantity.name, dim, m_env.resolve_definitions(dim, metainfo.Quantity)))
# All quantities used as dimensions in a package:
for definition in public.m_package.m_all_contents():
if definition.m_def == metainfo.Quantity.m_def:
for dim in definition.shape:
if isinstance(dim, str) and '..' not in dim:
print('%s[%s]: %s' % (quantity.name, dim, m_env.resolve_definitions(dim, metainfo.Quantity)))
# Categories are special classes, similar to sections and they Python definition is a
# subclass of MCategory or MSection:
print(public.atom_forces_type, issubclass(public.atom_forces_type, metainfo.MCategory))
print(public.section_system, issubclass(public.section_system, metainfo.MSection))
# Or the definition of the definition is Category or Section respectively:
print(public.atom_forces_type, public.atom_forces_type.m_def == metainfo.Category.m_def)
print(public.section_system, public.section_system.m_def == metainfo.Section.m_def)
# Get all sections and categories definitions in a package:
print(public.m_package.category_definitions)
print(public.m_package.section_definitions)
# Access the categories of a metainfo definition, e.g. quantity
print(public.section_single_configuration_calculation.energy_total.categories)
print(m_env.resolve_definition('EntryMetadata', metainfo.Section).all_quantities)
print(m_env.resolve_definition('Bulk', metainfo.Section).all_quantities)
print(m_env.resolve_definition('OptimadeEntry', metainfo.Section).all_quantities)
\ No newline at end of file
......@@ -2,8 +2,6 @@ from nomad import config
from nomad.client import ArchiveQuery
from nomad.metainfo import units
# this will not be necessary, once this is the official NOMAD version
config.client.url = 'http://labdev-nomad.esc.rzg.mpg.de/fairdi/nomad/testing-major/api'
query = ArchiveQuery(
query={
......
......@@ -3,7 +3,7 @@ from nomad import config, infrastructure, search
config.elastic.host = 'localhost'
config.elastic.port = 19202
config.elastic.index_name = 'fairdi_nomad_prod_v0_7'
config.elastic.index_name = 'fairdi_nomad_prod_v0_8'
infrastructure.setup_elastic()
......
{
"name": "nomad-fair-gui",
"version": "0.8.1",
"version": "0.8.2",
"commit": "e98694e",
"private": true,
"dependencies": {
......
......@@ -8,9 +8,9 @@ window.nomadEnv = {
'matomoUrl': 'https://repository.nomad-coe.eu/fairdi/stat',
'matomoSiteId': '2',
'version': {
"label": "0.8.1",
"isBeta": true,
"usesBetaData": true,
"label": "0.8.2",
"isBeta": false,
"usesBetaData": false,
"officialUrl": "https://repository.nomad-coe.eu/app/gui"
}
}
......@@ -248,7 +248,8 @@ class DataTableUnStyled extends React.Component {
whiteSpace: 'nowrap',
maxWidth: 200,
paddingLeft: theme.spacing(3),
paddingRight: theme.spacing(3)
paddingRight: theme.spacing(3),
height: theme.spacing.unit * 6
},
ellipsisFront: {
direction: 'rtl',
......@@ -293,14 +294,14 @@ class DataTableUnStyled extends React.Component {
super(props)
this.handleSelectAllClick = this.handleSelectAllClick.bind(this)
this.handleSelectedColumnsChanged = this.handleSelectedColumnsChanged.bind(this)
}
defaultSelectedColumns() {
let selectedColumns = this.props.selectedColumns || Object.keys(this.props.columns)
if (this.props.selectedColumnsKey) {
selectedColumns = globalSelectedColumns[this.props.selectedColumnsKey] || selectedColumns
}
this.state = {
...this.state,
selectedColumns: selectedColumns
}
return selectedColumns
}
state = {
......@@ -375,7 +376,8 @@ class DataTableUnStyled extends React.Component {
renderDetails(row) {
const { classes, entryDetails, id, entryActions } = this.props
const { selectedColumns, selectedEntry } = this.state
const { selectedEntry } = this.state
const selectedColumns = this.state.selectedColumns || this.defaultSelectedColumns()
if (entryDetails) {
return (
<tr>
......@@ -400,7 +402,8 @@ class DataTableUnStyled extends React.Component {
const {
classes, data, total, order, orderBy, id, rows, selectActions, actions,
entryDetails, entryActions, columns, entityLabels, pagination } = this.props
const { selectedColumns, selectedEntry } = this.state
const { selectedEntry } = this.state
const selectedColumns = this.state.selectedColumns || this.defaultSelectedColumns()
const totalNumber = total || 0
......
......@@ -75,6 +75,7 @@ export default function DatasetPage() {
query={{dataset_id: [datasetId]}}
ownerTypes={['all', 'public']}
initialResultTab="entries"
resultListProps={{showAccessColumn: true}}
availableResultTabs={['entries', 'groups', 'datasets']}
/>
</div>
......
......@@ -217,7 +217,7 @@ function Markdown(props) {
return (
<Typography variant="body1"
className={classes.root}
dangerouslySetInnerHTML={{__html: marked(content)}}
dangerouslySetInnerHTML={{__html: marked(content || '')}}
/>
)
}
......
......@@ -54,7 +54,7 @@ Once you assigned a DOI to a dataset, no entries can be removed or added to the
function UserdataPage() {
return <Search
ownerTypes={['user', 'staging']}
ownerTypes={['user', 'shared', 'staging']}
initialOwner="user"
initialRequest={{order_by: 'upload_time', uploads_grouped: true}}
initialResultTab="uploads"
......
......@@ -63,6 +63,10 @@ class RawFiles extends React.Component {
whiteSpace: 'nowrap',
direction: 'rtl',
textAlign: 'left'
},
mainfileLabel: {
fontSize: '10pt',
fontWeight: 'bold'
}
})
......@@ -251,10 +255,10 @@ class RawFiles extends React.Component {
<div style={{width: '25%'}}>
{availableFiles.filter(this.filterPotcar.bind(this)).map((file, index) => (
<FormGroup row key={index} className={classes.fileNameFormGroup}>
<Tooltip title={file}>
<Tooltip title={file + (index === 0 ? ', the main (output) file of this entry' : '')}>
<FormControlLabel
style={{flexGrow: 1, overflowX: 'hidden', textOverflow: 'ellipsis'}}
label={this.label(file)}
label={<span>{this.label(file)}{index === 0 ? <b className={classes.mainfileLabel}> (mainfile)</b> : ''}</span>}
classes={{
root: classes.fileNameFormGroupLabel,
label: file === shownFile ? classes.shownFile : classes.fileNameLabel}}
......
......@@ -127,7 +127,7 @@ class DefinitionCardUnstyled extends React.Component {
}
renderDescription(description) {
description = description.replace(/[^([A-Za-z0-9](([A-Za-z0-9]+_)+[A-Za-z0-9]+)[^)[A-Za-z0-9]/g, '`$1`')
description = (description || '').replace(/[^([A-Za-z0-9](([A-Za-z0-9]+_)+[A-Za-z0-9]+)[^)[A-Za-z0-9]/g, '`$1`')
return (
<Markdown>{description}</Markdown>
)
......
import React from 'react'
import React, { useContext } from 'react'
import PropTypes from 'prop-types'
import { withStyles, Link, Typography, Tooltip, IconButton, TablePagination, Button } from '@material-ui/core'
import { compose } from 'recompose'
......@@ -9,19 +9,51 @@ import { Link as RouterLink } from 'react-router-dom'
import DetailsIcon from '@material-ui/icons/MoreHoriz'
import EditUserMetadataDialog from '../EditUserMetadataDialog'
import DownloadButton from '../DownloadButton'
import PublishedIcon from '@material-ui/icons/Public'
import PrivateIcon from '@material-ui/icons/AccountCircle'
import PublicIcon from '@material-ui/icons/Public'
import UploaderIcon from '@material-ui/icons/AccountCircle'
import SharedIcon from '@material-ui/icons/SupervisedUserCircle'
import PrivateIcon from '@material-ui/icons/VisibilityOff'
import { domains } from '../domains'
import { apiContext, withApi } from '../api'
export function Published(props) {
const api = useContext(apiContext)
const {entry} = props
if (entry.published) {
return <Tooltip title={entry.with_embargo ? 'published with embargo' : 'published'}>
{entry.with_embargo ? <PrivateIcon color="primary"/> : <PublishedIcon color="primary" />}
</Tooltip>
if (entry.with_embargo) {
if (api.user && entry.uploader.user_id === api.user.sub) {
if (entry.owners.length === 1) {
return <Tooltip title="published with embargo by you and only accessible by you">
<UploaderIcon color="error" />
</Tooltip>
} else {
return <Tooltip title="published with embargo by you and only accessible to you and users you shared the data with">
<SharedIcon color="error" />
</Tooltip>
}
} else if (api.user && entry.owners.find(user => user.user_id === api.user.sub)) {
return <Tooltip title="published with embargo and shared with you">
<SharedIcon color="error" />
</Tooltip>
} else {
if (api.user) {
return <Tooltip title="published with embargo and not accessible by you">
<PrivateIcon color="error" />
</Tooltip>
} else {
return <Tooltip title="published with embargo and might become accessible after login">
<PrivateIcon color="error" />
</Tooltip>
}
}
} else {
return <Tooltip title="published and accessible by everyone">
<PublicIcon color="primary" />
</Tooltip>
}
} else {
return <Tooltip title="not published yet">
<PrivateIcon color="error"/>
return <Tooltip title="you have not published this entry yet">
<UploaderIcon color="error"/>
</Tooltip>
}
}
......@@ -45,7 +77,8 @@ export class EntryListUnstyled extends React.Component {
showEntryActions: PropTypes.func,
selectedColumns: PropTypes.arrayOf(PropTypes.string),
domain: PropTypes.object,
user: PropTypes.object
user: PropTypes.object,
showAccessColumn: PropTypes.bool
}
static styles = theme => ({
......@@ -94,11 +127,6 @@ export class EntryListUnstyled extends React.Component {
supportsSort: true,
description: 'The time this entry was uploaded.'
},
published: {
label: 'Published',
align: 'center',
render: (entry) => <Published entry={entry} />
},
authors: {
label: 'Authors',
render: entry => entry.authors.map(author => author.name).join('; '),
......@@ -159,6 +187,11 @@ export class EntryListUnstyled extends React.Component {
},
supportsSort: false,
description: 'The dataset names that this entry belongs to.'
},
published: {
label: 'Access',
align: 'center',
render: (entry) => <Published entry={entry} />
}
}
......@@ -293,7 +326,7 @@ export class EntryListUnstyled extends React.Component {
}
render() {
const { classes, data, order, order_by, page, per_page, domain, editable, title, query, actions, ...rest } = this.props
const { classes, data, order, order_by, page, per_page, domain, editable, title, query, actions, user, showAccessColumn, ...rest } = this.props
const { selected } = this.state
const results = data.results || []
......@@ -305,8 +338,14 @@ export class EntryListUnstyled extends React.Component {
...EntryListUnstyled.defaultColumns
}
const defaultSelectedColumns = this.props.selectedColumns || [
...domain.defaultSearchResultColumns, 'authors']
let selectedColumns = this.props.selectedColumns
if (!selectedColumns) {
selectedColumns = [...domain.defaultSearchResultColumns]
if (user !== undefined || showAccessColumn) {
selectedColumns.push('published')
}
selectedColumns.push('authors')
}
const pagination = <TablePagination
count={totalNumber}
......@@ -341,7 +380,7 @@ export class EntryListUnstyled extends React.Component {
id={row => row.calc_id}
total={total}
columns={columns}
selectedColumns={defaultSelectedColumns}
selectedColumns={selectedColumns}
selectedColumnsKey="entries"
entryDetails={this.renderEntryDetails.bind(this)}
entryActions={this.renderEntryActions.bind(this)}
......@@ -361,6 +400,6 @@ export class EntryListUnstyled extends React.Component {
}
}
const EntryList = compose(withRouter, withStyles(EntryListUnstyled.styles))(EntryListUnstyled)
const EntryList = compose(withRouter, withApi(false, false), withStyles(EntryListUnstyled.styles))(EntryListUnstyled)
export default EntryList
......@@ -113,7 +113,7 @@ Search.propTypes = {
const useSearchEntryStyles = makeStyles(theme => ({
search: {
marginTop: theme.spacing(2),
marginBottom: theme.spacing(8),
marginBottom: theme.spacing(2),
maxWidth: 1024,
margin: 'auto',
width: '100%'
......@@ -130,7 +130,8 @@ const useSearchEntryStyles = makeStyles(theme => ({
marginRight: 0
},
searchBar: {
marginTop: theme.spacing(1)
marginTop: theme.spacing(1),
marginBottom: theme.spacing(1)
},
selectButton: {
margin: theme.spacing(1)
......@@ -227,7 +228,7 @@ function UsersVisualization() {