Commit 385ec99b authored by Lauri Himanen's avatar Lauri Himanen
Browse files

Merge branch 'v0.8.0' into encyclopedia

parents b975d9da 2579d573
Pipeline #74359 passed with stages
in 19 minutes and 58 seconds
Accessing the Archive
=====================
Of course, you can access the NOMAD Archive directly via the NOMAD API (see the `API tutorial <api_tutorial.html>`_
and `API reference <api.html>`_). But, it is more effective and convenient to use NOMAD's Python client
library.
.. automodule:: nomad.client
\ No newline at end of file
......@@ -8,13 +8,14 @@ and infrastructure with a simplyfied architecture and consolidated code base.
:maxdepth: 2
introduction
setup
dev_guidelines
upload
api_tutorial
api
archive
metainfo
ops
setup
dev_guidelines
parser_tutorial
archive_tutorial
reference
gui
ops
......@@ -50,6 +50,10 @@ nomad.cli
------------
.. automodule:: nomad.cli
nomad.client
------------
.. automodule:: nomad.client
nomad.utils
-----------
.. automodule:: nomad.utils
......
==============
Uploading Data
==============
To contribute your data to the repository, please, login to our `upload page <../uploads>`_ (you need to register first, if you do not have a NOMAD account yet).
*A note for returning NOMAD users!* We revised the upload process with browser based upload
alongside new shell commands. The new Upload page allows you to monitor upload processing
and verify processing results before publishing your data to the Repository.
The `upload page <../uploads>`_ acts as a staging area for your data. It allows you to
upload data, to supervise the processing of your data, and to examine all metadata that
NOMAD extracts from your uploads. The data on the upload page will be private and can be
deleted again. If you are satisfied with our processing, you can publish the data.
Only then, data will become publicly available and cannot be deleted anymore.
You will always be able to access, share, and download your data. You may curate your data
and create datasets to give them a hierarchical structure. These functions are available
from the Your data page by selecting and editing data.
You should upload many files at the same time by creating .zip or .tar files of your folder structures.
Ideally, input and output files are accompanied by relevant auxiliary files. NOMAD will
consider everything within a single directory as related.
**A note for VASP users** on the handling of **POTCAR** files: NOMAD takes care of it; you don't
need to worry about it. We understand that according to your VASP license, POTCAR files are
not supposed to be visible to the public. Thus, in agreement with Georg Kresse, NOMAD will
extract the most important information of POTCAR files and store it in the files named
``POTCAR.stripped``. These files can be assessed and downloaded by anyone, while the original
POTCAR files are only available to the uploader and assigned co-authors.
This is done automatically; you don't need to do anything.
Once published, data cannot be erased. Linking a corrected version to a corresponding older one ("erratum") will be possible soon.
Files from an improved calculation, even for the same material, will be handled as a new entry.
You can publish data as being open access or restricted for up to three years (with embargo).
For the latter you may choose with whom you want to share your data. We strongly support the
idea of open access and thus suggest to impose as few restrictions as possible from the very
beginning. In case of open access data, all uploaded files are downloadable by any user.
Additional information, e.g. pointing to publications or how your data should be cited,
can be provided after the upload. Also DOIs can be requested. The restriction on data
can be lifted at any time. You cannot restrict data that was published as open access.
Unless published without an embargo, all your information will be private and only visible
to you (or NOMAD users you explicitly shared your data with). Viewing private data will
always require a login.
By uploading you confirm authorship of the uploaded calculations. Co-authors must be specified
after the upload process. This procedure is very much analogous to the submission of a
publication to a scientific journal.
Upload of data is free of charge.
\ No newline at end of file
......@@ -3,12 +3,14 @@ from nomad.client import query_archive
from nomad.metainfo import units
# this will not be necessary, once this is the official NOMAD version
config.client.url = 'https://labdev-nomad.esc.rzg.mpg.de/dev/nomad/v0-8-0/api'
config.client.url = 'https://labdev-nomad.esc.rzg.mpg.de/fairdi/nomad/testing-major/api'
aq = query_archive(
query = ArchiveQuery(
query={
'upload_id': ['6LUBCju3T3KK3D_fRCJ4qw']
'dft.compound_type': 'binary',
'dft.crystal_system': 'cubic',
'dft.code_name': 'FHI-aims',
'atoms': ['O']
},
required={
'section_run': {
......@@ -17,12 +19,11 @@ aq = query_archive(
}
}
},
per_page=100, max=1000)
print('total', aq.total)
per_page=10,
max=1000)
for i, e in enumerate(aq):
if i % 200 == 0:
print(e.section_run[0].section_single_configuration_calculation[0].energy_total.to(units.hartree))
print(query)
print(aq)
for result in query[0:10]:
energy = result.section_run[0].section_single_configuration_calculation[0].energy_total
print('Energy %s' % energy.to(units.hartree))
......@@ -4,6 +4,12 @@
"commit": "e98694e",
"private": true,
"dependencies": {
"@testing-library/jest-dom": "^4.2.4",
"@testing-library/react": "^9.3.2",
"@testing-library/user-event": "^7.1.2",
"react": "^16.13.1",
"react-dom": "^16.13.1",
"react-scripts": "3.4.1",
"@material-ui/core": "^4.0.0",
"@material-ui/icons": "^4.0.0",
"@material-ui/lab": "^4.0.0-alpha.49",
......@@ -26,19 +32,16 @@
"pace-js": "^1.0.2",
"piwik-react-router": "^0.12.1",
"qs": "^6.8.0",
"react": "^16.8.0",
"react-app-polyfill": "^1.0.1",
"react-autosuggest": "^9.4.3",
"react-cookie": "^3.0.8",
"react-copy-to-clipboard": "^5.0.1",
"react-dom": "^16.8.0",
"react-dropzone": "^5.0.1",
"react-highlight": "^0.12.0",
"react-infinite-scroller": "^1.2.4",
"react-json-view": "^1.19.1",
"react-keycloak": "^6.1.0",
"react-router-dom": "^5.1.2",
"react-scripts": "1.1.4",
"react-swipeable-views": "^0.13.0",
"recompose": "^0.28.2",
"swagger-client": "^3.8.22",
......@@ -51,13 +54,13 @@
"prebuild": "npm run generate-build-version",
"start": "react-scripts start",
"build": "react-scripts build",
"test": "react-scripts test --env=jsdom",
"test": "react-scripts test",
"eject": "react-scripts eject"
},
"devDependencies": {
"@material-ui/codemod": "^4.5.0",
"babel-eslint": "^8.2.6",
"eslint": "^4.19.1",
"babel-eslint": "^10.1.0",
"eslint": "^6.6.0",
"eslint-config-standard": "^11.0.0",
"eslint-plugin-import": "^2.14.0",
"eslint-plugin-node": "^8.0.1",
......@@ -67,5 +70,20 @@
"react-docgen": "^5.3.0",
"serve": "^10.0.0"
},
"eslintConfig": {
"extends": "react-app"
},
"browserslist": {
"production": [
">0.2%",
"not dead",
"not op_mini all"
],
"development": [
"last 1 chrome version",
"last 1 firefox version",
"last 1 safari version"
]
},
"homepage": "http://example.com/fairdi/nomad/latest/gui"
}
......@@ -30,8 +30,9 @@
-->
<!-- fonts needed by material-ui -->
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto+Mono:400,500">
<!-- <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500">
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto+Mono:400,500"> -->
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Titillium+Web:400,600">
<!-- icon fonts for the meta-info browser -->
<link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
......
......@@ -36,7 +36,7 @@ img.bg {
}
.pace .pace-progress {
background: #ffb300;
background: #00CED1;
position: fixed;
z-index: 2000;
top: 0;
......
import React from 'react'
import React, { useContext, useLayoutEffect, useRef, useCallback, useEffect } from 'react'
import {ReactComponent as AboutSvg} from './about.svg'
import PropTypes from 'prop-types'
import { withStyles } from '@material-ui/core/styles'
import Markdown from './Markdown'
import { appBase, optimadeBase, apiBase, debug, consent } from '../config'
import { compose } from 'recompose'
import { withApi } from './api'
import { apiContext } from './api'
import packageJson from '../../package.json'
import { domains } from './domains'
import { Grid, Card, CardContent, Typography, makeStyles, Link } from '@material-ui/core'
import { Link as RouterLink, useHistory } from 'react-router-dom'
class About extends React.Component {
static propTypes = {
classes: PropTypes.object.isRequired,
info: PropTypes.object,
raiseError: PropTypes.func.isRequired
const useCardStyles = makeStyles(theme => ({
title: {
marginBottom: theme.spacing(1)
}
}))
static styles = theme => ({
root: {
padding: theme.spacing(3)
function MarkdownCard({title, children, xs, top, bottom}) {
const classes = useCardStyles()
const style = {}
if (top) {
style['paddingBottom'] = 0
}
if (bottom) {
style['paddingTop'] = 0
}
return <Grid item xs={xs} style={style}>
<Card>
<CardContent>
<Typography variant="h6" className={classes.title}>{title}</Typography>
<Typography>{children}</Typography>
</CardContent>
</Card>
</Grid>
}
MarkdownCard.propTypes = {
title: PropTypes.string.isRequired,
children: PropTypes.node,
xs: PropTypes.number,
top: PropTypes.bool,
bottom: PropTypes.bool
}
const useStyles = makeStyles(theme => ({
root: {
padding: theme.spacing(3)
},
container: {
maxWidth: 1024,
margin: 'auto',
width: '100%'
}
}))
export default function About() {
const classes = useStyles()
const {info} = useContext(apiContext)
const svg = useRef()
const history = useHistory()
const makeClickable = useCallback((id, onClick) => {
const element = svg.current.querySelector('#' + id)
element.style.cursor = 'pointer'
element.firstChild.onclick = () => {
onClick()
}
})
}, [svg])
const setText = useCallback((id, lines) => {
const element = svg.current.querySelector('#' + id)
const x = element.getAttribute('x')
element.innerHTML = lines.map((line, i) => `<tspan x="${x}" dy="${i === 0 ? '0' : '1.2em'}">${line}</tspan>`).join('')
}, [svg])
useLayoutEffect(() => {
makeClickable('upload', () => {
history.push('/upload')
})
makeClickable('encyclopedia', () => {
window.location.href = 'https://encyclopedia.nomad-coe.eu/gui/#/search'
})
makeClickable('search', () => {
history.push('/search')
})
}, [svg])
render() {
const { classes, info } = this.props
useEffect(() => {
const statistics = (info && info.statistics) || {}
const value = (key, unit) => {
const nominal = statistics[key]
let stringValue = null
if (nominal) {
if (nominal >= 1.0e+9) {
stringValue = Math.floor(nominal / 1.0e+9) + ' bln.'
} else if (nominal >= 1.0e+6) {
stringValue = Math.floor(nominal / 1.0e+6) + ' mln.'
} else {
stringValue = Math.floor(nominal / 1.0e+3) + ' tsd.'
}
return `${stringValue || '...'} ${unit}`
}
}
setText('repositoryStats', [
value('n_entries', 'entries'),
value('n_uploads', 'uploads')
])
setText('archiveStats', [
value('n_calculations', 'results'),
value('n_quantities', 'quantities')
])
}, [svg, info])
return (
<div className={classes.root}>
return <div className={classes.root}>
<Grid className={classes.container} container spacing={2}>
<Grid item xs={12}>
<Markdown>{`
# The NOMAD Repository and Archive
This is the *graphical user interface* (GUI) for the NOMAD Repository and
Archive. It allows you to **search, access, and download all NOMAD data** in its
raw (Repository) and processed (Archive) form. You can **upload and manage your own
raw materials science data**. Learn more about what data can be uploaded
and how to prepare your data on the [NOMAD Repository homepage](https://repository.nomad-coe.eu/).
You can access all published data without an account. If you want to provide
your own data, please login or register for an account.
In the future, this web-page will include more and more features of other NOMAD
components as an effort to consolidate the various web applications from the
NOMAD Repository, Archive, Metainfo, Encyclopedia, and Analytics Toolkit.
### Getting Help
If you encounter any difficulties, please write to
[webmaster@nomad-coe.eu](mailto:webmaster@nomad-coe.eu). If you think
that this web-page is not working as expected, or if you want to start a discussion
about possible features, feel free to open an issue on our [issue tracking
system](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/issues).
### APIs
The NOMAD Repository and Archive can also be accessed programmatically via ReST APIs.
There is the proprietary NOMAD API and an implementation of the
[OPTiMaDe API (0.10.0)](https://github.com/Materials-Consortia/OPTiMaDe/tree/master)
standardized by the [OPTiMaDe consortium](https://www.optimade.org/)
Both APIs are described via [swagger](https://swagger.io/) (also known as OpenAPI spec.),
therefore you can use your favorite swagger client library
(e.g. [bravado](https://github.com/Yelp/bravado) for Python).
There are also web-based GUIs that allow to explore the APIs and their documentation:
- [NOMAD API](${apiBase}/)
- [OPTiMaDe API](${optimadeBase}/)
There is a [tutorial on how to use the API with Python](${appBase}/docs/api_tutorial.html).
There is also a Python library. You can use *pip* to install the library.
\`\`\`
pip install ${appBase}/dist/nomad-0.8.0.tar.gz
\`\`\`
The NOMAD Archive uses data that adheres to formal data definitions that we call
the NOMAD Metainfo. You can download these definition in their [JSON form
here](${apiBase}/archive/metainfo/all.nomadmetainfo.json). Otherwise, you
can use the Meta Info browser from the menu to explore.
${debug ? `
### Material science data and domains
Originally NOMAD was build for DFT calculations and data from the respective
community code. By NOMAD supports multiple materials science domains:
${info && info.domains.map(domain => domains[domain.name]).map(domain => `- ${domain.name}: ${domain.about}`).join('\n')}
` : ''}
### Developer Documentation
The [in-depth developer documentation](${appBase}/docs/index.html)
contains a general introduction to NOMAD, the underlying architecture,
is (meta)data, and processing. You will also find some information on how to use
the NOMAD ReST API. It contains information about how to develop NOMAD, how to
operate it, how to contribute parsers, and much more.
### Source code
The source-code for this new version of NOMAD (dubbed *nomad@FAIRDI*) is maintained
at the MPCDF's [gitlab](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR).
To push code, you need an MPCDF account and you can apply
[here](https://www.mpcdf.mpg.de/userspace/forms/onlineregistrationform).
${debug ? `
### Log management with Elastic stack
We use a central logging system based on the *elastic*-stack
(previously called *Elastic Logstash Kibana* (ELK)-stack).
This system pushes logs, events, monitoring data,
and other application metrics to a central database where it
can be analysed visually by us.
### Test user
During development this GUI might not be connected to the actual NOMAD
repository. Therefore, you cannot create a user or login with an existing
user. You might use the test user \`leonard.hofstadter@nomad-fairdi.tests.de\`
with password \`password\`. The user \`sheldon.cooper@nomad-fairdi.tests.de\` is
used for data that has no provenance with the original NOMAD CoE database.
` : ''}
### Terms of use and licenses
${consent}
### About this version
- version (API): \`${info ? info.version : 'loading'}/${info ? info.git.commit : 'loading'}\`
- version (GUI): \`${packageJson.version}/${packageJson.commit}\`
- domains: ${info ? Object.keys(info.domains).map(domain => info.domains[domain].name).join(', ') : 'loading'}
- git: \`${info ? info.git.ref : 'loading'}; ${info ? info.git.version : 'loading'}\`
- last commit message: *${info ? info.git.log : 'loading'}*
- supported codes: ${info ? info.codes.join(', ') : 'loading'}
- parsers: ${info ? info.parsers.join(', ') : 'loading'}
- normalizers: ${info ? info.normalizers.join(', ') : 'loading'}
# The NOMAD Repository and Archive
This is the *graphical user interface* (GUI) for the NOMAD Repository and
Archive. It allows you to **search, access, and download all NOMAD data** in its
raw (Repository) and processed (Archive) form. You can **upload and manage your own
raw materials science data**. Learn more about what data can be uploaded
and how to prepare your data on the [NOMAD Repository homepage](https://repository.nomad-coe.eu/).
You can access all published data without an account. If you want to provide
your own data, please login or register for an account.
`}</Markdown>
</div>
)
}
}
</Grid>
<MarkdownCard xs={6} title="Interactive Search" top>
NOMAD extracts <b>rich metadata</b> from uploaded raw-data. <Link component={RouterLink} to={'/search'}>
Explore NOMAD&apos;s data</Link> by creating complex queries from interactive data visualizations of key
properties, including the simulated composition/system, used method, upload metadata,
as well as material classifications and available quantities. Or use
the <b>Optimade</b> filter language to add arbitrarily nested queries.
</MarkdownCard>
<MarkdownCard xs={6} title="A common data format" top>
The <b>NOMAD Archive</b> provides data in processed and normalized form in a machine processable and common hierarchical format.
All data in the NOMAD Archive is organized into nested sections of quantities with well defined units,
data types, shapes, and descriptions. These definitions are called the <b>NOMAD Metainfo</b> and they
can be <Link component={RouterLink} to={'/metainfo'}>browsed here</Link>.
</MarkdownCard>
<Grid item xs={12} style={{paddingTop: 0, paddingBottom: 0}}>
<AboutSvg ref={svg}></AboutSvg>
</Grid>
<MarkdownCard xs={4} title="Uploading is simple" bottom>
<p>
You provide your own data <i>as is</i>. Just zip your code input and out files as they are,
including nested directory structures and potential auxiliary files, and upload
up to 32GB in a single .zip or .tar(.gz) file. NOMAD will automatically discover
and process the relevant files.
</p>
<p>
You can <b>privately</b> inspect, curate, or delete your data before publishing.
Data can be published with an <b>embargo (up to 3 years)</b> to only share data with
selected users.
</p>
<p>
Add additional metadata like <b>comments</b>, <b>references</b> to websites or papers, and your
<b>co-authors</b>. Curate your uploaded code runs into larger <b>datasets</b> and cite your data with a <b>DOI</b>
that we provide on request.
</p>
<p>
You can provide via GUI or shell command <Link component={RouterLink} to={'/uploads'}>here</Link>.
Manage already uploaded data <Link component={RouterLink} to={'/userdata'}>here</Link>.
</p>
</MarkdownCard>
<MarkdownCard xs={4} title="Processing" bottom>
<p>
Uploaded data is automatically processed and made available
in the uploaded <b>raw files</b> or in its processed and unified <b>Archive</b> form.
NOMAD parsers convert raw code input and output files into NOMAD&apos;s common data format.
You can inspect the Archive form and extracted metadata before
publishing your data.
</p>
<p>NOMAD supports most community codes: {info ? info.codes.join(', ') : '...'}</p>
<p>
To use NOMAD&apos;s parsers and normalizers outside of NOMAD.
Read <Link href="">here</Link> on how to install
our software and how to use NOMAD processing in your Python environment.
</p>
</MarkdownCard>
<MarkdownCard xs={4} title="APIs" bottom><Markdown>{`
The NOMAD can also be accessed programmatically via ReST APIs.
There is the proprietary NOMAD API and an implementation of the
standardized [OPTiMaDe API (0.10.0)](https://github.com/Materials-Consortia/OPTiMaDe/tree/master)
materials science database API.
Both APIs are described via [swagger/OpenAPI spec.](https://swagger.io/),
therefore you can use your favorite swagger client library
(e.g. [bravado](https://github.com/Yelp/bravado) for Python):
- [NOMAD API](${apiBase}/)
- [OPTiMaDe API](${optimadeBase}/)
There is a [tutorial on how to use the API with plain Python](${appBase}/docs/api_tutorial.html).
Another [tutorial covers how to install and use NOMAD's Python client library](${appBase}/docs/archive_tutorial.html).
The [NOMAD Analytics Toolkit](https://analytic-toolkit.nomad-coe.eu) allows to use
this without installation and directly on NOMAD servers.
`}</Markdown></MarkdownCard>
<Grid item xs={12}>
<Markdown>{`
### Getting Help
If you encounter any difficulties, please write to
[webmaster@nomad-coe.eu](mailto:webmaster@nomad-coe.eu). If you think
that this web-page is not working as expected, or if you want to start a discussion
about possible features, feel free to open an issue on our [issue tracking
system](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/issues).
### Developer Documentation
The [in-depth developer documentation](${appBase}/docs/index.html)
contains a general introduction to NOMAD, the underlying architecture,
is (meta)data, and processing. You will also find some information on how to use
the NOMAD ReST API. It contains information about how to develop NOMAD, how to
operate it, how to contribute parsers, and much more.
### Source code
The source-code for this new version of NOMAD (dubbed *nomad@FAIRDI*) is maintained
at the MPCDF's [gitlab](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR).
To push code, you need an MPCDF account and you can apply
[here](https://www.mpcdf.mpg.de/userspace/forms/onlineregistrationform).
export default compose(withApi(), withStyles(About.styles))(About)
${debug ? `
### Material science data and domains
Originally NOMAD was build for DFT calculations and data from the respective
community code. By NOMAD supports multiple materials science domains:
${info && info.domains.map(domain => domains[domain.name]).map(domain => `- ${domain.name}: ${domain.about}`).join('\n')}
` : ''}
${debug ? `
### Log management with Elastic stack
We use a central logging system based on the *elastic*-stack
(previously called *Elastic Logstash Kibana* (ELK)-stack).
This system pushes logs, events, monitoring data,
and other application metrics to a central database where it
can be analysed visually by us.
### Test user
During development this GUI might not be connected to the actual NOMAD
repository. Therefore, you cannot create a user or login with an existing
user. You might use the test user \`leonard.hofstadter@nomad-fairdi.tests.de\`
with password \`password\`. The user \`sheldon.cooper@nomad-fairdi.tests.de\` is
used for data that has no provenance with the original NOMAD CoE database.
` : ''}
### Terms of use and licenses
${consent}
### About this version
- version (API): \`${info ? info.version : 'loading'}/${info ? info.git.commit : 'loading'}\`
- version (GUI): \`${packageJson.version}/${packageJson.commit}\`
- domains: ${info ? Object.keys(info.domains).map(domain => info.domains[domain].name).join(', ') : 'loading'}
- git: \`${info ? info.git.ref : 'loading'}; ${info ? info.git.version : 'loading'}\`
- last commit message: *${info ? info.git.log : 'loading'}*
- supported codes: ${info ? info.codes.join(', ') : 'loading'}
- parsers: ${info ? info.parsers.join(', ') : 'loading'}
- normalizers: ${info ? info.normalizers.join(', ') : 'loading'}
`}</Markdown>
</Grid>
</Grid>
</div>
}
......@@ -5,7 +5,7 @@ import PropTypes, { instanceOf } from 'prop-types'
import { compose } from 'recompose'
import classNames from 'classnames'