Commit 00a82ded authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'experimental' into 'master'

Latest fixes.

See merge request !44
parents e4c25332 cc998535
Pipeline #51087 failed with stages
in 49 seconds
......@@ -35,6 +35,7 @@ RUN pip install pandas
RUN pip install h5py
RUN pip install hjson
RUN pip install scipy
RUN pip install scikit-learn==0.20.2
RUN pip install ase==3.15.0
RUN pip install Pint==0.7.2
RUN pip install matid
......
Subproject commit c76dab1e28847bac58b247d9995fdf40d970bf41
Subproject commit 43d77d198acfc3137d1c4d08a4601248fbf8548d
Subproject commit c0e936246972c0fb98b9b9c96a67167f1725156f
Subproject commit 95b3874b72035d0529f7dd0cde419192dba68100
Subproject commit 3028201afe3f5b839afe3212769f193c9f634069
Subproject commit a2d395a391109a14a76345c9c0cc89fd7f89253d
Subproject commit aa26d45aafc2e19dda14da1523913eb480092218
Subproject commit e772c6d954009d7b3419200ffd370f55fd62c1a4
......@@ -5,11 +5,13 @@ import Markdown from './Markdown'
import { kibanaBase, apiBase } from '../config'
import { compose } from 'recompose'
import { withApi } from './api'
import { withDomain } from './domains'
class About extends React.Component {
static propTypes = {
classes: PropTypes.object.isRequired,
api: PropTypes.object.isRequired,
domain: PropTypes.object.isRequired,
raiseError: PropTypes.func.isRequired
}
......@@ -32,23 +34,13 @@ class About extends React.Component {
}
render() {
const { classes } = this.props
const { classes, domain } = this.props
const { info } = this.state
return (
<div className={classes.root}>
<Markdown>{`
## The nomad**@FAIR** prototype
This is a prototype, a concept, for a continuation of the
[NOMAD-coe](http://nomad-coe.eu) project. It is an attempt to redesign
the nomad software and infrastructure with the following goals in mind:
* more immediate and near-time use-modes (*staging area*, *code integration*, *on-site data*)
* 3rd parties run instances of the nomad on their servers (*mirrors*, *oasis*, *industry* usage)
* mirrors(partially) synchronize data with the central nomad instance (*data federation*)
* the nomad architecture/infrastructure is used for related *domains* (e.g. experimental material science) or even more unrelated domains
* nomad is integrated with existing *Open Data* initiatives and databases (*FAIRDI*, *EUDAT*, *optimade*)
* we benefit from nomad being *Open Source* (public git, outside participation)
${domain.about}
### Developer Documentation
You find in depth developer documentation [here](${apiBase}/docs/index.html).
......@@ -79,9 +71,9 @@ class About extends React.Component {
### Test user
During development this GUI might not be connected to the actual nomad
repository. Therefore, you cannot create a user or login with an existing
user. You might use our test users \`sheldon.cooper@nomad-fairdi.tests.de\`
or \`leonard.hofstadter@nomad-fairdi.tests.de\` both
with password \`password\`.
user. You might use the test user \`leonard.hofstadter@nomad-fairdi.tests.de\`
with password \`password\`. The user \`sheldon.cooper@nomad-fairdi.tests.de\` is
used for data that has no provenance with the original Nomad CoE database.
### About this version
- version: \`${info ? info.version : 'loading'}/${info ? info.release : 'loading'}\`
......@@ -96,4 +88,4 @@ class About extends React.Component {
}
}
export default compose(withApi(), withStyles(About.styles))(About)
export default compose(withApi(), withDomain, withStyles(About.styles))(About)
......@@ -191,7 +191,6 @@ class Api {
}))
.catch(this.handleApiError)
.then(response => {
console.log(response)
const result = response.body || response.text || response.data
if (typeof result === 'string') {
try {
......
......@@ -23,6 +23,63 @@ class DomainProviderBase extends React.Component {
domains = {
DFT: {
name: 'DFT',
about: `
## The nomad**@FAIRDI** beta test
### About nomad@FAIRDI
After the conclusion of the original [NOMAD-coe](http://nomad-coe.eu) project,
the newly founded NGO *FAIR Data Infrastructures* (FAIRDI) provides an
umbrella to continue operation and further development of the Nomad
material science data sharing platform.
The immediate goal is to to consolidate and stabilize the nomad infrastructure, and
as a first step, we refined the Nomad upload and data processing. This GUI introduces
the *staging area* that allows you to observe your uploads processing and inspect
the uploaded data before you decide to either publish your data or delete/upload
again.
Currently this is designed as just a complement to the original [Nomad Repository GUI](https://repository.nomad-coe.eu/NomadRepository-1.1).
You upload, process, inspect, and publish your data here. Here you have some
capabilities to search and explore uploaded dat. But to add comments, co-authors, and references,
create data-sets, and manage your account you still have to use the original [Nomad Repository GUI](https://repository.nomad-coe.eu/NomadRepository-1.1).
This GUI allows you to (menu on the left):
* About: read about this, access the documentation, and API.
* Search: inspect for existing data, your's and others.
* Upload: drop data, view the processing, and publish your uploads.
* Metainfo: browse the *metainfo*, Nomad's (meta-)data schema for processed data.
### How to test the new upload and processing
**!Please read this, before you explore this new part of Nomad!**
Try to explore this as a *new user*. Travel through the menu on the left and just
use it. Feel free to upload data, look for limitations and things you do not like.
The goal should be to figure out what is wrong and missing.
Keep in mind that there are limitations:
* You can only login with users that already exist in the Nomad Repository. However,
you can use our test user: \`leonard.hofstadter@nomad-fairdi.tests.de\`, the password
is \`password\`.
* This is not yet connected to the actual Nomad Repository. Everything you upload
will only appear here and might be removed after this first testing period.
* Now all existing entries from the original Nomad appear in the search, since we
are still migrating data.
For feedback and any issues you find, feel free to [open an issue](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/issues) or write
an email to [markus.scheidgen@physik.hu-berlin.de](mailto:markus.scheidgen@physik.hu-berlin.de).
### Mid- and long term goals of nomad@FAIRDI
* more immediate and near-time use-modes (*staging area*, *code integration*, *on-site data*)
* 3rd parties run instances of the nomad on their servers (*mirrors*, *oasis*, *industry* usage)
* mirrors(partially) synchronize data with the central nomad instance (*data federation*)
* the nomad architecture/infrastructure is used for related *domains* (e.g. experimental material science) or even more unrelated domains
* nomad is integrated with existing *Open Data* initiatives and databases (*FAIRDI*, *EUDAT*, *optimade*)
* we benefit from nomad being *Open Source* (public git, outside participation)
`,
entryLabel: 'calculation',
searchPlaceholder: 'enter atoms, codes, functionals, or other quantity values',
/**
* A component that is used to render the search aggregations. The components needs
* to work with props: aggregations (the aggregation data from the api),
......@@ -38,23 +95,28 @@ class DomainProviderBase extends React.Component {
searchMetrics: {
code_runs: {
label: 'Entries',
renderResultString: count => (<span><b>{count}</b> entries</span>)
tooltip: 'The statistics will show the number of database entry. Each set of input/output files that represents a code run is an entry.',
renderResultString: count => (<span><b>{count.toLocaleString()}</b> entries</span>)
},
unique_code_runs: {
label: 'Unique entries',
renderResultString: count => (<span> and <b>{count}</b> unique entries</span>)
tooltip: 'Counts duplicates only once.',
renderResultString: count => (<span> and <b>{count.toLocaleString()}</b> unique entries</span>)
},
total_energies: {
label: 'Total energy calculations',
renderResultString: count => (<span> with <b>{count}</b> total energy calculations</span>)
tooltip: 'Aggregates the number of total energy calculations as each entry can contain many calculations.',
renderResultString: count => (<span> with <b>{count.toLocaleString()}</b> total energy calculations</span>)
},
geometries: {
label: 'Unique geometries',
renderResultString: count => (<span> that simulate <b>{count}</b> unique geometries</span>)
tooltip: 'Aggregates the number of unique simulated system geometries in all entries.',
renderResultString: count => (<span> that simulate <b>{count.toLocaleString()}</b> unique geometries</span>)
},
datasets: {
label: 'Datasets',
renderResultString: count => (<span> curated in <b>{count}</b> datasets</span>)
tooltip: 'Shows statistics in terms of datasets that entries belong to.',
renderResultString: count => (<span> curated in <b>{count.toLocaleString()}</b> datasets</span>)
}
},
/**
......@@ -99,6 +161,23 @@ class DomainProviderBase extends React.Component {
},
EMS: {
name: 'EMS',
about: `
## A Prototype for Experimental Material Science Data Sharing
The original goal of the NOMAD CoE project was to provide a data sharing and
publication platform for computational material science data. With this prototype,
we want to apply Nomad ideas and implementations to experimental material science
data.
As a first step, this site demonstrates Nomad's \`domain specific\` search interface
and how experiment (meta-)data can be represented. We want to explore what
meta-data exists for material experiments, what is necessary to provide meaningful
search capabilities, how we can implement FAIR data sharing principles, and
how can we establish a community process to integrate the various experimental
methods and respective data.
`,
entryLabel: 'experiment',
searchPlaceholder: 'enter atoms, experimental methods, or other quantity values',
/**
* A component that is used to render the search aggregations. The components needs
* to work with props: aggregations (the aggregation data from the api),
......@@ -114,10 +193,12 @@ class DomainProviderBase extends React.Component {
searchMetrics: {
code_runs: {
label: 'Entries',
tooltip: 'Statistics will show the number of database entry. Usually each entry represents a single experiment.',
renderResultString: count => (<span><b>{count}</b> entries</span>)
},
datasets: {
label: 'Datasets',
tooltip: 'Shows statistics in terms of datasets that entries belong to.',
renderResultString: count => (<span> curated in <b>{count}</b> datasets</span>)
}
},
......@@ -137,7 +218,7 @@ class DomainProviderBase extends React.Component {
},
experiment_time: {
label: 'Date/Time',
render: time => new Date(time * 1000).toLocaleString()
render: time => time !== 'unavailable' ? new Date(time * 1000).toLocaleString() : time
}
},
/**
......
......@@ -2,6 +2,7 @@ import React from 'react'
import PropTypes from 'prop-types'
import { withStyles, Card, CardHeader, CardContent } from '@material-ui/core'
import RawFiles from '../entry/RawFiles'
import Markdown from '../Markdown'
class EMSEntryCards extends React.Component {
static propTypes = {
......@@ -11,7 +12,10 @@ class EMSEntryCards extends React.Component {
}
static styles = theme => ({
root: {}
root: {},
description: {
marginBottom: theme.spacing.unit * 3
}
})
render() {
......@@ -19,8 +23,16 @@ class EMSEntryCards extends React.Component {
return (
<Card className={classes.root}>
<CardHeader title="Raw files" />
<CardHeader title="Raw Data and Meta Data Files" />
<CardContent classes={{root: classes.cardContent}}>
<Markdown classes={{root: classes.description}}>{`
The data for this experiment was uploaded to [zenodo.org](https://zenodo.org).
Visit the zenodo entry to download the raw experiment data:
[${data.repository_url}](${data.repository_url}).
The meta data describing this experiment in its original format, can be
downloaded here directly:
`}</Markdown>
<RawFiles data={data} />
</CardContent>
</Card>
......
......@@ -2,6 +2,7 @@ import React from 'react'
import PropTypes from 'prop-types'
import Quantity from '../Quantity'
import { Typography } from '@material-ui/core'
import { apiBase } from '../../config'
export default class EMSEntryOverview extends React.Component {
static propTypes = {
......@@ -11,20 +12,35 @@ export default class EMSEntryOverview extends React.Component {
render() {
const { data } = this.props
const { preview_url } = data
let relative_preview_url = null
if (!preview_url) {
relative_preview_url = 'broken'
} else if (preview_url.indexOf('http://') === 0 || preview_url.indexOf('https://') === 0) {
relative_preview_url = preview_url
} else {
const dirname = data.mainfile.substring(0, data.mainfile.lastIndexOf('/'))
relative_preview_url = `${apiBase}/raw/${data.upload_id}/${dirname}/${preview_url}`
}
return (
<Quantity column>
<Quantity quantity="experiment_summary" label="summary" {...this.props} />
<Quantity row>
<Quantity column>
<Quantity row>
<Quantity quantity="formula" label="sample formula" noWrap {...this.props} />
<Quantity quantity="chemical" label="sample chemical" noWrap {...this.props} />
{data.chemical !== 'unavailable'
? <Quantity quantity="chemical" label="sample chemical" noWrap {...this.props} />
: ''}
</Quantity>
<Quantity quantity="method" label="experimental method" noWrap {...this.props} />
<Quantity quantity="experiment_location" label="experiment location" noWrap {...this.props} />
<Quantity label="experiment time" {...this.props}>
<Typography noWrap>
{new Date(data.experiment_time * 1000).toLocaleString()}
</Typography>
<Typography noWrap>{
data.experiment_time !== 'unavailable' ? new Date(data.experiment_time * 1000).toLocaleString() : 'unavailable'
}</Typography>
</Quantity>
<Quantity label="data" {...this.props}>
<Typography noWrap>
......@@ -33,7 +49,8 @@ export default class EMSEntryOverview extends React.Component {
</Quantity>
</Quantity>
<Quantity label="preview" {...this.props}>
<img alt="preview" style={{maxWidth: '100%', height: 'auto'}} src={data.preview_url}></img>
<img alt="preview" style={{maxWidth: '100%', height: 'auto'}} src={relative_preview_url}></img>
</Quantity>
</Quantity>
</Quantity>
)
......
......@@ -66,9 +66,11 @@ class EMSSearchAggregations extends React.Component {
<Grid container spacing={24} className={classes.quantityGrid}>
<Grid item xs={6}>
{quantity('method', 'Method')}
{quantity('probing_method', 'Probing')}
</Grid>
<Grid item xs={6}>
{quantity('experiment_location', 'Location')}
{quantity('sample_microstructure', 'Sample structure')}
{quantity('sample_constituents', 'Sample constituents')}
</Grid>
</Grid>
</div>
......
......@@ -85,10 +85,12 @@ class ArchiveEntryView extends React.Component {
this.props.raiseError(error)
})
api.getMetaInfo().then(metaInfo => {
api.getInfo().then(info => {
this.props.api.getMetaInfo(info.domain.metainfo.all_package).then(metaInfo => {
if (!this.unmounted) {
this.setState({metaInfo: metaInfo})
}
})
}).catch(error => {
this.props.raiseError(error)
})
......
......@@ -131,9 +131,9 @@ class RepoEntryView extends React.Component {
{new Date(calcData.upload_time * 1000).toLocaleString()}
</Typography>
</Quantity>
<Quantity quantity="calc_id" label='calculation id' noWrap {...quantityProps} />
<Quantity quantity="calc_id" label={`${domain.entryLabel} id`} noWrap {...quantityProps} />
<Quantity quantity='mainfile' loading={loading} noWrap {...quantityProps} />
<Quantity quantity="calc_hash" label='calculation hash' loading={loading} noWrap {...quantityProps} />
<Quantity quantity="calc_hash" label={`${domain.entryLabel} hash`} loading={loading} noWrap {...quantityProps} />
<Quantity quantity="last_processing" label='last processing' loading={loading} placeholder="not processed" noWrap {...quantityProps}>
<Typography noWrap>
{new Date(calcData.last_processing * 1000).toLocaleString()}
......
......@@ -39,6 +39,8 @@ class QuantityHistogram extends React.Component {
}
componentDidMount() {
// TODO this just a workaround for bad layout on initial rendering
this.updateChart()
this.updateChart()
}
......@@ -65,7 +67,8 @@ class QuantityHistogram extends React.Component {
const width = this.container.current.offsetWidth
const height = Object.keys(this.props.data).length * 32
const data = Object.keys(this.props.data).map(key => ({
const data = Object.keys(this.props.data)
.map(key => ({
name: key,
value: this.props.data[key][this.props.metric]
}))
......
import React from 'react'
import PropTypes from 'prop-types'
import { withStyles, FormControl, FormLabel, FormGroup, FormControlLabel, Checkbox } from '@material-ui/core'
import { withStyles, FormControl, FormLabel, FormGroup, FormControlLabel, Checkbox, Tooltip } from '@material-ui/core'
import { withDomain } from '../domains'
import { compose } from 'recompose'
......@@ -49,12 +49,14 @@ class SearchAggregationsUnstyled extends React.Component {
<FormLabel>Metric used in statistics: </FormLabel>
<FormGroup row>
{Object.keys(metricsDefinitions).map(metric => (
<FormControlLabel key={metric}
<Tooltip key={metric} title={metricsDefinitions[metric].tooltip}>
<FormControlLabel
control={
<Checkbox checked={selectedMetric === metric} onChange={() => this.handleMetricChange(metric)} value={metric} />
}
label={metricsDefinitions[metric].label}
/>
</Tooltip>
))}
</FormGroup>
</FormControl>
......
......@@ -69,7 +69,7 @@ class SearchPage extends React.Component {
state = {
data: SearchPage.emptySearchData,
owner: 'all',
owner: 'migrated',
searchState: {
...SearchAggregations.defaultState
},
......@@ -147,12 +147,23 @@ class SearchPage extends React.Component {
const { pagination: { total }, metrics } = data
const ownerLabel = {
migrated: 'Only migrated',
all: 'All entries',
public: 'Only public entries',
user: 'Only your entries',
staging: 'Only entries from your staging area'
staging: 'Staging area only'
}
const ownerTooltips = {
migrated: 'Only show entries with established provenance in the original Nomad repository.',
all: 'This will show all entries in the database, even those that might be duplicates.',
public: 'Do not show entries that are only visible to you.',
user: 'Do only show entries visible to you.',
staging: 'Will only show entries that you uploaded, but not yet published.'
}
const withoutLogin = ['migrated', 'all']
const useMetric = Object.keys(metrics).find(metric => metric !== 'code_runs') || 'code_runs'
const helperText = <span>
There are {Object.keys(domain.searchMetrics).map(key => {
......@@ -194,29 +205,31 @@ class SearchPage extends React.Component {
`}</Help>
<DisableOnLoading>
{ user
? <div className={classes.searchEntry}>
<div className={classes.searchEntry}>
<FormControl>
<FormLabel>Filter entries and show: </FormLabel>
<FormGroup row>
{['all', 'public', 'user', 'staging'].map(owner => (
<FormControlLabel key={owner}
{['migrated', 'all', 'public', 'user', 'staging']
.filter(key => user || withoutLogin.indexOf(key) !== -1)
.map(owner => (
<Tooltip key={owner} title={ownerTooltips[owner]}>
<FormControlLabel
control={
<Checkbox checked={this.state.owner === owner} onChange={() => this.handleOwnerChange(owner)} value="owner" />
}
label={ownerLabel[owner]}
/>
</Tooltip>
))}
</FormGroup>
</FormControl>
</div> : ''
}
</div>
<div className={classes.search}>
<SearchBar classes={{autosuggestRoot: classes.searchBar}}
fullWidth fullWidthInput={false} helperText={helperText}
label="search"
placeholder="enter atoms, codes, functionals, or other quantity values"
placeholder={domain.searchPlaceholder}
data={data} searchValues={searchValues}
InputLabelProps={{
shrink: true
......
......@@ -34,7 +34,7 @@ class SearchResultListUnstyled extends React.Component {
label: 'Authors',
render: (authors) => {
if (authors.length > 3) {
return authors.filter((_, index) => index < 2).map(author => author.name).join('; ') + 'et al'
return authors.filter((_, index) => index < 2).map(author => author.name).join('; ') + ' et al'
} else {
return authors.map(author => author.name).join('; ')
}
......
......@@ -98,8 +98,13 @@ class Upload extends React.Component {
const {page, perPage, orderBy, order} = params
this.state.upload.get(page, perPage, orderBy, order === 'asc' ? 1 : -1)
.then(upload => {
const {tasks_running, process_running, current_task} = upload
const {tasks_running, process_running, current_task, published} = upload
if (!this._unmounted) {
if (published) {
this.setState({...params})
this.props.onDoesNotExist()
return
}
const continueUpdating = tasks_running || process_running || current_task === 'uploading'
this.setState({upload: upload, params: params, updating: continueUpdating})
if (continueUpdating) {
......
......@@ -258,8 +258,9 @@ class Uploads extends React.Component {
<div className={classes.root}>
<Agree message={agreement} cookie="agreedToUploadTerms">
<Help cookie="uploadHelp" component={Markdown}>{`
To upload your own data, please put all relevant files in a
\`*.zip\` or \`*.tar.gz\` archive. We encourage you to add all code input and
To upload your own data, please put all relevant files of all the calculations
you want to upload into a single \`*.zip\` or \`*.tar.gz\` archive.
We encourage you to add all code input and
output files, as well as any other auxiliary files that you might have created.
You can put data from multiple calculations, using your preferred directory
structure, into your archives. Drop your archive file(s) below.
......@@ -272,6 +273,9 @@ class Uploads extends React.Component {
for your data, and later publish the data. The *embargo* might last up to
36 month before it becomes public automatically. During an *embargo*
some meta-data will be available.
There is a limit of 32 GB per upload. Please upload multiple archives, if
you have more than 32 GB of data to upload.
`}</Help>
<Paper className={classes.dropzoneContainer}>
......@@ -282,7 +286,7 @@ class Uploads extends React.Component {
rejectClassName={classes.dropzoneReject}
onDrop={this.onDrop.bind(this)}
>
<p>drop files here</p>
<p>drop .tar.gz or .zip files here</p>
<UploadIcon style={{fontSize: 36}}/>
</Dropzone>
</Paper>
......@@ -290,7 +294,7 @@ class Uploads extends React.Component {
<Help cookie="uploadCommandHelp">{`
Alternatively, you can upload files via the following shell command.
Replace \`<local_file>\` with your archive file. After executing the command,
return here and reload (e.g. press the reload button below).
return here and press the reload button below). The same 32 GB limit applies.
`}</Help>
<Markdown>{`
......
......@@ -22,6 +22,7 @@ import os.path
from flask import send_file
from flask_restplus import abort, Resource
import json
import importlib
import nomad_meta_info
......@@ -125,11 +126,11 @@ class MetainfoResource(Resource):
return load_metainfo(metainfo_package_name), 200
except FileNotFoundError:
parser_prefix = metainfo_package_name[:-len('.nomadmetainfo.json')]
alternative_path = os.path.join(
metainfo_main_path,
'../{0}parser/{0}.nomadmetainfo.json'.format(parser_prefix))
try:
return load_metainfo(alternative_path, is_path=True), 200
return load_metainfo(dict(
parser='%sparser' % parser_prefix,
path='%s.nomadmetainfo.json' % parser_prefix)), 200
except FileNotFoundError:
abort(404, message='The metainfo %s does not exist.' % metainfo_package_name)
......@@ -137,24 +138,53 @@ class MetainfoResource(Resource):
metainfo_main_path = os.path.dirname(os.path.abspath(nomad_meta_info.__file__))
def load_metainfo(package_name: str, is_path: bool = False, loaded_packages: Dict[str, Any] = None) -> Dict[str, Any]:
def load_metainfo(
package_name_or_dependency: str, dependency_source: str = None,
loaded_packages: Dict[str, Any] = None) -> Dict[str, Any]:
"""
Loads the given metainfo package and all its dependencies. Returns a dict with
all loaded package_names and respective packages.