Commit 00a82ded authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'experimental' into 'master'

Latest fixes.

See merge request !44
parents e4c25332 cc998535
Pipeline #51087 failed with stages
in 49 seconds
...@@ -35,6 +35,7 @@ RUN pip install pandas ...@@ -35,6 +35,7 @@ RUN pip install pandas
RUN pip install h5py RUN pip install h5py
RUN pip install hjson RUN pip install hjson
RUN pip install scipy RUN pip install scipy
RUN pip install scikit-learn==0.20.2
RUN pip install ase==3.15.0 RUN pip install ase==3.15.0
RUN pip install Pint==0.7.2 RUN pip install Pint==0.7.2
RUN pip install matid RUN pip install matid
......
Subproject commit c76dab1e28847bac58b247d9995fdf40d970bf41 Subproject commit 43d77d198acfc3137d1c4d08a4601248fbf8548d
Subproject commit c0e936246972c0fb98b9b9c96a67167f1725156f Subproject commit 95b3874b72035d0529f7dd0cde419192dba68100
Subproject commit 3028201afe3f5b839afe3212769f193c9f634069 Subproject commit a2d395a391109a14a76345c9c0cc89fd7f89253d
Subproject commit aa26d45aafc2e19dda14da1523913eb480092218 Subproject commit e772c6d954009d7b3419200ffd370f55fd62c1a4
...@@ -5,11 +5,13 @@ import Markdown from './Markdown' ...@@ -5,11 +5,13 @@ import Markdown from './Markdown'
import { kibanaBase, apiBase } from '../config' import { kibanaBase, apiBase } from '../config'
import { compose } from 'recompose' import { compose } from 'recompose'
import { withApi } from './api' import { withApi } from './api'
import { withDomain } from './domains'
class About extends React.Component { class About extends React.Component {
static propTypes = { static propTypes = {
classes: PropTypes.object.isRequired, classes: PropTypes.object.isRequired,
api: PropTypes.object.isRequired, api: PropTypes.object.isRequired,
domain: PropTypes.object.isRequired,
raiseError: PropTypes.func.isRequired raiseError: PropTypes.func.isRequired
} }
...@@ -32,23 +34,13 @@ class About extends React.Component { ...@@ -32,23 +34,13 @@ class About extends React.Component {
} }
render() { render() {
const { classes } = this.props const { classes, domain } = this.props
const { info } = this.state const { info } = this.state
return ( return (
<div className={classes.root}> <div className={classes.root}>
<Markdown>{` <Markdown>{`
## The nomad**@FAIR** prototype ${domain.about}
This is a prototype, a concept, for a continuation of the
[NOMAD-coe](http://nomad-coe.eu) project. It is an attempt to redesign
the nomad software and infrastructure with the following goals in mind:
* more immediate and near-time use-modes (*staging area*, *code integration*, *on-site data*)
* 3rd parties run instances of the nomad on their servers (*mirrors*, *oasis*, *industry* usage)
* mirrors(partially) synchronize data with the central nomad instance (*data federation*)
* the nomad architecture/infrastructure is used for related *domains* (e.g. experimental material science) or even more unrelated domains
* nomad is integrated with existing *Open Data* initiatives and databases (*FAIRDI*, *EUDAT*, *optimade*)
* we benefit from nomad being *Open Source* (public git, outside participation)
### Developer Documentation ### Developer Documentation
You find in depth developer documentation [here](${apiBase}/docs/index.html). You find in depth developer documentation [here](${apiBase}/docs/index.html).
...@@ -79,9 +71,9 @@ class About extends React.Component { ...@@ -79,9 +71,9 @@ class About extends React.Component {
### Test user ### Test user
During development this GUI might not be connected to the actual nomad During development this GUI might not be connected to the actual nomad
repository. Therefore, you cannot create a user or login with an existing repository. Therefore, you cannot create a user or login with an existing
user. You might use our test users \`sheldon.cooper@nomad-fairdi.tests.de\` user. You might use the test user \`leonard.hofstadter@nomad-fairdi.tests.de\`
or \`leonard.hofstadter@nomad-fairdi.tests.de\` both with password \`password\`. The user \`sheldon.cooper@nomad-fairdi.tests.de\` is
with password \`password\`. used for data that has no provenance with the original Nomad CoE database.
### About this version ### About this version
- version: \`${info ? info.version : 'loading'}/${info ? info.release : 'loading'}\` - version: \`${info ? info.version : 'loading'}/${info ? info.release : 'loading'}\`
...@@ -96,4 +88,4 @@ class About extends React.Component { ...@@ -96,4 +88,4 @@ class About extends React.Component {
} }
} }
export default compose(withApi(), withStyles(About.styles))(About) export default compose(withApi(), withDomain, withStyles(About.styles))(About)
...@@ -191,7 +191,6 @@ class Api { ...@@ -191,7 +191,6 @@ class Api {
})) }))
.catch(this.handleApiError) .catch(this.handleApiError)
.then(response => { .then(response => {
console.log(response)
const result = response.body || response.text || response.data const result = response.body || response.text || response.data
if (typeof result === 'string') { if (typeof result === 'string') {
try { try {
......
...@@ -23,6 +23,63 @@ class DomainProviderBase extends React.Component { ...@@ -23,6 +23,63 @@ class DomainProviderBase extends React.Component {
domains = { domains = {
DFT: { DFT: {
name: 'DFT', name: 'DFT',
about: `
## The nomad**@FAIRDI** beta test
### About nomad@FAIRDI
After the conclusion of the original [NOMAD-coe](http://nomad-coe.eu) project,
the newly founded NGO *FAIR Data Infrastructures* (FAIRDI) provides an
umbrella to continue operation and further development of the Nomad
material science data sharing platform.
The immediate goal is to to consolidate and stabilize the nomad infrastructure, and
as a first step, we refined the Nomad upload and data processing. This GUI introduces
the *staging area* that allows you to observe your uploads processing and inspect
the uploaded data before you decide to either publish your data or delete/upload
again.
Currently this is designed as just a complement to the original [Nomad Repository GUI](https://repository.nomad-coe.eu/NomadRepository-1.1).
You upload, process, inspect, and publish your data here. Here you have some
capabilities to search and explore uploaded dat. But to add comments, co-authors, and references,
create data-sets, and manage your account you still have to use the original [Nomad Repository GUI](https://repository.nomad-coe.eu/NomadRepository-1.1).
This GUI allows you to (menu on the left):
* About: read about this, access the documentation, and API.
* Search: inspect for existing data, your's and others.
* Upload: drop data, view the processing, and publish your uploads.
* Metainfo: browse the *metainfo*, Nomad's (meta-)data schema for processed data.
### How to test the new upload and processing
**!Please read this, before you explore this new part of Nomad!**
Try to explore this as a *new user*. Travel through the menu on the left and just
use it. Feel free to upload data, look for limitations and things you do not like.
The goal should be to figure out what is wrong and missing.
Keep in mind that there are limitations:
* You can only login with users that already exist in the Nomad Repository. However,
you can use our test user: \`leonard.hofstadter@nomad-fairdi.tests.de\`, the password
is \`password\`.
* This is not yet connected to the actual Nomad Repository. Everything you upload
will only appear here and might be removed after this first testing period.
* Now all existing entries from the original Nomad appear in the search, since we
are still migrating data.
For feedback and any issues you find, feel free to [open an issue](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/issues) or write
an email to [markus.scheidgen@physik.hu-berlin.de](mailto:markus.scheidgen@physik.hu-berlin.de).
### Mid- and long term goals of nomad@FAIRDI
* more immediate and near-time use-modes (*staging area*, *code integration*, *on-site data*)
* 3rd parties run instances of the nomad on their servers (*mirrors*, *oasis*, *industry* usage)
* mirrors(partially) synchronize data with the central nomad instance (*data federation*)
* the nomad architecture/infrastructure is used for related *domains* (e.g. experimental material science) or even more unrelated domains
* nomad is integrated with existing *Open Data* initiatives and databases (*FAIRDI*, *EUDAT*, *optimade*)
* we benefit from nomad being *Open Source* (public git, outside participation)
`,
entryLabel: 'calculation',
searchPlaceholder: 'enter atoms, codes, functionals, or other quantity values',
/** /**
* A component that is used to render the search aggregations. The components needs * A component that is used to render the search aggregations. The components needs
* to work with props: aggregations (the aggregation data from the api), * to work with props: aggregations (the aggregation data from the api),
...@@ -38,23 +95,28 @@ class DomainProviderBase extends React.Component { ...@@ -38,23 +95,28 @@ class DomainProviderBase extends React.Component {
searchMetrics: { searchMetrics: {
code_runs: { code_runs: {
label: 'Entries', label: 'Entries',
renderResultString: count => (<span><b>{count}</b> entries</span>) tooltip: 'The statistics will show the number of database entry. Each set of input/output files that represents a code run is an entry.',
renderResultString: count => (<span><b>{count.toLocaleString()}</b> entries</span>)
}, },
unique_code_runs: { unique_code_runs: {
label: 'Unique entries', label: 'Unique entries',
renderResultString: count => (<span> and <b>{count}</b> unique entries</span>) tooltip: 'Counts duplicates only once.',
renderResultString: count => (<span> and <b>{count.toLocaleString()}</b> unique entries</span>)
}, },
total_energies: { total_energies: {
label: 'Total energy calculations', label: 'Total energy calculations',
renderResultString: count => (<span> with <b>{count}</b> total energy calculations</span>) tooltip: 'Aggregates the number of total energy calculations as each entry can contain many calculations.',
renderResultString: count => (<span> with <b>{count.toLocaleString()}</b> total energy calculations</span>)
}, },
geometries: { geometries: {
label: 'Unique geometries', label: 'Unique geometries',
renderResultString: count => (<span> that simulate <b>{count}</b> unique geometries</span>) tooltip: 'Aggregates the number of unique simulated system geometries in all entries.',
renderResultString: count => (<span> that simulate <b>{count.toLocaleString()}</b> unique geometries</span>)
}, },
datasets: { datasets: {
label: 'Datasets', label: 'Datasets',
renderResultString: count => (<span> curated in <b>{count}</b> datasets</span>) tooltip: 'Shows statistics in terms of datasets that entries belong to.',
renderResultString: count => (<span> curated in <b>{count.toLocaleString()}</b> datasets</span>)
} }
}, },
/** /**
...@@ -99,6 +161,23 @@ class DomainProviderBase extends React.Component { ...@@ -99,6 +161,23 @@ class DomainProviderBase extends React.Component {
}, },
EMS: { EMS: {
name: 'EMS', name: 'EMS',
about: `
## A Prototype for Experimental Material Science Data Sharing
The original goal of the NOMAD CoE project was to provide a data sharing and
publication platform for computational material science data. With this prototype,
we want to apply Nomad ideas and implementations to experimental material science
data.
As a first step, this site demonstrates Nomad's \`domain specific\` search interface
and how experiment (meta-)data can be represented. We want to explore what
meta-data exists for material experiments, what is necessary to provide meaningful
search capabilities, how we can implement FAIR data sharing principles, and
how can we establish a community process to integrate the various experimental
methods and respective data.
`,
entryLabel: 'experiment',
searchPlaceholder: 'enter atoms, experimental methods, or other quantity values',
/** /**
* A component that is used to render the search aggregations. The components needs * A component that is used to render the search aggregations. The components needs
* to work with props: aggregations (the aggregation data from the api), * to work with props: aggregations (the aggregation data from the api),
...@@ -114,10 +193,12 @@ class DomainProviderBase extends React.Component { ...@@ -114,10 +193,12 @@ class DomainProviderBase extends React.Component {
searchMetrics: { searchMetrics: {
code_runs: { code_runs: {
label: 'Entries', label: 'Entries',
tooltip: 'Statistics will show the number of database entry. Usually each entry represents a single experiment.',
renderResultString: count => (<span><b>{count}</b> entries</span>) renderResultString: count => (<span><b>{count}</b> entries</span>)
}, },
datasets: { datasets: {
label: 'Datasets', label: 'Datasets',
tooltip: 'Shows statistics in terms of datasets that entries belong to.',
renderResultString: count => (<span> curated in <b>{count}</b> datasets</span>) renderResultString: count => (<span> curated in <b>{count}</b> datasets</span>)
} }
}, },
...@@ -137,7 +218,7 @@ class DomainProviderBase extends React.Component { ...@@ -137,7 +218,7 @@ class DomainProviderBase extends React.Component {
}, },
experiment_time: { experiment_time: {
label: 'Date/Time', label: 'Date/Time',
render: time => new Date(time * 1000).toLocaleString() render: time => time !== 'unavailable' ? new Date(time * 1000).toLocaleString() : time
} }
}, },
/** /**
......
...@@ -2,6 +2,7 @@ import React from 'react' ...@@ -2,6 +2,7 @@ import React from 'react'
import PropTypes from 'prop-types' import PropTypes from 'prop-types'
import { withStyles, Card, CardHeader, CardContent } from '@material-ui/core' import { withStyles, Card, CardHeader, CardContent } from '@material-ui/core'
import RawFiles from '../entry/RawFiles' import RawFiles from '../entry/RawFiles'
import Markdown from '../Markdown'
class EMSEntryCards extends React.Component { class EMSEntryCards extends React.Component {
static propTypes = { static propTypes = {
...@@ -11,7 +12,10 @@ class EMSEntryCards extends React.Component { ...@@ -11,7 +12,10 @@ class EMSEntryCards extends React.Component {
} }
static styles = theme => ({ static styles = theme => ({
root: {} root: {},
description: {
marginBottom: theme.spacing.unit * 3
}
}) })
render() { render() {
...@@ -19,8 +23,16 @@ class EMSEntryCards extends React.Component { ...@@ -19,8 +23,16 @@ class EMSEntryCards extends React.Component {
return ( return (
<Card className={classes.root}> <Card className={classes.root}>
<CardHeader title="Raw files" /> <CardHeader title="Raw Data and Meta Data Files" />
<CardContent classes={{root: classes.cardContent}}> <CardContent classes={{root: classes.cardContent}}>
<Markdown classes={{root: classes.description}}>{`
The data for this experiment was uploaded to [zenodo.org](https://zenodo.org).
Visit the zenodo entry to download the raw experiment data:
[${data.repository_url}](${data.repository_url}).
The meta data describing this experiment in its original format, can be
downloaded here directly:
`}</Markdown>
<RawFiles data={data} /> <RawFiles data={data} />
</CardContent> </CardContent>
</Card> </Card>
......
...@@ -2,6 +2,7 @@ import React from 'react' ...@@ -2,6 +2,7 @@ import React from 'react'
import PropTypes from 'prop-types' import PropTypes from 'prop-types'
import Quantity from '../Quantity' import Quantity from '../Quantity'
import { Typography } from '@material-ui/core' import { Typography } from '@material-ui/core'
import { apiBase } from '../../config'
export default class EMSEntryOverview extends React.Component { export default class EMSEntryOverview extends React.Component {
static propTypes = { static propTypes = {
...@@ -11,30 +12,46 @@ export default class EMSEntryOverview extends React.Component { ...@@ -11,30 +12,46 @@ export default class EMSEntryOverview extends React.Component {
render() { render() {
const { data } = this.props const { data } = this.props
const { preview_url } = data
let relative_preview_url = null
if (!preview_url) {
relative_preview_url = 'broken'
} else if (preview_url.indexOf('http://') === 0 || preview_url.indexOf('https://') === 0) {
relative_preview_url = preview_url
} else {
const dirname = data.mainfile.substring(0, data.mainfile.lastIndexOf('/'))
relative_preview_url = `${apiBase}/raw/${data.upload_id}/${dirname}/${preview_url}`
}
return ( return (
<Quantity row> <Quantity column>
<Quantity column> <Quantity quantity="experiment_summary" label="summary" {...this.props} />
<Quantity row> <Quantity row>
<Quantity quantity="formula" label="sample formula" noWrap {...this.props} /> <Quantity column>
<Quantity quantity="chemical" label="sample chemical" noWrap {...this.props} /> <Quantity row>
</Quantity> <Quantity quantity="formula" label="sample formula" noWrap {...this.props} />
<Quantity quantity="method" label="experimental method" noWrap {...this.props} /> {data.chemical !== 'unavailable'
<Quantity quantity="experiment_location" label="experiment location" noWrap {...this.props} /> ? <Quantity quantity="chemical" label="sample chemical" noWrap {...this.props} />
<Quantity label="experiment time" {...this.props}> : ''}
<Typography noWrap> </Quantity>
{new Date(data.experiment_time * 1000).toLocaleString()} <Quantity quantity="method" label="experimental method" noWrap {...this.props} />
</Typography> <Quantity quantity="experiment_location" label="experiment location" noWrap {...this.props} />
<Quantity label="experiment time" {...this.props}>
<Typography noWrap>{
data.experiment_time !== 'unavailable' ? new Date(data.experiment_time * 1000).toLocaleString() : 'unavailable'
}</Typography>
</Quantity>
<Quantity label="data" {...this.props}>
<Typography noWrap>
<a href={data.repository_url}>{data.repository_name}</a>
</Typography>
</Quantity>
</Quantity> </Quantity>
<Quantity label="data" {...this.props}> <Quantity label="preview" {...this.props}>
<Typography noWrap> <img alt="preview" style={{maxWidth: '100%', height: 'auto'}} src={relative_preview_url}></img>
<a href={data.repository_url}>{data.repository_name}</a>
</Typography>
</Quantity> </Quantity>
</Quantity> </Quantity>
<Quantity label="preview" {...this.props}>
<img alt="preview" style={{maxWidth: '100%', height: 'auto'}} src={data.preview_url}></img>
</Quantity>
</Quantity> </Quantity>
) )
} }
......
...@@ -66,9 +66,11 @@ class EMSSearchAggregations extends React.Component { ...@@ -66,9 +66,11 @@ class EMSSearchAggregations extends React.Component {
<Grid container spacing={24} className={classes.quantityGrid}> <Grid container spacing={24} className={classes.quantityGrid}>
<Grid item xs={6}> <Grid item xs={6}>
{quantity('method', 'Method')} {quantity('method', 'Method')}
{quantity('probing_method', 'Probing')}
</Grid> </Grid>
<Grid item xs={6}> <Grid item xs={6}>
{quantity('experiment_location', 'Location')} {quantity('sample_microstructure', 'Sample structure')}
{quantity('sample_constituents', 'Sample constituents')}
</Grid> </Grid>
</Grid> </Grid>
</div> </div>
......
...@@ -85,10 +85,12 @@ class ArchiveEntryView extends React.Component { ...@@ -85,10 +85,12 @@ class ArchiveEntryView extends React.Component {
this.props.raiseError(error) this.props.raiseError(error)
}) })
api.getMetaInfo().then(metaInfo => { api.getInfo().then(info => {
if (!this.unmounted) { this.props.api.getMetaInfo(info.domain.metainfo.all_package).then(metaInfo => {
this.setState({metaInfo: metaInfo}) if (!this.unmounted) {
} this.setState({metaInfo: metaInfo})
}
})
}).catch(error => { }).catch(error => {
this.props.raiseError(error) this.props.raiseError(error)
}) })
......
...@@ -131,9 +131,9 @@ class RepoEntryView extends React.Component { ...@@ -131,9 +131,9 @@ class RepoEntryView extends React.Component {
{new Date(calcData.upload_time * 1000).toLocaleString()} {new Date(calcData.upload_time * 1000).toLocaleString()}
</Typography> </Typography>
</Quantity> </Quantity>
<Quantity quantity="calc_id" label='calculation id' noWrap {...quantityProps} /> <Quantity quantity="calc_id" label={`${domain.entryLabel} id`} noWrap {...quantityProps} />
<Quantity quantity='mainfile' loading={loading} noWrap {...quantityProps} /> <Quantity quantity='mainfile' loading={loading} noWrap {...quantityProps} />
<Quantity quantity="calc_hash" label='calculation hash' loading={loading} noWrap {...quantityProps} /> <Quantity quantity="calc_hash" label={`${domain.entryLabel} hash`} loading={loading} noWrap {...quantityProps} />
<Quantity quantity="last_processing" label='last processing' loading={loading} placeholder="not processed" noWrap {...quantityProps}> <Quantity quantity="last_processing" label='last processing' loading={loading} placeholder="not processed" noWrap {...quantityProps}>
<Typography noWrap> <Typography noWrap>
{new Date(calcData.last_processing * 1000).toLocaleString()} {new Date(calcData.last_processing * 1000).toLocaleString()}
......
...@@ -39,6 +39,8 @@ class QuantityHistogram extends React.Component { ...@@ -39,6 +39,8 @@ class QuantityHistogram extends React.Component {
} }
componentDidMount() { componentDidMount() {
// TODO this just a workaround for bad layout on initial rendering
this.updateChart()
this.updateChart() this.updateChart()
} }
...@@ -65,10 +67,11 @@ class QuantityHistogram extends React.Component { ...@@ -65,10 +67,11 @@ class QuantityHistogram extends React.Component {
const width = this.container.current.offsetWidth const width = this.container.current.offsetWidth
const height = Object.keys(this.props.data).length * 32 const height = Object.keys(this.props.data).length * 32
const data = Object.keys(this.props.data).map(key => ({ const data = Object.keys(this.props.data)
name: key, .map(key => ({
value: this.props.data[key][this.props.metric] name: key,
})) value: this.props.data[key][this.props.metric]
}))
data.sort((a, b) => { data.sort((a, b) => {
const nameA = a.name const nameA = a.name
......
import React from 'react' import React from 'react'
import PropTypes from 'prop-types' import PropTypes from 'prop-types'
import { withStyles, FormControl, FormLabel, FormGroup, FormControlLabel, Checkbox } from '@material-ui/core' import { withStyles, FormControl, FormLabel, FormGroup, FormControlLabel, Checkbox, Tooltip } from '@material-ui/core'
import { withDomain } from '../domains' import { withDomain } from '../domains'
import { compose } from 'recompose' import { compose } from 'recompose'
...@@ -49,12 +49,14 @@ class SearchAggregationsUnstyled extends React.Component { ...@@ -49,12 +49,14 @@ class SearchAggregationsUnstyled extends React.Component {
<FormLabel>Metric used in statistics: </FormLabel> <FormLabel>Metric used in statistics: </FormLabel>
<FormGroup row> <FormGroup row>
{Object.keys(metricsDefinitions).map(metric => ( {Object.keys(metricsDefinitions).map(metric => (
<FormControlLabel key={metric} <Tooltip key={metric} title={metricsDefinitions[metric].tooltip}>
control={ <FormControlLabel
<Checkbox checked={selectedMetric === metric} onChange={() => this.handleMetricChange(metric)} value={metric} /> control={
} <Checkbox checked={selectedMetric === metric} onChange={() => this.handleMetricChange(metric)} value={metric} />
label={metricsDefinitions[metric].label} }
/> label={metricsDefinitions[metric].label}
/>
</Tooltip>
))} ))}
</FormGroup> </FormGroup>
</FormControl> </FormControl>
......
...@@ -69,7 +69,7 @@ class SearchPage extends React.Component { ...@@ -69,7 +69,7 @@ class SearchPage extends React.Component {
state = { state = {
data: SearchPage.emptySearchData, data: SearchPage.emptySearchData,
owner: 'all', owner: 'migrated',
searchState: { searchState: {
...SearchAggregations.defaultState ...SearchAggregations.defaultState
},