Commit b41da687 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge commit 'c8ea64b0' into metainfo

parents 76eeefe3 c8ea64b0
Pipeline #71437 passed with stages
in 16 minutes and 4 seconds
......@@ -23,4 +23,4 @@ target/
vscode/
nomad.yaml
gunicorn.log.conf
gunicorn.conf
\ No newline at end of file
gunicorn.conf
......@@ -91,7 +91,7 @@ tests:
NOMAD_ELASTIC_HOST: elastic
NOMAD_MONGO_HOST: mongo
NOMAD_KEYCLOAK_PASSWORD: ${CI_KEYCLOAK_ADMIN_PASSWORD}
NOMAD_SPRINGER_DB_PATH: /nomad/fairdi/db/data/springer.db
NOMAD_NORMALIZE_SPRINGER_DB_PATH: /nomad/fairdi/db/data/springer.msg
script:
- cd /app
- ls /builds
......
Command Line Interface (CLI)
----------------------------
The :code:`nomad` python package comes with a command line interface (CLI) that
can be accessed after installation by simply running the :code:`nomad` command
in your terminal. The CLI provides a hiearchy of commands by using the `click
package <https://click.palletsprojects.com/>`_.
This documentation describes how the CLI can be used to manage a NOMAD
installation. For commmon use cases see :ref:`cli_use_cases`. For a full
reference of the CLI commands see :ref:`cli_ref`.
.. toctree::
:maxdepth: 2
cli_use_cases.rst
cli_ref.rst
.. _cli_ref:
CLI Reference
*************
Client CLI commands
""""""""""""""""""""""""""""""""""""""""
.. click:: nomad.cli.client.client:client
:prog: nomad client
:show-nested:
Admin CLI commands
""""""""""""""""""""""""""""""""""""""""
.. click:: nomad.cli.admin.admin:admin
:prog: nomad admin
:show-nested:
.. _cli_use_cases:
Use cases
*********
Mirroring data between production environments
""""""""""""""""""""""""""""""""""""""""""""""
Sometimes you would wish to transfer data between separate deployments of the
NOMAD infrastructure. This use case covers the situation when the deployments
are up and running and both have access to the underlying file storage, part of
which is mounted inside each container under :code:`.volumes/fs`.
With both the source and target deployment running, you can use the
:code::ref:`cli_ref:mirror` command to transfer the data from source to target. The
mirror will copy everything: i.e. the raw data, archive data and associated
metadata in the database.
The data to be mirrored is specified by using a query API path. For example to
mirror the upload from source deployment to target deployment, you would use
the following CLI command inside the target deployment:
.. code-block:: sh
nomad client -n <api_url> -u <username> -w <password> mirror <query_json> --source-mapping <target_docker_path>:<shared_path>
Here is a breakdown of the different arguments:
* :code:`-n <url>`: Url to the API endpoint in the source deployment. This API will
be queried to fetch the data to be mirrored. E.g.
http://repository.nomad-coe.eu/api
* :code:`-u <username>`: Your username that is used for authentication in the API call.
* :code:`-w <password>`: Your password that is used for authentication in the API call.
* :code:`mirror <query>`: Your query as a JSON dictionary. See the documentation for
available keywords. E.g. "{"upload_id: "<upload_id>"}"
* :code:`--source-mapping <mapping>`: The deployments use a separate folder to store
the archive and raw data. To correctly find the data that should be
mirrored, the absolute path on the filesystem that is shared between the
deployments needs to be provided. E.g. *.volumes/fs:/nomad/fairdi/prod/fs*.
The first part of this mapping indicates a docker volume path
(*.volumes/fs* in this example) that should be mapped to the second
filepath on the shared filesystem (*/nomad/fairdi/prod/fs* in this example).
Updating the AFLOW prototype information
""""""""""""""""""""""""""""""""""""""""
NOMAD uses the `AFLOW prototype library
<http://www.aflowlib.org/CrystalDatabase/>`_ to link bulk crystal entries with
prototypical structures based on their symmetry. The
:ref:`cli_ref:prototypes-update` subcommand can be used to update this
database from the online information provided by AFLOW. The command produces a
prototype dataset as a python module.
The dataset should be recreated if the AFLOW dataset has been updated or if the
symmetry matching routine used within NOMAD is updated (e.g. the symmetry
tolerance is modified). To produce a new dataset run the following command:
.. code-block:: sh
nomad admin ops prototypes-update <module_path>
Here is a breakdown of the different arguments:
* :code:`<module_name>`: Name of the python module in which the data should
be stored. If the file does not exist it will be created. The prototype
data used by NOMAD is under the path:
*nomad/normalizing/data/aflow_prototypes.py*
The command also provides a :code:`--matches-only` flag for only updating the
dataset entry that is used for matching the prototypes. This means that the
online information from AFLOW is not queried. This makes the process faster
e.g. in the case when you want only to update the matches after modifying the
symmetry routines.
......@@ -45,6 +45,8 @@ extensions = [
'sphinx.ext.coverage',
'sphinx.ext.ifconfig',
'sphinx.ext.napoleon',
'sphinx.ext.autosectionlabel',
'sphinx_click.ext',
'sphinxcontrib.httpdomain',
'sphinxcontrib.autohttp.flask',
'sphinxcontrib.autohttp.flaskqref',
......@@ -52,6 +54,9 @@ extensions = [
'm2r'
]
# Prefix the automatically generated labels with the document name
autosectionlabel_prefix_document = True
# Add any paths that contain templates here, relative to this directory.
templates_path = ['.templates']
......
.. mdinclude:: ../ops/docker-compose/nomad/README.md
.. mdinclude:: ../ops/helm/nomad/README.md
.. mdinclude:: ../ops/containers/README.md
.. mdinclude:: ../ops/docker-compose/nomad-oasis/README.md
Operating NOMAD
===============
###############
.. mdinclude:: ../ops/README.md
.. mdinclude:: ../ops/docker-compose/nomad/README.md
.. mdinclude:: ../ops/helm/nomad/README.md
.. mdinclude:: ../ops/containers/README.md
.. mdinclude:: ../ops/docker-compose/nomad-oasis/README.md
.. toctree::
:maxdepth: 2
depl_docker
depl_helm
depl_images
cli
oasis
......@@ -36,13 +36,15 @@ class DFTSearchAggregations extends React.Component {
<Grid item xs={4}>
<Quantity quantity="dft.code_name" title="Code" scale={0.25} metric={usedMetric} />
</Grid>
<Grid item xs={4}>
<Quantity quantity="dft.system" title="System type" scale={0.25} metric={usedMetric} />
<Quantity quantity="dft.crystal_system" title="Crystal system" scale={1} metric={usedMetric} />
</Grid>
<Grid item xs={4}>
<Quantity quantity="dft.basis_set" title="Basis set" scale={0.25} metric={usedMetric} />
<Quantity quantity="dft.xc_functional" title="XC functionals" scale={0.5} metric={usedMetric} />
<Quantity quantity="dft.compound_type" title="Compound type" scale={1} metric={usedMetric} />
</Grid>
<Grid item xs={4}>
<Quantity quantity="dft.system" title="System type" scale={0.25} metric={usedMetric} />
<Quantity quantity="dft.crystal_system" title="Crystal system" scale={1} metric={usedMetric} />
<Quantity quantity="dft.labels_springer_compound_class" title="Springer compound class" scale={1} metric={usedMetric} />
</Grid>
</Grid>
)
......
import React from 'react'
import PropTypes from 'prop-types'
import { Grid } from '@material-ui/core'
import { Quantity } from '../search/QuantityHistogram'
import SearchContext from '../search/SearchContext'
import { withApi } from '../api'
class DFTSearchByPropertyAggregations extends React.Component {
static propTypes = {
info: PropTypes.object
}
static contextType = SearchContext.type
render() {
const {info} = this.props
const {state: {response: {statistics}, usedMetric}} = this.context
if (statistics.code_name && info) {
// filter based on known codes, since elastic search might return 0 aggregations on
// obsolete code names
const filteredCodeNames = {}
const defaultValue = {
code_runs: 0
}
defaultValue[usedMetric] = 0
info.codes.forEach(key => {
filteredCodeNames[key] = statistics.code_name[key] || defaultValue
})
statistics.code_name = filteredCodeNames
}
return (
<Grid container spacing={24}>
<Grid item xs={4}>
<Quantity quantity="dft.quantities_energy" title="Energy" scale={1} metric={usedMetric} />
<Quantity quantity="dft.quantities_forces" title="Forces" scale={1} metric={usedMetric} />
<Quantity quantity="dft.quantities_electronic" title="Electronic" scale={1} metric={usedMetric} />
</Grid>
<Grid item xs={4}>
<Quantity quantity="dft.quantities_magnetic" title="Magnetic" scale={1} metric={usedMetric} />
<Quantity quantity="dft.quantities_vibrational" title="Vibrational" scale={1} metric={usedMetric} />
<Quantity quantity="dft.quantities_optical" title="Optical" scale={1} metric={usedMetric} />
</Grid>
<Grid item xs={4}>
<Quantity quantity="dft.labels_springer_classification" title="Springer classification" scale={1} metric={usedMetric} />
</Grid>
</Grid>
)
}
}
export default withApi(false, false)(DFTSearchByPropertyAggregations)
......@@ -5,6 +5,7 @@ import DFTEntryCards from './dft/DFTEntryCards'
import EMSSearchAggregations from './ems/EMSSearchAggregations'
import EMSEntryOverview from './ems/EMSEntryOverview'
import EMSEntryCards from './ems/EMSEntryCards'
import DFTSearchByPropertyAggregations from './dft/DFTSearchByPropertyAggregations'
export const domains = ({
dft: {
......@@ -23,6 +24,10 @@ export const domains = ({
* onChange (callback to propagate searchValue changes).
*/
SearchAggregations: DFTSearchAggregations,
/**
* A component that is used to render the search aggregations by property.
*/
SearchByPropertyAggregations: DFTSearchByPropertyAggregations,
/**
* Metrics are used to show values for aggregations. Each metric has a key (used
* for API calls), a label (used in the select form), and result string (to show
......
......@@ -9,6 +9,39 @@ import SearchContext from '../search/SearchContext'
const unprocessed_label = 'not processed'
const unavailable_label = 'unavailable'
const _mapping = {
'energy_total': 'Total energy',
'energy_total_T0': 'Total energy (0K)',
'energy_free': 'Free energy',
'energy_electrostatic': 'Electrostatic',
'energy_X': 'Exchange',
'energy_XC': 'Exchange-correlation',
'energy_sum_eigenvalues': 'Band energy',
'dos_values': 'DOS',
'eigenvalues_values': 'Eigenvalues',
'volumetric_data_values': 'Volumetric data',
'electronic_kinetic_energy': 'Kinetic energy',
'total_charge': 'Charge',
'atom_forces_free': 'Free atomic forces',
'atom_forces_raw': 'Raw atomic forces',
'atom_forces_T0': 'Atomic forces (0K)',
'atom_forces': 'Atomic forces',
'stress_tensor': 'Stress tensor',
'thermodynamical_property_heat_capacity_C_v': 'Heat capacity',
'vibrational_free_energy_at_constant_volume': 'Free energy (const=V)',
'band_energies': 'Band energies',
'spin_S2': 'Spin momentum operator',
'excitation_energies': 'Excitation energies',
'oscillator_strengths': 'Oscillator strengths',
'transition_dipole_moments': 'Transition dipole moments'}
function map_key (name) {
if (name in _mapping) {
return _mapping[name]
}
return name
}
class QuantityHistogramUnstyled extends React.Component {
static propTypes = {
classes: PropTypes.object.isRequired,
......@@ -69,7 +102,7 @@ class QuantityHistogramUnstyled extends React.Component {
const data = Object.keys(this.props.data)
.map(key => ({
name: key,
name: map_key(key),
value: this.props.data[key][this.props.metric]
}))
......
......@@ -16,6 +16,7 @@ import UploadList from './UploadsList'
import GroupList from './GroupList'
import ApiDialogButton from '../ApiDialogButton'
import SearchIcon from '@material-ui/icons/Search'
import UploadsChart from './UploadsChart'
class Search extends React.Component {
static tabs = {
......@@ -95,6 +96,16 @@ class Search extends React.Component {
render: props => <DomainVisualization {...props}/>,
label: 'Meta data',
description: 'Shows histograms on key metadata'
},
'property': {
render: props => <PropertyVisualization {...props}/>,
label: 'Properties',
description: 'Shows histograms on key properties'
},
'users': {
render: props => <UsersVisualization {...props}/>,
label: 'Users',
description: 'Show statistics on user metadata'
}
}
......@@ -215,6 +226,44 @@ class DomainVisualization extends React.Component {
}
}
class PropertyVisualization extends React.Component {
static propTypes = {
open: PropTypes.bool
}
static contextType = SearchContext.type
render() {
const {domain} = this.context.state
const {open} = this.props
return <KeepState visible={open} render={() =>
<domain.SearchByPropertyAggregations />
}/>
}
}
class UsersVisualization extends React.Component {
static propTypes = {
open: PropTypes.bool
}
static contextType = SearchContext.type
render () {
const {domain} = this.context.state
const {open} = this.props
return <KeepState visible={open} render={() =>
<Card>
<CardContent>
<UploadsChart metricsDefinitions={domain.searchMetrics}/>
</CardContent>
</Card>
}/>
}
}
class ElementsVisualization extends React.Component {
static propTypes = {
open: PropTypes.bool
......
......@@ -50,7 +50,8 @@ class SearchContext extends React.Component {
order_by: 'upload_time',
order: -1,
page: 1,
per_page: 10
per_page: 10,
date_histogram: true
},
metric: this.defaultMetric,
usedMetric: this.defaultMetric,
......
import React from 'react'
import PropTypes from 'prop-types'
import { withStyles, Select, MenuItem } from '@material-ui/core'
import Grid from '@material-ui/core/Grid'
import TextField from '@material-ui/core/TextField'
import * as d3 from 'd3'
import { scaleBand, scalePow } from 'd3-scale'
import { nomadSecondaryColor } from '../../config.js'
import SearchContext from './SearchContext'
import { compose } from 'recompose'
import { withApi } from '../api'
import { Quantity } from './QuantityHistogram'
class UploadsHistogramUnstyled extends React.Component {
static propTypes = {
classes: PropTypes.object.isRequired,
height: PropTypes.number.isRequired,
data: PropTypes.object,
interval: PropTypes.string,
metric: PropTypes.string.isRequired,
metricsDefinitions: PropTypes.object.isRequired,
onChanged: PropTypes.func.isRequired,
defaultScale: PropTypes.number
}
static styles = theme => ({
root: {},
content: {
paddingTop: 10
}
})
constructor(props) {
super(props)
this.state = {
scalePower: this.props.defaultScale || 1.0,
interval: this.props.interval || '1M',
time: null,
from_time: 0,
until_time: 0
}
this.container = React.createRef()
this.svgEl = React.createRef()
}
startDate = '2013-01-01'
scales = [
{
label: 'Linear',
value: 1.0
},
{
label: '1/2',
value: 0.5
},
{
label: '1/4',
value: 0.25
},
{
label: '1/8',
value: 0.25
}
]
intervals = [
{
label: 'Yearly',
value: '1y',
number: 31536000000
},
{
label: 'Monthly',
value: '1M',
number: 2678400000
},
{
label: 'Daily',
value: '1d',
number: 86400000
},
{
label: 'Hourly',
value: '1h',
number: 3600000
},
{
label: 'Minute',
value: '1m',
number: 60000
},
{
label: 'Second',
value: '1s',
number: 1000
}
]
timeInterval = Object.assign({}, ...this.intervals.map(e => ( {[e.value]: e.number})))
componentDidMount() {
const from_time = new Date(this.startDate).getTime()
const until_time = new Date().getTime()
this.handleTimeChange(from_time, 'from_time', 'all')
this.handleTimeChange(until_time, 'until_time', 'all')
}
componentDidUpdate() {
this.updateChart()
}
handleQueryChange() {
const interval = this.state.interval
const from_time = new Date(this.state.from_time)
const until_time = new Date(this.state.until_time)
this.props.onChanged(from_time.toISOString(), until_time.toISOString(), interval)
}
handleIntervalChange(newInterval) {
// TODO: add a refresh button so directly updating interval is not necessary
this.state.interval = newInterval
//this.setState({interval: newInterval})
this.handleQueryChange()
}
handleTimeChange(newTime, key, target) {
let date
if (!newTime) {
date = key === 'from_time' ? new Date(this.startDate) : new Date()
} else {
date = new Date(newTime)
}
if (target === 'state' || target === 'all') {
key === 'from_time' ? this.setState({from_time: date.getTime()}) : this.setState({until_time: date.getTime()})
}
if (target === 'picker' || target === 'all') {
document.getElementById(key).value = date.toISOString().substring(0,10)
}
}
handleItemClicked(item) {
const selected = item.time
if (selected === this.state.time) {
this.props.onChanged(null, null, null)
} else {
const deltaT = this.timeInterval[this.state.interval]
this.handleTimeChange(selected, 'from_time', 'all')
this.handleTimeChange(selected + deltaT, 'until_time', 'all')
this.handleQueryChange()
}
}
resolveDate (name) {
const date = new Date(parseInt(name, 10))
const year = date.toLocaleDateString(undefined, {year: 'numeric'})
const month = date.toLocaleDateString(undefined, {month: 'short'})
const day = date.toLocaleDateString(undefined, {day: 'numeric'})
const hour = date.toLocaleTimeString(undefined, {hour: 'numeric'})
const min = date.toLocaleTimeString(undefined, {minute: 'numeric'})
const sec= date.toLocaleTimeString(undefined, {second: 'numeric'})
const intervals = {
'1y': year,
'1M': month,
'1d': day,
'1h': hour,
'1m': min,
'1s': sec
}
return intervals[this.state.interval]
}
hover (svg, bar) {
const textOffset = 25
const tooltip = svg.append('g')
.attr('class', 'tooltip')
.style('display', 'none')
const hoverBox = tooltip.append('rect')
.attr('width', 10)
.attr('height', 20)
.attr('fill', 'white')
.style('opacity', 0.0)
const text = tooltip.append('text')
.attr('x', textOffset)
.attr('dy', '1.2em')
.style('text-anchor', 'start')
.attr('font-size', '12px')
//.attr('font-weight', 'bold')
bar
.on(