Commit 21450c4a authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'v0.7.6' into dev-wip

parents 7512fcfb 83f50cfe
......@@ -31,6 +31,8 @@ Omitted versions are plain bugfix releases with only minor changes and fixes.
### v0.7.5
- AFLOWLIB prototypes (archive)
- primitive label search
- improved search performance based on excluded fields
- improved logs
- minor bugfixes
......
Subproject commit 5bfbdf09cbd0e9a2934467bdeca569296952ee06
Subproject commit b50054a10b28efddb82554051d797b44f2b1067e
Subproject commit 15d0110cbeda05aaea05e4d30ba3aeb0874dafef
Subproject commit 022a2af6bad45364dbdfac6b6c913f04186ac7d4
Subproject commit 5b601b99d9115b3062517d4a93fa372301bdc7e7
Subproject commit fe15759f080e8176d88af91447949243608b0d7e
......@@ -17,8 +17,13 @@ We do not assume many specific python packages. Only the *bravado* package (avai
via pipy) is required. It allows us to use the nomad ReST API in a more friendly and
pythonic way. You can simply install it the usual way
Optionally, if you need to access your private data, the package *python-keycloak* is
required to conveniently acquire the necessary tokens to authenticate your self towards
NOMAD.
```
pip install bravado
pip install python-keycloak
```
For the following code snippets, we need the following imports:
......@@ -33,6 +38,12 @@ import os.path
import sys
```
And optionally:
```python
from bravado.requests_client import RequestsClient, Authenticator
from keycloak import KeycloakOpenID
```
### An example file
Lets assume you have an example upload file ready. Its a `.zip` (`.tgz` would also work)
with some *VASP* data from a single run at `/example/AcAg/vasprun.xml`, `/example/AcAg/OUTCAR`, ...
......@@ -55,13 +66,47 @@ password = 'password'
### Using bravado
Bravado reads a ReST API's definition from a `swagger.json` as it is provided by
many APIs, including nomad's of course. Bravado also allows to use authentication,
which makes it even easier. The following would be a typical setup:
many APIs, including nomad's of course.
```python
host = urlparse(nomad_url).netloc.split(':')[0]
http_client = RequestsClient()
http_client.set_basic_auth(host, user, password)
client = SwaggerClient.from_url('%s/swagger.json' % nomad_url, http_client=http_client)
```
Bravado also allows to use authentication, if required. The following would be a typical setup:
```python
class KeycloakAuthenticator(Authenticator):
""" A bravado authenticator for NOMAD's keycloak-based user management. """
def __init__(self, user, password):
super().__init__(host=urlparse(nomad_url).netloc.split(':')[0])
self.user = user
self.password = password
self.token = None
self.__oidc = KeycloakOpenID(
server_url='https://repository.nomad-coe.eu/fairdi/keycloak/auth/',
realm_name='fairdi_nomad_prod',
client_id='nomad_public')
def apply(self, request):
if self.token is None:
self.token = self.__oidc.token(username=self.user, password=self.password)
self.token['time'] = time()
elif self.token['expires_in'] < int(time()) - self.token['time'] + 10:
try:
self.token = self.__oidc.refresh_token(self.token['refresh_token'])
self.token['time'] = time()
except Exception:
self.token = self.__oidc.token(username=self.user, password=self.password)
self.token['time'] = time()
request.headers.setdefault('Authorization', 'Bearer %s' % self.token['access_token'])
return request
http_client = RequestsClient()
http_client.authenticator = KeycloakAuthenticator(user=user, password=password)
client = SwaggerClient.from_url('%s/swagger.json' % nomad_url, http_client=http_client)
```
......@@ -192,7 +237,30 @@ print('%s/raw/%s/%s/*' % (nomad_url, calc['upload_id'], os.path.dirname(calc['ma
There are different options to download individual files, or zips with multiple files.
## Using *curl* to access the API
The shell tool *curl* can be used to call most API endpoints. Most endpoints for searching
or downloading data are only **GET** operations controlled by URL parameters. For example:
Downloading data:
```
curl http://repository.nomad-coe.eu/app/api/raw/query?upload_id=<your_upload_id> -o download.zip
```
It is a litle bit trickier, if you need to authenticate yourself, e.g. to download
not yet published or embargoed data. All endpoints support and most require the use of
an access token. To acquire an access token from our usermanagement system with curl:
```
curl --data 'grant_type=password&client_id=nomad_public&username=<your_username>&password=<your password>' \
https://repository.nomad-coe.eu/fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/token
```
You can use the access-token with:
```
curl -H 'Authorization: Bearer <you_access_token>' \
http://repository.nomad-coe.eu/app/api/raw/query?upload_id=<your_upload_id> -o download.zip
```
## Conclusions
This was just a small glimpse into the nomad API. You should checkout our swagger documentation
for more details on all the API endpoints and their parameters. You can explore the
API via swagger-ui and even try it in your browser. Just visit the API url.
This was just a small glimpse into the nomad API. You should checkout our [swagger-ui](https://repository.nomad-coe.eu/app/api/) for more details on all the API endpoints and their parameters. You can explore the
API via the swagger-ui and even try it in your browser.
{
"name": "nomad-fair-gui",
"version": "0.7.5",
"version": "0.7.6",
"commit": "nomad-gui-commit-placeholder",
"private": true,
"dependencies": {
......
......@@ -3,6 +3,8 @@ import { withStyles, Button, IconButton, Dialog, DialogTitle, DialogContent, Dia
import Markdown from './Markdown'
import PropTypes from 'prop-types'
import HelpIcon from '@material-ui/icons/Help'
import { compose } from 'recompose'
import { withDomain } from './domains'
export const HelpContext = React.createContext()
......@@ -10,9 +12,10 @@ class HelpDialogUnstyled extends React.Component {
static propTypes = {
classes: PropTypes.object.isRequired,
title: PropTypes.string,
content: PropTypes.string.isRequired,
content: PropTypes.func.isRequired,
icon: PropTypes.node,
maxWidth: PropTypes.string
maxWidth: PropTypes.string,
domain: PropTypes.object.isRequired
}
static styles = theme => ({
......@@ -38,7 +41,7 @@ class HelpDialogUnstyled extends React.Component {
}
render() {
const {classes, title, content, icon, maxWidth, ...rest} = this.props
const {classes, title, content, icon, maxWidth, domain, ...rest} = this.props
return (
<div className={classes.root}>
<Tooltip title={title}>
......@@ -53,7 +56,7 @@ class HelpDialogUnstyled extends React.Component {
>
<DialogTitle>{title || 'Help'}</DialogTitle>
<DialogContent>
<Markdown>{content}</Markdown>
<Markdown>{content(domain)}</Markdown>
</DialogContent>
<DialogActions>
<Button onClick={() => this.handleClose()} color="primary">
......@@ -66,4 +69,4 @@ class HelpDialogUnstyled extends React.Component {
}
}
export default withStyles(HelpDialogUnstyled.styles)(HelpDialogUnstyled)
export default compose(withDomain, withStyles(HelpDialogUnstyled.styles))(HelpDialogUnstyled)
......@@ -3,7 +3,7 @@ import { withApi } from './api'
import Search from './search/Search'
import SearchContext from './search/SearchContext'
export const help = `
export const help = domain => `
This page allows you to **inspect** and **manage** you own data. It is similar to the
*search page*, but it will only show data that was uploaded by you or is shared with you.
......
......@@ -207,6 +207,8 @@ class Api {
this.onStartLoading = () => null
this.onFinishLoading = () => null
this.statistics = {}
this._swaggerClient = Swagger(`${apiBase}/swagger.json`)
this.keycloak = keycloak
......@@ -384,9 +386,34 @@ class Api {
async search(search) {
this.onStartLoading()
return this.swagger()
.then(client => client.apis.repo.search(search))
.then(client => client.apis.repo.search({
exclude: ['atoms', 'only_atoms', 'files', 'quantities', 'optimade', 'labels', 'geometries'],
...search}))
.catch(handleApiError)
.then(response => response.body)
.then(response => {
// fill absent statistics values with values from prior searches
// this helps to keep consistent values, e.g. in the metadata search view
if (response.statistics) {
const empty = {}
Object.keys(response.statistics.total.all).forEach(metric => empty[metric] = 0)
Object.keys(response.statistics)
.filter(key => !['total', 'authors', 'atoms'].includes(key))
.forEach(key => {
if (!this.statistics[key]) {
this.statistics[key] = new Set()
}
const values = this.statistics[key]
Object.keys(response.statistics[key]).forEach(value => values.add(value))
values.forEach(value => {
if (!response.statistics[key][value]) {
response.statistics[key][value] = empty
}
})
})
}
return response
})
.finally(this.onFinishLoading)
}
......
......@@ -102,10 +102,10 @@ class DomainProviderBase extends React.Component {
defaultSearchMetric: 'code_runs',
additionalSearchKeys: {
raw_id: {},
external_id: {},
upload_id: {},
calc_id: {},
paths: {},
external_id: {},
pid: {},
mainfile: {},
calc_hash: {},
......@@ -113,7 +113,9 @@ class DomainProviderBase extends React.Component {
optimade: {},
quantities: {},
spacegroup: {},
specegroup_symbol: {}
spacegroup_symbol: {},
labels: {},
upload_name: {}
},
/**
* An dict where each object represents a column. Possible keys are label, render.
......
......@@ -21,13 +21,14 @@ class LogEntryUnstyled extends React.Component {
color: amber[700]
},
exception: {
overflowX: 'scroll',
margin: 0
}
})
render() {
const { classes, entry } = this.props
let data = undefined
let data
try {
data = JSON.parse(entry)
} catch (e) {
......@@ -86,9 +87,9 @@ class ArchiveLogView extends React.Component {
});
static defaultState = {
data: null,
doesNotExist: false
}
data: null,
doesNotExist: false
}
state = {...ArchiveLogView.defaultState}
......
......@@ -10,7 +10,7 @@ import qs from 'qs'
import KeepState from '../KeepState'
import { guiBase } from '../../config'
export const help=`
export const help = domain => `
The *raw files* tab, will show you all files that belong to the entry and offers a download
on individual, or all files. The files can be selected and downloaded. You can also
view the contents of some files directly here on this page.
......
......@@ -177,8 +177,8 @@ class RawFiles extends React.Component {
}
filterPotcar(file) {
if (file.toLowerCase().endsWith('potcar')) {
return this.props.data.uploader.user_id === this.props.user.sub
if (file.includes('POTCAR') && !file.endsWith('.stripped')) {
return this.props.user && this.props.data.uploader.user_id === this.props.user.sub
} else {
return true
}
......
......@@ -132,16 +132,16 @@ class RepoEntryView extends React.Component {
<Quantity column style={{maxWidth: 350}}>
<Quantity quantity="calc_id" label={`${domain.entryLabel} id`} noWrap withClipboard {...quantityProps} />
<Quantity quantity="pid" label='PID' loading={loading} placeholder="not yet assigned" noWrap {...quantityProps} withClipboard />
<Quantity quantity="raw_id" label='raw id' loading={loading} noWrap {...quantityProps} withClipboard />
<Quantity quantity="external_id" label='external id' loading={loading} noWrap {...quantityProps} withClipboard />
<Quantity quantity="mainfile" loading={loading} noWrap ellipsisFront {...quantityProps} withClipboard />
<Quantity quantity="calc_hash" label={`${domain.entryLabel} hash`} loading={loading} noWrap {...quantityProps} />
<Quantity quantity="upload_id" label='upload id' {...quantityProps} noWrap withClipboard />
<Quantity quantity="upload_time" label='upload time' noWrap {...quantityProps} >
<Typography noWrap>
{new Date(calcData.upload_time * 1000).toLocaleString()}
</Typography>
</Quantity>
<Quantity quantity='mainfile' loading={loading} noWrap ellipsisFront {...quantityProps} withClipboard />
<Quantity quantity="calc_hash" label={`${domain.entryLabel} hash`} loading={loading} noWrap {...quantityProps} />
<Quantity quantity="raw_id" label='raw id' loading={loading} noWrap {...quantityProps} withClipboard />
<Quantity quantity="external_id" label='external id' loading={loading} noWrap {...quantityProps} withClipboard />
<Quantity quantity="last_processing" label='last processing' loading={loading} placeholder="not processed" noWrap {...quantityProps}>
<Typography noWrap>
{new Date(calcData.last_processing * 1000).toLocaleString()}
......
......@@ -9,7 +9,7 @@ import { FormControl, withStyles, Select, Input, MenuItem, ListItemText, InputLa
import { compose } from 'recompose'
import { schema } from '../MetaInfoRepository'
export const help = `
export const help = domain => `
The NOMAD *metainfo* defines all quantities used to represent archive data in
NOMAD. You could say it is the archive *schema*. You can browse this schema and
all its definitions here.
......
......@@ -264,18 +264,10 @@ export class EntryListUnstyled extends React.Component {
<Quantity column >
{/* <Quantity quantity="pid" label='PID' placeholder="not yet assigned" noWrap data={row} withClipboard /> */}
<Quantity quantity="calc_id" label={`${domain.entryLabel} id`} noWrap withClipboard data={row} />
<Quantity quantity="upload_id" label='upload id' data={row} noWrap withClipboard />
<Quantity quantity="raw_id" label={`raw id`} noWrap withClipboard data={row} />
<Quantity quantity="external_id" label={`external id`} noWrap withClipboard data={row} />
<Quantity quantity='mainfile' noWrap ellipsisFront data={row} withClipboard />
<Quantity quantity="upload_time" label='upload time' noWrap data={row} >
<Typography noWrap>
{new Date(row.upload_time * 1000).toLocaleString()}
</Typography>
</Quantity>
<Quantity quantity="last_processing" label='processing version' noWrap placeholder="not processed" data={row}>
<Typography noWrap>
{row.nomad_version}/{row.nomad_commit}
</Typography>
</Quantity>
<Quantity quantity="upload_id" label='upload id' data={row} noWrap withClipboard />
</Quantity>
</div>
</div>
......
......@@ -7,8 +7,9 @@ import { withApi } from '../api'
import Search from './Search'
import SearchContext from './SearchContext'
import qs from 'qs'
import { withDomain } from '../domains'
export const help = `
export const help = domain => `
This page allows you to **search** in NOMAD's data. The upper part of this page
gives you various options to enter and configure your search. The lower part
shows all data that fulfills your search criteria.
......@@ -29,6 +30,11 @@ The visual representations show metrics for all data that fit your criteria.
You can display *entries* (i.e. code runs), *unique entries*, and *datasets*.
Other more specific metrics might be available.
Some quantities have no autocompletion for their values. You can still search for them,
if you know exactly what you are looking for. To search for a particular entry by its id
for example, type \`calc_id=<the_id>\` and press entry (or select the respective item from the menu).
The usable *hidden* quantities are: ${Object.keys(domain.additionalSearchKeys).map(key => `\`${key}\``).join(', ')}.
The results tabs gives you a quick overview of all entries and datasets that fit your search.
You can click entries to see more details, download data, see the archive, etc. The *entries*
tab displays individual entries (i.e. code runs), the *grouped entries* tab will group
......@@ -53,7 +59,8 @@ class SearchPage extends React.Component {
user: PropTypes.object,
location: PropTypes.object,
raiseError: PropTypes.func.isRequired,
update: PropTypes.number
update: PropTypes.number,
domain: PropTypes.object
}
static styles = theme => ({
......@@ -93,4 +100,4 @@ class SearchPage extends React.Component {
}
}
export default compose(withApi(false), withErrors, withStyles(SearchPage.styles))(SearchPage)
export default compose(withDomain, withApi(false), withErrors, withStyles(SearchPage.styles))(SearchPage)
......@@ -130,6 +130,10 @@ class Upload extends React.Component {
width: 350,
overflowX: 'hidden'
},
titleRow: {
display: 'flex',
flexDirection: 'row'
},
shortTitle: {
textOverflow: 'ellipsis',
whiteSpace: 'nowrap',
......@@ -374,7 +378,10 @@ class Upload extends React.Component {
return (
<div className={classes.titleContainer}>
<Typography variant="h6" className={name ? classes.shortTitle : classes.title}>
<div className={classes.titleRow}>
<Typography variant="h6" className={name ? classes.shortTitle : classes.title}>
{name || new Date(Date.parse(create_time)).toLocaleString()}
</Typography>
<CopyToClipboard
text={upload_id} onCopy={() => null}
>
......@@ -384,8 +391,7 @@ class Upload extends React.Component {
</IconButton>
</Tooltip>
</CopyToClipboard>
{name || new Date(Date.parse(create_time)).toLocaleString()}
</Typography>
</div>
{name
? <Typography variant="subtitle1">
{new Date(Date.parse(create_time)).toLocaleString()}
......
......@@ -17,7 +17,7 @@ import { CopyToClipboard } from 'react-copy-to-clipboard'
import { guiBase } from '../../config'
import qs from 'qs'
export const help = `
export const help = domain => `
NOMAD allows you to upload data. After upload, NOMAD will process your data: it will
identify the main output files of [supported codes](https://www.nomad-coe.eu/the-project/nomad-repository/nomad-repository-howtoupload)
and then it will parse these files. The result will be a list of entries (one per each identified mainfile).
......@@ -308,7 +308,7 @@ class UploadPage extends React.Component {
</Tooltip>
{/* <button>Copy to clipboard with button</button> */}
</CopyToClipboard>
<HelpDialog icon={<MoreIcon/>} maxWidth="md" title="Alternative shell commands" content={`
<HelpDialog icon={<MoreIcon/>} maxWidth="md" title="Alternative shell commands" content={domain => `
As an experienced shell and *curl* user, you can modify the commands to
your liking.
......
......@@ -147,6 +147,7 @@ class ArchiveDownloadResource(Resource):
search_request = search.SearchRequest()
apply_search_parameters(search_request, args)
search_request.include('calc_id', 'upload_id', 'mainfile')
calcs = search_request.execute_scan(
order_by='upload_id',
......@@ -273,6 +274,7 @@ class ArchiveQueryResource(Resource):
search_request = search.SearchRequest()
apply_search_parameters(search_request, args)
search_request.include('calc_id', 'upload_id', 'mainfile')
try:
if scroll:
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment