Commit 14329873 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'v0.7.10' into v0.8.0<

parents 2b83f245 22b09bb2
...@@ -90,7 +90,6 @@ tests: ...@@ -90,7 +90,6 @@ tests:
NOMAD_RABBITMQ_HOST: rabbitmq NOMAD_RABBITMQ_HOST: rabbitmq
NOMAD_ELASTIC_HOST: elastic NOMAD_ELASTIC_HOST: elastic
NOMAD_MONGO_HOST: mongo NOMAD_MONGO_HOST: mongo
NOMAD_KEYCLOAK_CLIENT_SECRET: ${CI_KEYCLOAK_TEST_CLIENT_SECRET}
NOMAD_KEYCLOAK_PASSWORD: ${CI_KEYCLOAK_ADMIN_PASSWORD} NOMAD_KEYCLOAK_PASSWORD: ${CI_KEYCLOAK_ADMIN_PASSWORD}
NOMAD_SPRINGER_DB_PATH: /nomad/fairdi/db/data/springer.db NOMAD_SPRINGER_DB_PATH: /nomad/fairdi/db/data/springer.db
script: script:
......
...@@ -29,6 +29,14 @@ contributing, and API reference. ...@@ -29,6 +29,14 @@ contributing, and API reference.
Omitted versions are plain bugfix releases with only minor changes and fixes. Omitted versions are plain bugfix releases with only minor changes and fixes.
### v0.7.9
- Everything to run a simple NOMAD OASIS based on the central user-management
- minor bugfixes
### v0.7.7
- Shows dataset contents with embargo data, but hides the entry details (raw-files, archive)
- minor bugfixes
### v0.7.5 ### v0.7.5
- AFLOWLIB prototypes (archive) - AFLOWLIB prototypes (archive)
- primitive label search - primitive label search
......
Subproject commit b50054a10b28efddb82554051d797b44f2b1067e Subproject commit d918460c31728058834432b736062d44e1e1c074
Subproject commit cd354f066cb8b85904a2725bb93abf7c443b3fdf Subproject commit d30ef0bd9275206380866c89946a0c129e7d8df9
Subproject commit 92005ec9ff4b8e13bd86373d14bd5fafe2b52cd1 Subproject commit 75a5cd92dbd6299067e0fca0b9949f8b4410ec91
Subproject commit 5f97f32086c281ebda5ab6084ae2c7eba16b516f Subproject commit e113cbf21f23054394ad6099ad4836cbd9e21790
Subproject commit b932711d741c2457a80bf2447c180ce49c23e6c9 Subproject commit bd5b5c6f947ec9b7172ef7970a92825c737e1e60
Subproject commit fe15759f080e8176d88af91447949243608b0d7e Subproject commit d60013e1597493972237210a36549bfcf0a2706f
Subproject commit 3811ced85fb7d68ca579d5ca8d93e800f48c53a5 Subproject commit f2b7f39ca62438d25a21cdbaf267269fbc4f62ac
Subproject commit d9c9b3c14ecab80e58adab70917267e5e7fbe3f2 Subproject commit 5f07d80f9d1838b3f6b95e39266221002061e0d1
Operating nomad Operating NOMAD
=============== ===============
.. mdinclude:: ../ops/README.md .. mdinclude:: ../ops/README.md
.. mdinclude:: ../ops/docker-compose/nomad/README.md .. mdinclude:: ../ops/docker-compose/nomad/README.md
.. mdinclude:: ../ops/helm/nomad/README.md .. mdinclude:: ../ops/helm/nomad/README.md
.. mdinclude:: ../ops/containers/README.md .. mdinclude:: ../ops/containers/README.md
.. mdinclude:: ../ops/docker-compose/nomad-oasis/README.md
...@@ -11,18 +11,19 @@ The nomad infrastructure consists of a series of nomad and 3rd party services: ...@@ -11,18 +11,19 @@ The nomad infrastructure consists of a series of nomad and 3rd party services:
- rabbitmq: a task queue used to distribute work in a cluster - rabbitmq: a task queue used to distribute work in a cluster
All 3rd party services should be run via *docker-compose* (see blow). The All 3rd party services should be run via *docker-compose* (see blow). The
nomad python services can also be run via *docker-compose* or manually started with python. nomad python services can be run with python to develop them.
The gui can be run manually with a development server via yarn, or with The gui can be run with a development server via yarn.
*docker-compose*
Below you will find information on how to install all python dependencies and code Below you will find information on how to install all python dependencies and code
manually. How to use *docker*/*docker-compose*. How run services with *docker-compose* manually. How to use *docker*/*docker-compose*. How run 3rd-party services with *docker-compose*.
or manually.
Keep in mind the *docker-compose* configures all services in a way that mirror Keep in mind the *docker-compose* configures all services in a way that mirror
the configuration of the python code in `nomad/config.py` and the gui config in the configuration of the python code in `nomad/config.py` and the gui config in
`gui/.env.development`. `gui/.env.development`.
To learn about how to run everything in docker, e.g. to operate a NOMAD OASIS in
production, go (here)(/app/docs/ops.html).
## Install python code and dependencies ## Install python code and dependencies
### Cloning and development tools ### Cloning and development tools
...@@ -158,35 +159,12 @@ having to copy the git itself to the docker build context. ...@@ -158,35 +159,12 @@ having to copy the git itself to the docker build context.
The images are build via *docker-compose* and don't have to be created manually. The images are build via *docker-compose* and don't have to be created manually.
### Build with docker-compose ### Run necessary 3-rd party services with docker-compose
We have multiple *docker-compose* files that must be used together.
- `docker-compose.yml` contains the base definitions for all services
- `docker-compose.override.yml` configures services for development (notably builds images for nomad services)
- `docker-compose.dev-elk.yml` will also provide the ELK service
- `docker-compose.prod.yml` configures services for production (notable uses a pre-build image for nomad services that was build during CI/CD)
It is sufficient to use the implicit `docker-compose.yml` only (like in the command below).
The `override` will be used automatically.
Now we can build the *docker-compose* that contains all external services (rabbitmq,
mongo, elastic, elk) and nomad services (worker, app, gui).
```
docker-compose build
```
Docker-compose tries to cache individual building steps. Sometimes this causes
troubles and not everything necessary is build when you changed something. In
this cases use:
```
docker-compose build --no-cache
```
### Run everything with docker-compose
You can run all containers with: You can run all containers with:
``` ```
docker-compose up cd ops/docker-compose/nomad
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d mongo elastic rabbitmq
``` ```
To shut down everything, just `ctrl-c` the running output. If you started everything To shut down everything, just `ctrl-c` the running output. If you started everything
...@@ -195,25 +173,6 @@ in *deamon* mode (`-d`) use: ...@@ -195,25 +173,6 @@ in *deamon* mode (`-d`) use:
docker-compose down docker-compose down
``` ```
### Run containers selectively
The following services/containers are managed via our docker-compose:
- rabbitmq, mongo, elastic, (elk, only for production)
- worker, app
- gui
- proxy
The *proxy* container runs *nginx* based reverse proxies that put all services under
a single port and different paths.
You can also run services selectively, e.g.
```
docker-compose up -d rabbitmq, mongo, elastic
docker-compose up worker
docker-compose up app gui proxy
```
## Accessing 3'rd party services
Usually these services only used by the nomad containers, but sometimes you also Usually these services only used by the nomad containers, but sometimes you also
need to check something or do some manual steps. need to check something or do some manual steps.
...@@ -234,12 +193,7 @@ The index prefix for logs is `logstash-`. The ELK is only available with the ...@@ -234,12 +193,7 @@ The index prefix for logs is `logstash-`. The ELK is only available with the
You can access mongodb and elastic search via your preferred tools. Just make sure You can access mongodb and elastic search via your preferred tools. Just make sure
to use the right ports (see above). to use the right ports (see above).
## Run nomad services
## Run nomad services manually
You can run the worker, app, and gui as part of the docker infrastructure, like
seen above. But, of course there are always reasons to run them manually during
development, like running them in a debugger, profiler, etc.
### API and worker ### API and worker
...@@ -253,11 +207,6 @@ To run it directly with celery, do (from the root) ...@@ -253,11 +207,6 @@ To run it directly with celery, do (from the root)
celery -A nomad.processing worker -l info celery -A nomad.processing worker -l info
``` ```
Run the app via docker, or (from the root):
```
nomad admin run app
```
You can also run worker and app together: You can also run worker and app together:
``` ```
nomad admin run appworker nomad admin run appworker
......
{ {
"name": "nomad-fair-gui", "name": "nomad-fair-gui",
"version": "0.7.6", "version": "0.7.10",
"commit": "nomad-gui-commit-placeholder", "commit": "nomad-gui-commit-placeholder",
"private": true, "private": true,
"dependencies": { "dependencies": {
......
...@@ -3,12 +3,13 @@ import PropTypes from 'prop-types' ...@@ -3,12 +3,13 @@ import PropTypes from 'prop-types'
import { withStyles } from '@material-ui/core/styles' import { withStyles } from '@material-ui/core/styles'
import { compose } from 'recompose' import { compose } from 'recompose'
import { withErrors } from './errors' import { withErrors } from './errors'
import { withApi, DoesNotExist } from './api' import { withApi } from './api'
import Search from './search/Search' import Search from './search/Search'
import SearchContext from './search/SearchContext' import SearchContext from './search/SearchContext'
import { Typography } from '@material-ui/core' import { Typography } from '@material-ui/core'
import { DatasetActions, DOI } from './search/DatasetList' import { DatasetActions, DOI } from './search/DatasetList'
import { withRouter } from 'react-router' import { withRouter } from 'react-router'
import { withDomain } from './domains'
export const help = ` export const help = `
This page allows you to **inspect** and **download** NOMAD datasets. It alsow allows you This page allows you to **inspect** and **download** NOMAD datasets. It alsow allows you
...@@ -21,7 +22,8 @@ class DatasetPage extends React.Component { ...@@ -21,7 +22,8 @@ class DatasetPage extends React.Component {
api: PropTypes.object.isRequired, api: PropTypes.object.isRequired,
datasetId: PropTypes.string.isRequired, datasetId: PropTypes.string.isRequired,
raiseError: PropTypes.func.isRequired, raiseError: PropTypes.func.isRequired,
history: PropTypes.object.isRequired history: PropTypes.object.isRequired,
domain: PropTypes.object.isRequired
} }
static styles = theme => ({ static styles = theme => ({
...@@ -44,6 +46,7 @@ class DatasetPage extends React.Component { ...@@ -44,6 +46,7 @@ class DatasetPage extends React.Component {
state = { state = {
dataset: {}, dataset: {},
empty: false,
update: 0 update: 0
} }
...@@ -58,14 +61,13 @@ class DatasetPage extends React.Component { ...@@ -58,14 +61,13 @@ class DatasetPage extends React.Component {
const entry = data.results[0] const entry = data.results[0]
const dataset = entry && entry.datasets.find(ds => ds.id + '' === datasetId) const dataset = entry && entry.datasets.find(ds => ds.id + '' === datasetId)
if (!dataset) { if (!dataset) {
this.setState({dataset: {}}) this.setState({dataset: {}, empty: true})
raiseError(new DoesNotExist('Dataset does not exist any more or is not visible to you.'))
} }
this.setState({dataset: { this.setState({dataset: {
...dataset, example: entry ...dataset, example: entry, empty: false
}}) }})
}).catch(error => { }).catch(error => {
this.setState({dataset: {}}) this.setState({dataset: {}, empty: false})
raiseError(error) raiseError(error)
}) })
} }
...@@ -76,7 +78,7 @@ class DatasetPage extends React.Component { ...@@ -76,7 +78,7 @@ class DatasetPage extends React.Component {
componentDidUpdate(prevProps) { componentDidUpdate(prevProps) {
if (prevProps.api !== this.props.api || prevProps.datasetId !== this.props.datasetId) { if (prevProps.api !== this.props.api || prevProps.datasetId !== this.props.datasetId) {
this.setState({dataset: {}}, () => this.update()) this.setState({dataset: {}, empty: false}, () => this.update())
} }
} }
...@@ -89,14 +91,14 @@ class DatasetPage extends React.Component { ...@@ -89,14 +91,14 @@ class DatasetPage extends React.Component {
} }
render() { render() {
const { classes, datasetId } = this.props const { classes, datasetId, domain } = this.props
const { dataset, update } = this.state const { dataset, update, empty } = this.state
return ( return (
<div> <div>
<div className={classes.header}> <div className={classes.header}>
<div className={classes.description}> <div className={classes.description}>
<Typography variant="h4">{dataset.name || 'loading ...'}</Typography> <Typography variant="h4">{dataset.name || (empty && 'Empty or non existing dataset') ||'loading ...'}</Typography>
<Typography> <Typography>
dataset{dataset.doi ? <span>, with DOI <DOI doi={dataset.doi} /></span> : ''} dataset{dataset.doi ? <span>, with DOI <DOI doi={dataset.doi} /></span> : ''}
</Typography> </Typography>
...@@ -111,13 +113,20 @@ class DatasetPage extends React.Component { ...@@ -111,13 +113,20 @@ class DatasetPage extends React.Component {
</div> </div>
<SearchContext <SearchContext
query={{dataset_id: datasetId}} ownerTypes={['all', 'public']} update={update} initialQuery={{owner: 'all'}}
query={{dataset_id: datasetId}}
ownerTypes={['all', 'public']} update={update}
> >
<Search resultTab="entries" tabs={['entries', 'groups', 'datasets']} /> <Search
resultTab="entries" tabs={['entries', 'groups', 'datasets']}
entryListProps={{
selectedColumns: [...domain.defaultSearchResultColumns, 'published', 'authors']
}}
/>
</SearchContext> </SearchContext>
</div> </div>
) )
} }
} }
export default compose(withRouter, withApi(false), withErrors, withStyles(DatasetPage.styles))(DatasetPage) export default compose(withRouter, withDomain, withApi(false), withErrors, withStyles(DatasetPage.styles))(DatasetPage)
...@@ -647,7 +647,7 @@ class InviteUserDialogUnstyled extends React.Component { ...@@ -647,7 +647,7 @@ class InviteUserDialogUnstyled extends React.Component {
If you want to add a user as co-author or share your data with someone that If you want to add a user as co-author or share your data with someone that
is not already a NOMAD user, you can invite this person here. We need just a few is not already a NOMAD user, you can invite this person here. We need just a few
details about this person. After your invite, the new user will receive an details about this person. After your invite, the new user will receive an
Email that allows him to set a password and further details. Anyhow, you will Email that allows her to set a password and further details. Anyhow, you will
be able to add the user as co-author or someone to share with immediately after the be able to add the user as co-author or someone to share with immediately after the
invite. invite.
</DialogContentText> </DialogContentText>
...@@ -730,7 +730,8 @@ class EditUserMetadataDialogUnstyled extends React.Component { ...@@ -730,7 +730,8 @@ class EditUserMetadataDialogUnstyled extends React.Component {
user: PropTypes.object, user: PropTypes.object,
onEditComplete: PropTypes.func, onEditComplete: PropTypes.func,
disabled: PropTypes.bool, disabled: PropTypes.bool,
title: PropTypes.string title: PropTypes.string,
info: PropTypes.object
} }
static styles = theme => ({ static styles = theme => ({
...@@ -1055,7 +1056,7 @@ class EditUserMetadataDialogUnstyled extends React.Component { ...@@ -1055,7 +1056,7 @@ class EditUserMetadataDialogUnstyled extends React.Component {
} }
renderDialogActions(submitting, submitEnabled) { renderDialogActions(submitting, submitEnabled) {
const {classes} = this.props const {classes, info} = this.props
if (submitting) { if (submitting) {
return <DialogActions> return <DialogActions>
...@@ -1070,7 +1071,7 @@ class EditUserMetadataDialogUnstyled extends React.Component { ...@@ -1070,7 +1071,7 @@ class EditUserMetadataDialogUnstyled extends React.Component {
</DialogActions> </DialogActions>
} else { } else {
return <DialogActions> return <DialogActions>
<InviteUserDialog /> {info && !info.oasis && <InviteUserDialog />}
<span style={{flexGrow: 1}} /> <span style={{flexGrow: 1}} />
<Button onClick={this.handleClose} disabled={submitting}> <Button onClick={this.handleClose} disabled={submitting}>
Cancel Cancel
......
...@@ -107,12 +107,15 @@ class FAQ extends React.Component { ...@@ -107,12 +107,15 @@ class FAQ extends React.Component {
publishing anything. publishing anything.
Second, you can publish your uploads with an *embargo* period. This can last up to Second, you can publish your uploads with an *embargo* period. This can last up to
3 years. You can lift the embargo at anytime. This allows you to privately create 3 years. You can lift the embargo at anytime. Embargoed data is
datasets and DOIs, share data with selected people, before your work is published, e.g. visible to and findable by others. This makes only some few metadata (e.g.
in a paper. chemical formula, system type, spacegroup, etc.) public, but the raw-file
and archive contents remain hidden (except to you, and users you explicitly
Non *published* and *embargoed* data is only visible to you (the uploader) and users share the data with).
that you explicitly share your entries with. You can already create datasets and assign DOIs for data with embargo, e.g.
to put it into your unpublished paper.
The embargo will last up to 36 month. Afterwards, your data will be made publicly
available. You can also lift the embargo on entries at any time.
### How do I cite uploaded data in a paper? ### How do I cite uploaded data in a paper?
......
...@@ -308,12 +308,13 @@ class Api { ...@@ -308,12 +308,13 @@ class Api {
.finally(this.onFinishLoading) .finally(this.onFinishLoading)
} }
async getRawFile(uploadId, path, kwargs) { async getRawFile(uploadId, calcId, path, kwargs) {
this.onStartLoading() this.onStartLoading()
const length = (kwargs && kwargs.length) || 4096 const length = (kwargs && kwargs.length) || 4096
return this.swagger() return this.swagger()
.then(client => client.apis.raw.get({ .then(client => client.apis.raw.get_file_from_calc({
upload_id: uploadId, upload_id: uploadId,
calc_id: calcId,
path: path, path: path,
decompress: true, decompress: true,
...(kwargs || {}), ...(kwargs || {}),
...@@ -697,6 +698,8 @@ export const ApiProvider = compose(withKeycloak, withErrors)(ApiProviderComponen ...@@ -697,6 +698,8 @@ export const ApiProvider = compose(withKeycloak, withErrors)(ApiProviderComponen
const LoginRequired = withStyles(LoginRequiredUnstyled.styles)(LoginRequiredUnstyled) const LoginRequired = withStyles(LoginRequiredUnstyled.styles)(LoginRequiredUnstyled)
const __reauthorize_trigger_changes = ['api', 'calcId', 'uploadId', 'calc_id', 'upload_id']
class WithApiComponent extends React.Component { class WithApiComponent extends React.Component {
static propTypes = { static propTypes = {
raiseError: PropTypes.func.isRequired, raiseError: PropTypes.func.isRequired,
...@@ -718,7 +721,7 @@ class WithApiComponent extends React.Component { ...@@ -718,7 +721,7 @@ class WithApiComponent extends React.Component {
} }
componentDidUpdate(prevProps) { componentDidUpdate(prevProps) {
if (prevProps.api !== this.props.api) { if (__reauthorize_trigger_changes.find(key => this.props[key] !== prevProps[key])) {
this.setState({notAuthorized: false}) this.setState({notAuthorized: false})
} }
} }
...@@ -746,12 +749,12 @@ class WithApiComponent extends React.Component { ...@@ -746,12 +749,12 @@ class WithApiComponent extends React.Component {
if (notAuthorized) { if (notAuthorized) {
if (keycloak.authenticated) { if (keycloak.authenticated) {
return ( return (
<div> <div style={{marginTop: 24}}>
<Typography variant="h6">Not Authorized</Typography> <Typography variant="h6">Not Authorized</Typography>
<Typography> <Typography>
You are not authorized to access this information. If someone send You are not authorized to access this information. If someone send
you this link, ask him to make his data publicly available or share you a link to this data, ask the authors to make the data publicly available
it with you. or share it with you.
</Typography> </Typography>
</div> </div>
) )
......
...@@ -130,9 +130,9 @@ class RawFiles extends React.Component { ...@@ -130,9 +130,9 @@ class RawFiles extends React.Component {
} }
handleFileClicked(file) { handleFileClicked(file) {
const {api, uploadId, raiseError} = this.props const {api, uploadId, calcId, raiseError} = this.props
this.setState({shownFile: file, fileContents: null}) this.setState({shownFile: file, fileContents: null})
api.getRawFile(uploadId, file, {length: 16 * 1024}) api.getRawFile(uploadId, calcId, file.split('/').reverse()[0], {length: 16 * 1024})
.then(contents => this.setState({fileContents: contents})) .then(contents => this.setState({fileContents: contents}))
.catch(raiseError) .catch(raiseError)
} }
...@@ -154,7 +154,7 @@ class RawFiles extends React.Component { ...@@ -154,7 +154,7 @@ class RawFiles extends React.Component {
}) })
if (fileContents.contents.length < (page + 1) * 16 * 1024) { if (fileContents.contents.length < (page + 1) * 16 * 1024) {
api.getRawFile(uploadId, shownFile, {offset: page * 16 * 1024, length: 16 * 1024}) api.getRawFile(uploadId, calcId, shownFile.split('/').reverse()[0], {offset: page * 16 * 1024, length: 16 * 1024})
.then(contents => { .then(contents => {
const {fileContents} = this.state const {fileContents} = this.state
// The back-button navigation might cause a scroll event, might cause to loadmore, // The back-button navigation might cause a scroll event, might cause to loadmore,
...@@ -200,6 +200,20 @@ class RawFiles extends React.Component { ...@@ -200,6 +200,20 @@ class RawFiles extends React.Component {
</Typography> </Typography>
} }
let downloadUrl
if (selectedFiles.length === 1) {
// download the individual file
downloadUrl = `raw/${uploadId}/${selectedFiles[0]}`
} else if (selectedFiles.length === availableFiles.length) {
// use an endpoint that downloads all files of the calc
downloadUrl = `raw/calc/${uploadId}/${calcId}/*?strip=true`
} else if (selectedFiles.length > 0) {
// use a prefix to shorten the url
const prefix = selectedFiles[0].substring(0, selectedFiles[0].lastIndexOf("/"))
const files = selectedFiles.map(path => path.substring(path.lastIndexOf("/") + 1)).join(',')
downloadUrl = `raw/${uploadId}?files=${encodeURIComponent(files)}&prefix=${prefix}&strip=true`
}
return ( return (
<div className={classes.root}> <div className={classes.root}>
<FormGroup row> <FormGroup row>
...@@ -225,7 +239,7 @@ class RawFiles extends React.Component { ...@@ -225,7 +239,7 @@ class RawFiles extends React.Component {
<Download component={IconButton} disabled={selectedFiles.length === 0} <Download component={IconButton} disabled={selectedFiles.length === 0}
color="secondary" color="secondary"
tooltip="download selected files" tooltip="download selected files"
url={(selectedFiles.length === 1) ? `raw/${uploadId}/${selectedFiles[0]}` : `raw/${uploadId}?files=${encodeURIComponent(selectedFiles.join(','))}&strip=true`} url={downloadUrl}
fileName={selectedFiles.length === 1 ? this.label(selectedFiles[0]) : `${calcId}.zip`} fileName={selectedFiles.length === 1 ? this.label(selectedFiles[0]) : `${calcId}.zip`}
> >