Commit 14329873 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'v0.7.10' into v0.8.0<

parents 2b83f245 22b09bb2
......@@ -90,7 +90,6 @@ tests:
NOMAD_RABBITMQ_HOST: rabbitmq
NOMAD_ELASTIC_HOST: elastic
NOMAD_MONGO_HOST: mongo
NOMAD_KEYCLOAK_CLIENT_SECRET: ${CI_KEYCLOAK_TEST_CLIENT_SECRET}
NOMAD_KEYCLOAK_PASSWORD: ${CI_KEYCLOAK_ADMIN_PASSWORD}
NOMAD_SPRINGER_DB_PATH: /nomad/fairdi/db/data/springer.db
script:
......
......@@ -29,6 +29,14 @@ contributing, and API reference.
Omitted versions are plain bugfix releases with only minor changes and fixes.
### v0.7.9
- Everything to run a simple NOMAD OASIS based on the central user-management
- minor bugfixes
### v0.7.7
- Shows dataset contents with embargo data, but hides the entry details (raw-files, archive)
- minor bugfixes
### v0.7.5
- AFLOWLIB prototypes (archive)
- primitive label search
......
Subproject commit b50054a10b28efddb82554051d797b44f2b1067e
Subproject commit d918460c31728058834432b736062d44e1e1c074
Subproject commit cd354f066cb8b85904a2725bb93abf7c443b3fdf
Subproject commit d30ef0bd9275206380866c89946a0c129e7d8df9
Subproject commit 92005ec9ff4b8e13bd86373d14bd5fafe2b52cd1
Subproject commit 75a5cd92dbd6299067e0fca0b9949f8b4410ec91
Subproject commit 5f97f32086c281ebda5ab6084ae2c7eba16b516f
Subproject commit e113cbf21f23054394ad6099ad4836cbd9e21790
Subproject commit b932711d741c2457a80bf2447c180ce49c23e6c9
Subproject commit bd5b5c6f947ec9b7172ef7970a92825c737e1e60
Subproject commit fe15759f080e8176d88af91447949243608b0d7e
Subproject commit d60013e1597493972237210a36549bfcf0a2706f
Subproject commit 3811ced85fb7d68ca579d5ca8d93e800f48c53a5
Subproject commit f2b7f39ca62438d25a21cdbaf267269fbc4f62ac
Subproject commit d9c9b3c14ecab80e58adab70917267e5e7fbe3f2
Subproject commit 5f07d80f9d1838b3f6b95e39266221002061e0d1
Operating nomad
Operating NOMAD
===============
.. mdinclude:: ../ops/README.md
.. mdinclude:: ../ops/docker-compose/nomad/README.md
.. mdinclude:: ../ops/helm/nomad/README.md
.. mdinclude:: ../ops/containers/README.md
.. mdinclude:: ../ops/docker-compose/nomad-oasis/README.md
......@@ -11,18 +11,19 @@ The nomad infrastructure consists of a series of nomad and 3rd party services:
- rabbitmq: a task queue used to distribute work in a cluster
All 3rd party services should be run via *docker-compose* (see blow). The
nomad python services can also be run via *docker-compose* or manually started with python.
The gui can be run manually with a development server via yarn, or with
*docker-compose*
nomad python services can be run with python to develop them.
The gui can be run with a development server via yarn.
Below you will find information on how to install all python dependencies and code
manually. How to use *docker*/*docker-compose*. How run services with *docker-compose*
or manually.
manually. How to use *docker*/*docker-compose*. How run 3rd-party services with *docker-compose*.
Keep in mind the *docker-compose* configures all services in a way that mirror
the configuration of the python code in `nomad/config.py` and the gui config in
`gui/.env.development`.
To learn about how to run everything in docker, e.g. to operate a NOMAD OASIS in
production, go (here)(/app/docs/ops.html).
## Install python code and dependencies
### Cloning and development tools
......@@ -158,35 +159,12 @@ having to copy the git itself to the docker build context.
The images are build via *docker-compose* and don't have to be created manually.
### Build with docker-compose
We have multiple *docker-compose* files that must be used together.
- `docker-compose.yml` contains the base definitions for all services
- `docker-compose.override.yml` configures services for development (notably builds images for nomad services)
- `docker-compose.dev-elk.yml` will also provide the ELK service
- `docker-compose.prod.yml` configures services for production (notable uses a pre-build image for nomad services that was build during CI/CD)
It is sufficient to use the implicit `docker-compose.yml` only (like in the command below).
The `override` will be used automatically.
Now we can build the *docker-compose* that contains all external services (rabbitmq,
mongo, elastic, elk) and nomad services (worker, app, gui).
```
docker-compose build
```
Docker-compose tries to cache individual building steps. Sometimes this causes
troubles and not everything necessary is build when you changed something. In
this cases use:
```
docker-compose build --no-cache
```
### Run everything with docker-compose
### Run necessary 3-rd party services with docker-compose
You can run all containers with:
```
docker-compose up
cd ops/docker-compose/nomad
docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d mongo elastic rabbitmq
```
To shut down everything, just `ctrl-c` the running output. If you started everything
......@@ -195,25 +173,6 @@ in *deamon* mode (`-d`) use:
docker-compose down
```
### Run containers selectively
The following services/containers are managed via our docker-compose:
- rabbitmq, mongo, elastic, (elk, only for production)
- worker, app
- gui
- proxy
The *proxy* container runs *nginx* based reverse proxies that put all services under
a single port and different paths.
You can also run services selectively, e.g.
```
docker-compose up -d rabbitmq, mongo, elastic
docker-compose up worker
docker-compose up app gui proxy
```
## Accessing 3'rd party services
Usually these services only used by the nomad containers, but sometimes you also
need to check something or do some manual steps.
......@@ -234,12 +193,7 @@ The index prefix for logs is `logstash-`. The ELK is only available with the
You can access mongodb and elastic search via your preferred tools. Just make sure
to use the right ports (see above).
## Run nomad services manually
You can run the worker, app, and gui as part of the docker infrastructure, like
seen above. But, of course there are always reasons to run them manually during
development, like running them in a debugger, profiler, etc.
## Run nomad services
### API and worker
......@@ -253,11 +207,6 @@ To run it directly with celery, do (from the root)
celery -A nomad.processing worker -l info
```
Run the app via docker, or (from the root):
```
nomad admin run app
```
You can also run worker and app together:
```
nomad admin run appworker
......
{
"name": "nomad-fair-gui",
"version": "0.7.6",
"version": "0.7.10",
"commit": "nomad-gui-commit-placeholder",
"private": true,
"dependencies": {
......
......@@ -3,12 +3,13 @@ import PropTypes from 'prop-types'
import { withStyles } from '@material-ui/core/styles'
import { compose } from 'recompose'
import { withErrors } from './errors'
import { withApi, DoesNotExist } from './api'
import { withApi } from './api'
import Search from './search/Search'
import SearchContext from './search/SearchContext'
import { Typography } from '@material-ui/core'
import { DatasetActions, DOI } from './search/DatasetList'
import { withRouter } from 'react-router'
import { withDomain } from './domains'
export const help = `
This page allows you to **inspect** and **download** NOMAD datasets. It alsow allows you
......@@ -21,7 +22,8 @@ class DatasetPage extends React.Component {
api: PropTypes.object.isRequired,
datasetId: PropTypes.string.isRequired,
raiseError: PropTypes.func.isRequired,
history: PropTypes.object.isRequired
history: PropTypes.object.isRequired,
domain: PropTypes.object.isRequired
}
static styles = theme => ({
......@@ -44,6 +46,7 @@ class DatasetPage extends React.Component {
state = {
dataset: {},
empty: false,
update: 0
}
......@@ -58,14 +61,13 @@ class DatasetPage extends React.Component {
const entry = data.results[0]
const dataset = entry && entry.datasets.find(ds => ds.id + '' === datasetId)
if (!dataset) {
this.setState({dataset: {}})
raiseError(new DoesNotExist('Dataset does not exist any more or is not visible to you.'))
this.setState({dataset: {}, empty: true})
}
this.setState({dataset: {
...dataset, example: entry
...dataset, example: entry, empty: false
}})
}).catch(error => {
this.setState({dataset: {}})
this.setState({dataset: {}, empty: false})
raiseError(error)
})
}
......@@ -76,7 +78,7 @@ class DatasetPage extends React.Component {
componentDidUpdate(prevProps) {
if (prevProps.api !== this.props.api || prevProps.datasetId !== this.props.datasetId) {
this.setState({dataset: {}}, () => this.update())
this.setState({dataset: {}, empty: false}, () => this.update())
}
}
......@@ -89,14 +91,14 @@ class DatasetPage extends React.Component {
}
render() {
const { classes, datasetId } = this.props
const { dataset, update } = this.state
const { classes, datasetId, domain } = this.props
const { dataset, update, empty } = this.state
return (
<div>
<div className={classes.header}>
<div className={classes.description}>
<Typography variant="h4">{dataset.name || 'loading ...'}</Typography>
<Typography variant="h4">{dataset.name || (empty && 'Empty or non existing dataset') ||'loading ...'}</Typography>
<Typography>
dataset{dataset.doi ? <span>, with DOI <DOI doi={dataset.doi} /></span> : ''}
</Typography>
......@@ -111,13 +113,20 @@ class DatasetPage extends React.Component {
</div>
<SearchContext
query={{dataset_id: datasetId}} ownerTypes={['all', 'public']} update={update}
initialQuery={{owner: 'all'}}
query={{dataset_id: datasetId}}
ownerTypes={['all', 'public']} update={update}
>
<Search resultTab="entries" tabs={['entries', 'groups', 'datasets']} />
<Search
resultTab="entries" tabs={['entries', 'groups', 'datasets']}
entryListProps={{
selectedColumns: [...domain.defaultSearchResultColumns, 'published', 'authors']
}}
/>
</SearchContext>
</div>
)
}
}
export default compose(withRouter, withApi(false), withErrors, withStyles(DatasetPage.styles))(DatasetPage)
export default compose(withRouter, withDomain, withApi(false), withErrors, withStyles(DatasetPage.styles))(DatasetPage)
......@@ -647,7 +647,7 @@ class InviteUserDialogUnstyled extends React.Component {
If you want to add a user as co-author or share your data with someone that
is not already a NOMAD user, you can invite this person here. We need just a few
details about this person. After your invite, the new user will receive an
Email that allows him to set a password and further details. Anyhow, you will
Email that allows her to set a password and further details. Anyhow, you will
be able to add the user as co-author or someone to share with immediately after the
invite.
</DialogContentText>
......@@ -730,7 +730,8 @@ class EditUserMetadataDialogUnstyled extends React.Component {
user: PropTypes.object,
onEditComplete: PropTypes.func,
disabled: PropTypes.bool,
title: PropTypes.string
title: PropTypes.string,
info: PropTypes.object
}
static styles = theme => ({
......@@ -1055,7 +1056,7 @@ class EditUserMetadataDialogUnstyled extends React.Component {
}
renderDialogActions(submitting, submitEnabled) {
const {classes} = this.props
const {classes, info} = this.props
if (submitting) {
return <DialogActions>
......@@ -1070,7 +1071,7 @@ class EditUserMetadataDialogUnstyled extends React.Component {
</DialogActions>
} else {
return <DialogActions>
<InviteUserDialog />
{info && !info.oasis && <InviteUserDialog />}
<span style={{flexGrow: 1}} />
<Button onClick={this.handleClose} disabled={submitting}>
Cancel
......
......@@ -107,12 +107,15 @@ class FAQ extends React.Component {
publishing anything.
Second, you can publish your uploads with an *embargo* period. This can last up to
3 years. You can lift the embargo at anytime. This allows you to privately create
datasets and DOIs, share data with selected people, before your work is published, e.g.
in a paper.
Non *published* and *embargoed* data is only visible to you (the uploader) and users
that you explicitly share your entries with.
3 years. You can lift the embargo at anytime. Embargoed data is
visible to and findable by others. This makes only some few metadata (e.g.
chemical formula, system type, spacegroup, etc.) public, but the raw-file
and archive contents remain hidden (except to you, and users you explicitly
share the data with).
You can already create datasets and assign DOIs for data with embargo, e.g.
to put it into your unpublished paper.
The embargo will last up to 36 month. Afterwards, your data will be made publicly
available. You can also lift the embargo on entries at any time.
### How do I cite uploaded data in a paper?
......
......@@ -308,12 +308,13 @@ class Api {
.finally(this.onFinishLoading)
}
async getRawFile(uploadId, path, kwargs) {
async getRawFile(uploadId, calcId, path, kwargs) {
this.onStartLoading()
const length = (kwargs && kwargs.length) || 4096
return this.swagger()
.then(client => client.apis.raw.get({
.then(client => client.apis.raw.get_file_from_calc({
upload_id: uploadId,
calc_id: calcId,
path: path,
decompress: true,
...(kwargs || {}),
......@@ -697,6 +698,8 @@ export const ApiProvider = compose(withKeycloak, withErrors)(ApiProviderComponen
const LoginRequired = withStyles(LoginRequiredUnstyled.styles)(LoginRequiredUnstyled)
const __reauthorize_trigger_changes = ['api', 'calcId', 'uploadId', 'calc_id', 'upload_id']
class WithApiComponent extends React.Component {
static propTypes = {
raiseError: PropTypes.func.isRequired,
......@@ -718,7 +721,7 @@ class WithApiComponent extends React.Component {
}
componentDidUpdate(prevProps) {
if (prevProps.api !== this.props.api) {
if (__reauthorize_trigger_changes.find(key => this.props[key] !== prevProps[key])) {
this.setState({notAuthorized: false})
}
}
......@@ -746,12 +749,12 @@ class WithApiComponent extends React.Component {
if (notAuthorized) {
if (keycloak.authenticated) {
return (
<div>
<div style={{marginTop: 24}}>
<Typography variant="h6">Not Authorized</Typography>
<Typography>
You are not authorized to access this information. If someone send
you this link, ask him to make his data publicly available or share
it with you.
you a link to this data, ask the authors to make the data publicly available
or share it with you.
</Typography>
</div>
)
......
......@@ -130,9 +130,9 @@ class RawFiles extends React.Component {
}
handleFileClicked(file) {
const {api, uploadId, raiseError} = this.props
const {api, uploadId, calcId, raiseError} = this.props
this.setState({shownFile: file, fileContents: null})
api.getRawFile(uploadId, file, {length: 16 * 1024})
api.getRawFile(uploadId, calcId, file.split('/').reverse()[0], {length: 16 * 1024})
.then(contents => this.setState({fileContents: contents}))
.catch(raiseError)
}
......@@ -154,7 +154,7 @@ class RawFiles extends React.Component {
})
if (fileContents.contents.length < (page + 1) * 16 * 1024) {
api.getRawFile(uploadId, shownFile, {offset: page * 16 * 1024, length: 16 * 1024})
api.getRawFile(uploadId, calcId, shownFile.split('/').reverse()[0], {offset: page * 16 * 1024, length: 16 * 1024})
.then(contents => {
const {fileContents} = this.state
// The back-button navigation might cause a scroll event, might cause to loadmore,
......@@ -200,6 +200,20 @@ class RawFiles extends React.Component {
</Typography>
}
let downloadUrl
if (selectedFiles.length === 1) {
// download the individual file
downloadUrl = `raw/${uploadId}/${selectedFiles[0]}`
} else if (selectedFiles.length === availableFiles.length) {
// use an endpoint that downloads all files of the calc
downloadUrl = `raw/calc/${uploadId}/${calcId}/*?strip=true`
} else if (selectedFiles.length > 0) {
// use a prefix to shorten the url
const prefix = selectedFiles[0].substring(0, selectedFiles[0].lastIndexOf("/"))
const files = selectedFiles.map(path => path.substring(path.lastIndexOf("/") + 1)).join(',')
downloadUrl = `raw/${uploadId}?files=${encodeURIComponent(files)}&prefix=${prefix}&strip=true`
}
return (
<div className={classes.root}>
<FormGroup row>
......@@ -225,7 +239,7 @@ class RawFiles extends React.Component {
<Download component={IconButton} disabled={selectedFiles.length === 0}
color="secondary"
tooltip="download selected files"
url={(selectedFiles.length === 1) ? `raw/${uploadId}/${selectedFiles[0]}` : `raw/${uploadId}?files=${encodeURIComponent(selectedFiles.join(','))}&strip=true`}
url={downloadUrl}
fileName={selectedFiles.length === 1 ? this.label(selectedFiles[0]) : `${calcId}.zip`}
>
<DownloadIcon />
......
......@@ -57,9 +57,9 @@ class RepoEntryView extends React.Component {
update() {
const {uploadId, calcId} = this.props
this.props.api.repo(uploadId, calcId).then(data => {
this.setState({calcData: data})
this.setState({calcData: data, doesNotExist: false})
}).catch(error => {
this.setState({calcData: null})
this.setState({calcData: null, doesNotExist: false})
if (error.name === 'DoesNotExist') {
this.setState({doesNotExist: true})
} else {
......
......@@ -217,6 +217,10 @@ class DatasetListUnstyled extends React.Component {
label: 'Dataset name',
render: (dataset) => dataset.name
},
created: {
label: 'Created',
render: (dataset) => dataset.created && new Date(dataset.created).toLocaleString()
},
DOI: {
label: 'Dataset DOI',
render: (dataset) => dataset.doi && <DOI doi={dataset.doi} />
......@@ -283,8 +287,7 @@ class DatasetListUnstyled extends React.Component {
id={row => row.id}
total={total}
columns={this.columns}
// selectedColumns={defaultSelectedColumns}
// entryDetails={this.renderEntryDetails.bind(this)}
selectedColumns={['name', 'DOI', 'entries', 'authors']}
entryActions={this.renderEntryActions}
data={results}
rows={per_page}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment