Merge branch 'v0.7.10' into v0.8.0<

.gitlab-ci.yml

@@ -90,7 +90,6 @@ tests:
     NOMAD_RABBITMQ_HOST: rabbitmq
     NOMAD_ELASTIC_HOST: elastic
     NOMAD_MONGO_HOST: mongo
-    NOMAD_KEYCLOAK_CLIENT_SECRET: ${CI_KEYCLOAK_TEST_CLIENT_SECRET}
     NOMAD_KEYCLOAK_PASSWORD: ${CI_KEYCLOAK_ADMIN_PASSWORD}
     NOMAD_SPRINGER_DB_PATH: /nomad/fairdi/db/data/springer.db
   script:

README.md

@@ -29,6 +29,14 @@ contributing, and API reference.
 
 Omitted versions are plain bugfix releases with only minor changes and fixes.
 
+### v0.7.9
+- Everything to run a simple NOMAD OASIS based on the central user-management
+- minor bugfixes
+
+### v0.7.7
+- Shows dataset contents with embargo data, but hides the entry details (raw-files, archive)
+- minor bugfixes
+
 ### v0.7.5
 - AFLOWLIB prototypes (archive)
 - primitive label search

nomad-meta-info @ d918460c

-Subproject commit b50054a10b28efddb82554051d797b44f2b1067e
+Subproject commit d918460c31728058834432b736062d44e1e1c074

band @ d30ef0bd

-Subproject commit cd354f066cb8b85904a2725bb93abf7c443b3fdf
+Subproject commit d30ef0bd9275206380866c89946a0c129e7d8df9

gamess @ 75a5cd92

-Subproject commit 92005ec9ff4b8e13bd86373d14bd5fafe2b52cd1
+Subproject commit 75a5cd92dbd6299067e0fca0b9949f8b4410ec91

gulp @ e113cbf2

-Subproject commit 5f97f32086c281ebda5ab6084ae2c7eba16b516f
+Subproject commit e113cbf21f23054394ad6099ad4836cbd9e21790

onetep @ bd5b5c6f

-Subproject commit b932711d741c2457a80bf2447c180ce49c23e6c9
+Subproject commit bd5b5c6f947ec9b7172ef7970a92825c737e1e60

quantum-espresso @ d60013e1

-Subproject commit fe15759f080e8176d88af91447949243608b0d7e
+Subproject commit d60013e1597493972237210a36549bfcf0a2706f

turbomole @ f2b7f39c

-Subproject commit 3811ced85fb7d68ca579d5ca8d93e800f48c53a5
+Subproject commit f2b7f39ca62438d25a21cdbaf267269fbc4f62ac

vasp @ 5f07d80f

-Subproject commit d9c9b3c14ecab80e58adab70917267e5e7fbe3f2
+Subproject commit 5f07d80f9d1838b3f6b95e39266221002061e0d1

docs/ops.rst

-Operating nomad
+Operating NOMAD
 ===============
 
 .. mdinclude:: ../ops/README.md
 .. mdinclude:: ../ops/docker-compose/nomad/README.md
 .. mdinclude:: ../ops/helm/nomad/README.md
 .. mdinclude:: ../ops/containers/README.md
+
+.. mdinclude:: ../ops/docker-compose/nomad-oasis/README.md

docs/setup.md

@@ -11,18 +11,19 @@ The nomad infrastructure consists of a series of nomad and 3rd party services:
 - rabbitmq: a task queue used to distribute work in a cluster
 
 All 3rd party services should be run via *docker-compose* (see blow). The
-nomad python  services can also be run via *docker-compose* or manually started with python.
-The gui can be run manually with a development server via yarn, or with
-*docker-compose*
+nomad python  services can be run with python to develop them.
+The gui can be run with a development server via yarn.
 
 Below you will find information on how to install all python dependencies and code
-manually. How to use *docker*/*docker-compose*. How run services with *docker-compose*
-or manually.
+manually. How to use *docker*/*docker-compose*. How run 3rd-party services with *docker-compose*.
 
 Keep in mind the *docker-compose* configures all services in a way that mirror
 the configuration of the python code in `nomad/config.py` and the gui config in
 `gui/.env.development`.
 
+To learn about how to run everything in docker, e.g. to operate a NOMAD OASIS in
+production, go (here)(/app/docs/ops.html).
+
 ## Install python code and dependencies
 
 ### Cloning and development tools
@@ -158,35 +159,12 @@ having to copy the git itself to the docker build context.
 
 The images are build via *docker-compose* and don't have to be created manually.
 
-### Build with docker-compose
-
-We have multiple *docker-compose* files that must be used together.
-- `docker-compose.yml` contains the base definitions for all services
-- `docker-compose.override.yml` configures services for development (notably builds images for nomad services)
-- `docker-compose.dev-elk.yml` will also provide the ELK service
-- `docker-compose.prod.yml` configures services for production (notable uses a pre-build image for nomad services that was build during CI/CD)
-
-It is sufficient to use the implicit `docker-compose.yml` only (like in the command below).
-The `override` will be used automatically.
-
-Now we can build the *docker-compose* that contains all external services (rabbitmq,
-mongo, elastic, elk) and nomad services (worker, app, gui).
-```
-docker-compose build
-```
-
-Docker-compose tries to cache individual building steps. Sometimes this causes
-troubles and not everything necessary is build when you changed something. In
-this cases use:
-```
-docker-compose build --no-cache
-```
-
-### Run everything with docker-compose
+### Run necessary 3-rd party services with docker-compose
 
 You can run all containers with:
 ```
-docker-compose up
+cd ops/docker-compose/nomad
+docker-compose -f docker-compose.yml -f docker-compose.override.yml up -d mongo elastic rabbitmq
 ```
 
 To shut down everything, just `ctrl-c` the running output. If you started everything
@@ -195,25 +173,6 @@ in *deamon* mode (`-d`) use:
 docker-compose down
 ```
 
-### Run containers selectively
-The following services/containers are managed via our docker-compose:
-- rabbitmq, mongo, elastic, (elk, only for production)
-- worker, app
-- gui
-- proxy
-
-The *proxy* container runs *nginx* based reverse proxies that put all services under
-a single port and different paths.
-
-You can also run services selectively, e.g.
-```
-docker-compose up -d rabbitmq, mongo, elastic
-docker-compose up worker
-docker-compose up app gui proxy
-```
-
-## Accessing 3'rd party services
-
 Usually these services only used by the nomad containers, but sometimes you also
 need to check something or do some manual steps.
 
@@ -234,12 +193,7 @@ The index prefix for logs is `logstash-`. The ELK is only available with the
 You can access mongodb and elastic search via your preferred tools. Just make sure
 to use the right ports (see above).
 
-
-## Run nomad services manually
-
-You can run the worker, app, and gui as part of the docker infrastructure, like
-seen above. But, of course there are always reasons to run them manually during
-development, like running them in a debugger, profiler, etc.
+## Run nomad services
 
 ### API and worker
 
@@ -253,11 +207,6 @@ To run it directly with celery, do (from the root)
 celery -A nomad.processing worker -l info
 ```
 
-Run the app via docker, or (from the root):
-```
-nomad admin run app
-```
-
 You can also run worker and app together:
 ```
 nomad admin run appworker

gui/package.json

 {
   "name": "nomad-fair-gui",
-  "version": "0.7.6",
+  "version": "0.7.10",
   "commit": "nomad-gui-commit-placeholder",
   "private": true,
   "dependencies": {

gui/src/components/DatasetPage.js

@@ -3,12 +3,13 @@ import PropTypes from 'prop-types'
 import { withStyles } from '@material-ui/core/styles'
 import { compose } from 'recompose'
 import { withErrors } from './errors'
-import { withApi, DoesNotExist } from './api'
+import { withApi } from './api'
 import Search from './search/Search'
 import SearchContext from './search/SearchContext'
 import { Typography } from '@material-ui/core'
 import { DatasetActions, DOI } from './search/DatasetList'
 import { withRouter } from 'react-router'
+import { withDomain } from './domains'
 
 export const help = `
 This page allows you to **inspect** and **download** NOMAD datasets. It alsow allows you
@@ -21,7 +22,8 @@ class DatasetPage extends React.Component {
     api: PropTypes.object.isRequired,
     datasetId: PropTypes.string.isRequired,
     raiseError: PropTypes.func.isRequired,
-    history: PropTypes.object.isRequired
+    history: PropTypes.object.isRequired,
+    domain: PropTypes.object.isRequired
   }
 
   static styles = theme => ({
@@ -44,6 +46,7 @@ class DatasetPage extends React.Component {
 
   state = {
     dataset: {},
+    empty: false,
     update: 0
   }
 
@@ -58,14 +61,13 @@ class DatasetPage extends React.Component {
       const entry = data.results[0]
       const dataset = entry && entry.datasets.find(ds => ds.id + '' === datasetId)
       if (!dataset) {
-        this.setState({dataset: {}})
-        raiseError(new DoesNotExist('Dataset does not exist any more or is not visible to you.'))
+        this.setState({dataset: {}, empty: true})
       }
       this.setState({dataset: {
-        ...dataset, example: entry
+        ...dataset, example: entry, empty: false
       }})
     }).catch(error => {
-      this.setState({dataset: {}})
+      this.setState({dataset: {}, empty: false})
       raiseError(error)
     })
   }
@@ -76,7 +78,7 @@ class DatasetPage extends React.Component {
 
   componentDidUpdate(prevProps) {
     if (prevProps.api !== this.props.api || prevProps.datasetId !== this.props.datasetId) {
-      this.setState({dataset: {}}, () => this.update())
+      this.setState({dataset: {}, empty: false}, () => this.update())
     }
   }
 
@@ -89,14 +91,14 @@ class DatasetPage extends React.Component {
   }
 
   render() {
-    const { classes, datasetId } = this.props
-    const { dataset, update } = this.state
+    const { classes, datasetId, domain } = this.props
+    const { dataset, update, empty } = this.state
 
     return (
       <div>
         <div className={classes.header}>
           <div className={classes.description}>
-            <Typography variant="h4">{dataset.name || 'loading ...'}</Typography>
+            <Typography variant="h4">{dataset.name || (empty && 'Empty or non existing dataset') ||'loading ...'}</Typography>
             <Typography>
               dataset{dataset.doi ? <span>, with DOI <DOI doi={dataset.doi} /></span> : ''}
             </Typography>
@@ -111,13 +113,20 @@ class DatasetPage extends React.Component {
         </div>
 
         <SearchContext
-          query={{dataset_id: datasetId}} ownerTypes={['all', 'public']} update={update}
+          initialQuery={{owner: 'all'}}
+          query={{dataset_id: datasetId}}
+          ownerTypes={['all', 'public']} update={update}
         >
-          <Search resultTab="entries" tabs={['entries', 'groups', 'datasets']} />
+          <Search
+            resultTab="entries" tabs={['entries', 'groups', 'datasets']}
+            entryListProps={{
+              selectedColumns: [...domain.defaultSearchResultColumns, 'published', 'authors']
+            }}
+          />
         </SearchContext>
       </div>
     )
   }
 }
 
-export default compose(withRouter, withApi(false), withErrors, withStyles(DatasetPage.styles))(DatasetPage)
+export default compose(withRouter, withDomain, withApi(false), withErrors, withStyles(DatasetPage.styles))(DatasetPage)

gui/src/components/EditUserMetadataDialog.js

@@ -647,7 +647,7 @@ class InviteUserDialogUnstyled extends React.Component {
             If you want to add a user as co-author or share your data with someone that
             is not already a NOMAD user, you can invite this person here. We need just a few
             details about this person. After your invite, the new user will receive an
-            Email that allows him to set a password and further details. Anyhow, you will
+            Email that allows her to set a password and further details. Anyhow, you will
             be able to add the user as co-author or someone to share with immediately after the
             invite.
           </DialogContentText>
@@ -730,7 +730,8 @@ class EditUserMetadataDialogUnstyled extends React.Component {
     user: PropTypes.object,
     onEditComplete: PropTypes.func,
     disabled: PropTypes.bool,
-    title: PropTypes.string
+    title: PropTypes.string,
+    info: PropTypes.object
   }
 
   static styles = theme => ({
@@ -1055,7 +1056,7 @@ class EditUserMetadataDialogUnstyled extends React.Component {
   }
 
   renderDialogActions(submitting, submitEnabled) {
-    const {classes} = this.props
+    const {classes, info} = this.props
 
     if (submitting) {
       return <DialogActions>
@@ -1070,7 +1071,7 @@ class EditUserMetadataDialogUnstyled extends React.Component {
       </DialogActions>
     } else {
       return <DialogActions>
-        <InviteUserDialog />
+        {info && !info.oasis && <InviteUserDialog />}
         <span style={{flexGrow: 1}} />
         <Button onClick={this.handleClose} disabled={submitting}>
           Cancel

gui/src/components/FAQ.js

@@ -107,12 +107,15 @@ class FAQ extends React.Component {
           publishing anything.
 
           Second, you can publish your uploads with an *embargo* period. This can last up to
-          3 years. You can lift the embargo at anytime. This allows you to privately create
-          datasets and DOIs, share data with selected people, before your work is published, e.g.
-          in a paper.
-
-          Non *published* and *embargoed* data is only visible to you (the uploader) and users
-          that you explicitly share your entries with.
+          3 years. You can lift the embargo at anytime. Embargoed data is
+          visible to and findable by others. This makes only some few metadata (e.g.
+          chemical formula, system type, spacegroup, etc.) public, but the raw-file
+          and archive contents remain hidden (except to you, and users you explicitly
+          share the data with).
+          You can already create datasets and assign DOIs for data with embargo, e.g.
+          to put it into your unpublished paper.
+          The embargo will last up to 36 month. Afterwards, your data will be made publicly
+          available. You can also lift the embargo on entries at any time.
 
           ### How do I cite uploaded data in a paper?
 

gui/src/components/api.js

@@ -308,12 +308,13 @@ class Api {
       .finally(this.onFinishLoading)
   }
 
-  async getRawFile(uploadId, path, kwargs) {
+  async getRawFile(uploadId, calcId, path, kwargs) {
     this.onStartLoading()
     const length = (kwargs && kwargs.length) || 4096
     return this.swagger()
-      .then(client => client.apis.raw.get({
+      .then(client => client.apis.raw.get_file_from_calc({
         upload_id: uploadId,
+        calc_id: calcId,
         path: path,
         decompress: true,
         ...(kwargs || {}),
@@ -697,6 +698,8 @@ export const ApiProvider = compose(withKeycloak, withErrors)(ApiProviderComponen
 
 const LoginRequired = withStyles(LoginRequiredUnstyled.styles)(LoginRequiredUnstyled)
 
+const __reauthorize_trigger_changes = ['api', 'calcId', 'uploadId', 'calc_id', 'upload_id']
+
 class WithApiComponent extends React.Component {
   static propTypes = {
     raiseError: PropTypes.func.isRequired,
@@ -718,7 +721,7 @@ class WithApiComponent extends React.Component {
   }
 
   componentDidUpdate(prevProps) {
-    if (prevProps.api !== this.props.api) {
+    if (__reauthorize_trigger_changes.find(key => this.props[key] !== prevProps[key])) {
       this.setState({notAuthorized: false})
     }
   }
@@ -746,12 +749,12 @@ class WithApiComponent extends React.Component {
     if (notAuthorized) {
       if (keycloak.authenticated) {
         return (
-          <div>
+          <div style={{marginTop: 24}}>
             <Typography variant="h6">Not Authorized</Typography>
             <Typography>
               You are not authorized to access this information. If someone send
-              you this link, ask him to make his data publicly available or share
-              it with you.
+              you a link to this data, ask the authors to make the data publicly available
+              or share it with you.
             </Typography>
           </div>
         )

gui/src/components/entry/RawFiles.js

@@ -130,9 +130,9 @@ class RawFiles extends React.Component {
   }
 
   handleFileClicked(file) {
-    const {api, uploadId, raiseError} = this.props
+    const {api, uploadId, calcId, raiseError} = this.props
     this.setState({shownFile: file, fileContents: null})
-    api.getRawFile(uploadId, file, {length: 16 * 1024})
+    api.getRawFile(uploadId, calcId, file.split('/').reverse()[0], {length: 16 * 1024})
       .then(contents => this.setState({fileContents: contents}))
       .catch(raiseError)
   }
@@ -154,7 +154,7 @@ class RawFiles extends React.Component {
     })
 
     if (fileContents.contents.length < (page + 1) * 16 * 1024) {
-      api.getRawFile(uploadId, shownFile, {offset: page * 16 * 1024, length: 16 * 1024})
+      api.getRawFile(uploadId, calcId, shownFile.split('/').reverse()[0], {offset: page * 16 * 1024, length: 16 * 1024})
         .then(contents => {
           const {fileContents} = this.state
           // The back-button navigation might cause a scroll event, might cause to loadmore,
@@ -200,6 +200,20 @@ class RawFiles extends React.Component {
       </Typography>
     }
 
+    let downloadUrl
+    if (selectedFiles.length === 1) {
+      // download the individual file
+      downloadUrl = `raw/${uploadId}/${selectedFiles[0]}`
+    } else if (selectedFiles.length === availableFiles.length) {
+      // use an endpoint that downloads all files of the calc
+      downloadUrl = `raw/calc/${uploadId}/${calcId}/*?strip=true`
+    } else if (selectedFiles.length > 0) {
+      // use a prefix to shorten the url
+      const prefix = selectedFiles[0].substring(0, selectedFiles[0].lastIndexOf("/"))
+      const files = selectedFiles.map(path => path.substring(path.lastIndexOf("/") + 1)).join(',')
+      downloadUrl = `raw/${uploadId}?files=${encodeURIComponent(files)}&prefix=${prefix}&strip=true`
+    }
+
     return (
       <div className={classes.root}>
         <FormGroup row>
@@ -225,7 +239,7 @@ class RawFiles extends React.Component {
           <Download component={IconButton} disabled={selectedFiles.length === 0}
             color="secondary"
             tooltip="download selected files"
-            url={(selectedFiles.length === 1) ? `raw/${uploadId}/${selectedFiles[0]}` : `raw/${uploadId}?files=${encodeURIComponent(selectedFiles.join(','))}&strip=true`}
+            url={downloadUrl}
             fileName={selectedFiles.length === 1 ? this.label(selectedFiles[0]) : `${calcId}.zip`}
           >
             <DownloadIcon />

gui/src/components/entry/RepoEntryView.js

@@ -57,9 +57,9 @@ class RepoEntryView extends React.Component {
   update() {
     const {uploadId, calcId} = this.props
     this.props.api.repo(uploadId, calcId).then(data => {
-      this.setState({calcData: data})
+      this.setState({calcData: data, doesNotExist: false})
     }).catch(error => {
-      this.setState({calcData: null})
+      this.setState({calcData: null, doesNotExist: false})
       if (error.name === 'DoesNotExist') {
         this.setState({doesNotExist: true})
       } else {

gui/src/components/search/DatasetList.js

@@ -217,6 +217,10 @@ class DatasetListUnstyled extends React.Component {
       label: 'Dataset name',
       render: (dataset) => dataset.name
     },
+    created: {
+      label: 'Created',
+      render: (dataset) => dataset.created && new Date(dataset.created).toLocaleString()
+    },
     DOI: {
       label: 'Dataset DOI',
       render: (dataset) => dataset.doi && <DOI doi={dataset.doi} />
@@ -283,8 +287,7 @@ class DatasetListUnstyled extends React.Component {
       id={row => row.id}
       total={total}
       columns={this.columns}
-      // selectedColumns={defaultSelectedColumns}
-      // entryDetails={this.renderEntryDetails.bind(this)}
+      selectedColumns={['name', 'DOI', 'entries', 'authors']}
       entryActions={this.renderEntryActions}
       data={results}
       rows={per_page}