Commit 43ffd5e0 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'v0.5.0-markus' into 'master'

Release of v0.5.0

See merge request !49
parents 9fd7128b 2822c649
Pipeline #53515 passed with stage
in 16 seconds
......@@ -133,22 +133,23 @@ release_version:
only:
- tags
deploy_coe_staging:
stage: deploy
image: dtzar/helm-kubectl
before_script:
- mkdir -p /etc/deploy
# kube_config is a CI/CD variable set in GitLab GUI
- echo $CI_KUBE_CONFIG | base64 -d > /etc/deploy/config
- helm init --upgrade
- helm repo add stable https://kubernetes-charts.storage.googleapis.com/
- helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
- helm repo update
script:
- cd ops/helm/nomad
- export KUBECONFIG=/etc/deploy/config
- helm dep build
- helm upgrade nomad-coe-staging . --recreate-pods;
except:
- /^dev-.*$/
when: manual
## TODO This has to be fixed, we need one for testing, staging, production
# deploy_coe_staging:
# stage: deploy
# image: dtzar/helm-kubectl
# before_script:
# - mkdir -p /etc/deploy
# # kube_config is a CI/CD variable set in GitLab GUI
# - echo $CI_KUBE_CONFIG | base64 -d > /etc/deploy/config
# - helm init --upgrade
# - helm repo add stable https://kubernetes-charts.storage.googleapis.com/
# - helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
# - helm repo update
# script:
# - cd ops/helm/nomad
# - export KUBECONFIG=/etc/deploy/config
# - helm dep build
# - helm upgrade nomad-coe-staging . --recreate-pods;
# except:
# - /^dev-.*$/
# when: manual
[![pipeline status](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/badges/master/pipeline.svg)](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/commits/master)
[![coverage report](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/badges/master/coverage.svg)](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/commits/master)
This project tries and test approaches that might lead to an improved architecture for
nomad@FAIR.
This project implements the new *nomad@FAIRDI* infrastructure. It is currently used
to enable users to upload data, process the data, maintain a version of the NOMAD
archive and meta-info, provide search, inspection, and download to all NOMAD raw and
archive data. As a long term strategy, this project will integrate, refactor, and re-write
more and more of the existing NOMAD CoE components.
The overall goal of *nomad@FAIRDI* is to provide common interfaces to the main services of NOMAD:
*Repository*, *Archive*, and *Encyclopedia*. These interfaces comprise a graphical web-based
UI that allows users to upload data, supervise data processing, inspect and download
metadata, raw-files, and archive data, provide visual tools to explore the data, and
to learn more about advanced use modes, like API and Analytics Toolkit. The second interface
is a unified REST API with various endpoints that represent the core NOMAD services.
This will allow users the automated use of NOMAD for managing their data, and using
data on NOMAD for analytics. A specific way of using the API is through the NOMAD
Analytics Toolkit, which is revamped as a
[separate project](https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-jupyterhub).
Furthermore, this projects aims at establishing NOMAD as a distributed platform for
material science data sharing and management. This includes the on-site deployment of
NOMAD as a standalone service (*oasis*), the federated use of NOMAD through a
serious of full and partial *mirrors*, the integration of 3rd party material science
databases (i.e. [Aflow](http://www.aflow.org/), [OQMD](http://oqmd.org/),
[Materials Project](https://materialsproject.org/)), and support for open APIs and
standards like the [Optimade](http://www.optimade.org/) API.
## Getting started
......@@ -21,7 +43,7 @@ First, clone this repo and init its submodules:
```
git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-FAIR.git
cd nomad-FAIR
git submodules init --depth 1
git submodule init --depth 1
```
Second, create and source your own virtual python environment:
......@@ -68,6 +90,7 @@ The first production version of nomad@fairdi as the upload API and gui for NOMAD
- reprocessing of published results (e.g. after parser/normalizer improvements)
- mirror functionality
- refactored command line interface (CLI)
- potential GUI user tracking capabilities
- many minor bugfixes
### v0.4.7
......
......@@ -14,9 +14,9 @@ The NOMAD *metainfo* defines all quantities used to represent archive data in
NOMAD. You could say it is the archive *schema*. You can browse this schema and
all its definitions here.
The NOMAD metainfo knows three different *kinds* of definitions:
The NOMAD metainfo contains three different *kinds* of definitions:
- **sections**: A section are nested groups of quantities that allow a hierarchical data structure
- **sections**: A section is a nested groups of quantities that allow a hierarchical data structure
- **values**: Actual quantities that contain data
- **references**: References that allow to connect related sections.
......@@ -25,7 +25,7 @@ are organized in packages. There is a *common* package with definitions that are
used by all codes and there are packages for each code with code specific definitions.
You can select the package to browse below.
Depending on the selected package, there are quiet a large number of definitions.
Depending on the selected package, there are quite a large number of definitions.
You can use the *definition* field to search based on definition names.
All definitions are represented as *cards* below. Click on the various card items
......
......@@ -27,7 +27,7 @@ more control about how NOMAD will present your data.
#### Prepare and upload files
To upload your own data, please put all relevant files of all the calculations
To upload your own data, please put all the relevant files of all the calculations
you want to upload into a single \`*.zip\` or \`*.tar.gz\` archive.
We encourage you to add all code input and
output files, as well as any other auxiliary files that you might have created.
......@@ -60,9 +60,9 @@ it with selected users, create a DOI for your data, and later publish the data.
The *embargo* might last up to 36 month before it becomes public automatically.
During an *embargo* some meta-data will be available.
When you published your upload, it will take a night before it will appear in the
When you publish your upload, it will take a night before it will appear in the
[NOMAD Repository](https://repository.nomad-coe.eu/NomadRepository-1.1/).
We are working on improving this process.
We are working on expediting this process.
#### Processing errors
......@@ -72,9 +72,9 @@ upload, bad file formats, etc. The latter (for more common) case means that not
code input/output files could not be parsed by our parsers for various reasons.
The processing logs of the failed entries might provide some insight.
We do not allow to publish uploads that failed processing completely. Frankly, in most
We do not allow the publishing of uploads that fail processing completely. Frankly, in most
cases there won't be any data to publish anyways. In the case of failed processing of
some entires, the data can still be published. You will be able to share it and create
some entries, the data can still be published. You will be able to share it and create
DOIs for it, etc. The only shortcomings will be missing metadata (labeled *not processed*
or *unavailable*) and missing archive data. We continuously improve our parsers and
the missing information might be made available in the future.
......
......@@ -9,7 +9,10 @@ window.nomadEnv = window.nomadEnv || {}
export const apiBase = window.nomadEnv.apiBase
export const appBase = process.env.PUBLIC_URL
export const kibanaBase = window.nomadEnv.kibanaBase
export const matomoUrl = window.nomadEnv.matomoUrl
export const matomoSiteId = window.nomadEnv.matomoSiteId
export const debug = window.nomadEnv.debug || false
export const sendTrackingData = window.nomadEnv.sendTrackingData
export const repoPrimaryColor = repo
......
......@@ -8,17 +8,15 @@ import registerServiceWorker from './registerServiceWorker'
import { Router } from 'react-router-dom'
import history from './history'
import PiwikReactRouter from 'piwik-react-router'
import { debug } from './config'
import { sendTrackingData, matomoUrl, matomoSiteId } from './config'
const matomo = PiwikReactRouter({
url: 'https://labdev-nomad.esc.rzg.mpg.de/fairdi/matomo/',
siteId: 1
url: matomoUrl,
siteId: matomoSiteId
})
console.log(debug)
ReactDOM.render(
<Router history={debug ? history : matomo.connectToHistory(history)}>
<Router history={sendTrackingData ? matomo.connectToHistory(history) : history}>
<App />
</Router>, document.getElementById('root'))
registerServiceWorker()
......@@ -81,11 +81,13 @@ class CalcProcReproduction:
if not os.path.exists(local_path) or override:
# download raw if not already downloaded or if override is set
# download with request, since bravado does not support streaming
# TODO currently only downloads mainfile
self.logger.info('Downloading calc.', mainfile=self.mainfile)
try:
token = client.auth.get_user().response().result.token
req = requests.get('%s/raw/%s/%s' % (config.client.url, self.upload_id, os.path.dirname(self.mainfile)) + '/*', stream=True, headers={'X-Token': token})
dir_name = os.path.dirname(self.mainfile)
req = requests.get(
'%s/raw/%s/%s' % (config.client.url, self.upload_id, dir_name) + '/*',
stream=True, headers={'X-Token': token})
with open(local_path, 'wb') as f:
for chunk in req.iter_content(chunk_size=io.DEFAULT_BUFFER_SIZE):
f.write(chunk)
......@@ -154,7 +156,12 @@ def local(calc_id, show_backend=False, show_metadata=False, **kwargs):
utils.configure_logging()
utils.get_logger(__name__).info('Using %s' % config.client.url)
with CalcProcReproduction(calc_id, **kwargs) as local:
if local.upload_id != 'unknown':
print(
'Data being saved to .volumes/fs/tmp/repro_'
'%s if not already there' % local.upload_id)
backend = local.parse()
# Run suite of nomalizers on parsed backend.
local.normalize_all(parser_backend=backend)
if show_backend:
backend.write_json(sys.stdout, pretty=True)
......
proxy:
nodePort: 30001
external:
host: "labdev-nomad.esc.rzg.mpg.de"
path: "/fairdi/nomad/coe"
api:
disableReset: false
worker:
replicas: 1
routing: "queue"
processes: 10
dbname: fairdi_nomad_coe
postgres:
publish_enabled: true
mode: 'coe'
dbname: 'nomad_prod'
host: 'db-repository-nomad.esc'
user: 'nomadrep'
password_secret: 'nomad-production-repository-password'
uploadurl: 'https://labdev-nomad.rzg.mpg.de/fairdi/nomad/coe/upload'
volumes:
prefixSize: 2
public: /nomad/fairdi/coe/fs/public
staging: /nomad/fairdi/coe/fs/staging
tmp: /nomad/fairdi/coe/fs/tmp
coe_extracted: /nomad/fairdi/coe/fs/extracted
nomad: /nomad
mail:
enabled: "true"
host: 'mailrelay.mpcdf.mpg.de'
port: 25
from: 'webmaster@nomad-repository.eu'
proxy:
nodePort: 30004
external:
host: "labdev-nomad.esc.rzg.mpg.de"
path: "/fairdi/nomad/coe"
api:
disableReset: false
worker:
replicas: 1
routing: "queue"
processes: 10
dbname: fairdi_nomad_coe
postgres:
publish_enabled: true
mode: 'coe'
dbname: 'nomad_prod'
host: 'db-repository-nomad.esc'
user: 'nomadrep'
password_secret: 'nomad-production-repository-password'
uploadurl: 'https://labdev-nomad.rzg.mpg.de/fairdi/nomad/coe/upload'
volumes:
prefixSize: 2
public: /nomad/fairdi/coe/fs/public
staging: /nomad/fairdi/coe/fs/staging
tmp: /nomad/fairdi/coe/fs/tmp
coe_extracted: /nomad/fairdi/coe/fs/extracted
nomad: /nomad
mail:
enabled: "true"
host: 'mailrelay.mpcdf.mpg.de'
port: 25
from: 'webmaster@nomad-repository.eu'
export NOMAD_CLIENT_URL=https://labdev-nomad.esc.rzg.mpg.de/fairdi/nomad/coe/api
export NOMAD_CLIENT_USER=admin
export NOMAD_MONGO_DB_NAME=fairdi_nomad_coe
export NOMAD_ELASTIC_INDEX_NAME=fairdi_nomad_coe
export NOMAD_FS_LOCAL_TMP=/nomad/fairdi/coe/fs/tmp
export NOMAD_FS_STAGING=/nomad/fairdi/coe/fs/staging
export NOMAD_FS_PUBLIC=/nomad/fairdi/coe/fs/public
export NOMAD_FS_TMP=/nomad/fairdi/coe/fs/tmp
\ No newline at end of file
......@@ -28,6 +28,9 @@ data:
window.nomadEnv = {
"apiBase": "{{ .Values.proxy.external.path }}/api",
"kibanaBase": "{{ .Values.proxy.external.kibanaPath }}",
"matomoSiteId": {{ .Values.gui.matomoSiteId }},
"matomoUrl": {{ .Values.gui.matomoUrl }},
"sendTrackingData": {{ .Values.gui.sendTrackingData }},
"debug": {{ .Values.gui.debug }}
};
---
......
......@@ -63,6 +63,12 @@ gui:
port: 8080
## This variable is used in the GUI to show or hide additional information
debug: false
## URL for matomo(piwik) user tracking
matomoUrl: 'http://nowhere.no',
## site id for matomo(piwik) user tracking
matomoSiteId: 1,
## send matomo(piwik) user tracking data
sendTrackingData: false
## Everything concerning the proxy that combined gui, api, kibana
# It is run via NodePort service
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment