Commit 05778e75 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'v0.7.0' into 'master'

V0.7.0

See merge request !71
parents 77b45967 769f314d
Pipeline #65673 passed with stage
in 27 seconds
...@@ -17,5 +17,6 @@ local/ ...@@ -17,5 +17,6 @@ local/
target/ target/
*.swp *.swp
*.vscode *.vscode
.vscode/
vscode/ vscode/
nomad.yaml nomad.yaml
...@@ -76,10 +76,6 @@ tests: ...@@ -76,10 +76,6 @@ tests:
stage: test stage: test
image: $TEST_IMAGE image: $TEST_IMAGE
services: services:
- postgres:latest
# this will cause a warning, as the gitlab ci runner health check will test the wrong
# port on rabbitmq container:
# https://gitlab.com/gitlab-org/gitlab-runner/issues/3163
- rabbitmq - rabbitmq
- name: docker.elastic.co/elasticsearch/elasticsearch:6.3.2 - name: docker.elastic.co/elasticsearch/elasticsearch:6.3.2
alias: elastic alias: elastic
...@@ -87,17 +83,15 @@ tests: ...@@ -87,17 +83,15 @@ tests:
# https://gitlab.com/gitlab-org/gitlab-ce/issues/42214 # https://gitlab.com/gitlab-org/gitlab-ce/issues/42214
command: [ "bin/elasticsearch", "-Ediscovery.type=single-node" ] command: [ "bin/elasticsearch", "-Ediscovery.type=single-node" ]
variables: variables:
POSTGRES_DB: nomad
POSTGRES_USER: postgres
POSTGRES_PASSWORD: nomad
RABBITMQ_ERLANG_COOKIE: SWQOKODSQALRPCLNMEQG RABBITMQ_ERLANG_COOKIE: SWQOKODSQALRPCLNMEQG
RABBITMQ_DEFAULT_USER: rabbitmq RABBITMQ_DEFAULT_USER: rabbitmq
RABBITMQ_DEFAULT_PASS: rabbitmq RABBITMQ_DEFAULT_PASS: rabbitmq
RABBITMQ_DEFAULT_VHOST: / RABBITMQ_DEFAULT_VHOST: /
NOMAD_REPOSITORY_DB_HOST: postgres
NOMAD_RABBITMQ_HOST: rabbitmq NOMAD_RABBITMQ_HOST: rabbitmq
NOMAD_ELASTIC_HOST: elastic NOMAD_ELASTIC_HOST: elastic
NOMAD_MONGO_HOST: mongo NOMAD_MONGO_HOST: mongo
NOMAD_KEYCLOAK_CLIENT_SECRET: ${CI_KEYCLOAK_TEST_CLIENT_SECRET}
NOMAD_KEYCLOAK_PASSWORD: ${CI_KEYCLOAK_ADMIN_PASSWORD}
script: script:
- cd /app - cd /app
- python -m pytest --cov=nomad -sv tests - python -m pytest --cov=nomad -sv tests
......
...@@ -44,6 +44,7 @@ RUN pip install mdanalysis==0.16.2 ...@@ -44,6 +44,7 @@ RUN pip install mdanalysis==0.16.2
# Make will be necessary to build the docs with sphynx # Make will be necessary to build the docs with sphynx
RUN apt-get update && apt-get install -y make RUN apt-get update && apt-get install -y make
RUN apt-get update && apt-get install -y vim
# We also install the -dev dependencies, to use this image for test and qa # We also install the -dev dependencies, to use this image for test and qa
COPY requirements.txt /install/requirements.txt COPY requirements.txt /install/requirements.txt
......
...@@ -78,6 +78,13 @@ your browser. ...@@ -78,6 +78,13 @@ your browser.
## Change log ## Change log
Omitted versions are plain bugfix releases with only minor changes and fixes. Omitted versions are plain bugfix releases with only minor changes and fixes.
### v0.7.0
- User metadata editing and datasets with DOIs
- Revised GUI lists (entries, grouped entries, datasets, uploads)
- Keycloak based user management
- Rawfile preview
- no dependencies with the old NOMAD CoE Repository
### v0.6.2 ### v0.6.2
- GUI performance enhancements - GUI performance enhancements
- API /raw/query endpoint takes file pattern to further filter download contents and - API /raw/query endpoint takes file pattern to further filter download contents and
......
...@@ -81,13 +81,11 @@ processing of uploaded files and therein contained calculations. We use ...@@ -81,13 +81,11 @@ processing of uploaded files and therein contained calculations. We use
[mongoengine](http://docs.mongoengine.org/) to program with mongodb. [mongoengine](http://docs.mongoengine.org/) to program with mongodb.
### PostgreSQL ### Keycloak
A relational database is used to store all user provided metadata: users, datasets [Keycloak](https://www.keycloak.org/) is used for user management. It manages users and
(curated sets of uploaded data), references, comments, DOIs, coauthors, etc. provide functions for registering, password forget, editing user accounts, and single
Furthermore, it is still used to store some of the calculation metadata derived sign on of fairdi@nomad and other related services.
via parsing. *This will most likely move out of Postgres in the future.* We
use [SQLAlchemy](https://docs.sqlalchemy.org/en/latest/) as on ORM framework.
### flask, et al. ### flask, et al.
...@@ -124,7 +122,7 @@ The component library [Material-UI](https://material-ui.com/) ...@@ -124,7 +122,7 @@ The component library [Material-UI](https://material-ui.com/)
### docker ### docker
To run a **nomad@FAIRDI** instance, many services have to be orchestrated: To run a **nomad@FAIRDI** instance, many services have to be orchestrated:
the nomad app, nomad worker, mongodb, Elasticsearch, PostgreSQL, RabbitMQ, the nomad app, nomad worker, mongodb, Elasticsearch, Keycloak, RabbitMQ,
Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together. Elasticstack (logging), the nomad GUI, and a reverse proxy to keep everything together.
Further services might be needed (e.g. JypiterHUB), when nomad grows. Further services might be needed (e.g. JypiterHUB), when nomad grows.
The container platform [Docker](https://docs.docker.com/) allows us to provide all services The container platform [Docker](https://docs.docker.com/) allows us to provide all services
...@@ -224,7 +222,7 @@ passed, stored, etc. by the various nomad modules. ...@@ -224,7 +222,7 @@ passed, stored, etc. by the various nomad modules.
### Implementation ### Implementation
The different entities have often multiple implementations for different storage systems. The different entities have often multiple implementations for different storage systems.
For example, aspects of calculations are stored in files (raw files, calc metadata, archive data), For example, aspects of calculations are stored in files (raw files, calc metadata, archive data),
Postgres (user metadata), Elasticsearch (metadata), and mongodb (processing state). Elasticsearch (metadata), and mongodb (metadata, processing state).
Different transformation between different implementations exist. See Different transformation between different implementations exist. See
:py:mod:`nomad.datamodel` for further information. :py:mod:`nomad.datamodel` for further information.
......
...@@ -42,10 +42,6 @@ nomad.search ...@@ -42,10 +42,6 @@ nomad.search
.. automodule:: nomad.search .. automodule:: nomad.search
:members: :members:
nomad.coe_repo
--------------
.. automodule:: nomad.coe_repo
nomad.app nomad.app
--------- ---------
.. automodule:: nomad.app .. automodule:: nomad.app
......
...@@ -281,7 +281,7 @@ The rest should be mocked or provided by the tests. Make sure that you do no run ...@@ -281,7 +281,7 @@ The rest should be mocked or provided by the tests. Make sure that you do no run
worker, as they will fight for tasks in the queue. worker, as they will fight for tasks in the queue.
``` ```
cd ops/docker-compose cd ops/docker-compose
docker-compose up -d elastic rabbitmq postgres docker-compose up -d elastic rabbitmq
cd ../.. cd ../..
pytest -svx tests pytest -svx tests
``` ```
......
...@@ -143,7 +143,7 @@ def publish_upload(upload, calc_metadata): ...@@ -143,7 +143,7 @@ def publish_upload(upload, calc_metadata):
'comment': 'Data from a cool external project', 'comment': 'Data from a cool external project',
'references': ['http://external.project.eu'], 'references': ['http://external.project.eu'],
# '_uploader': <nomad_user_id>, # only works if the admin user is publishing # '_uploader': <nomad_user_id>, # only works if the admin user is publishing
# 'co_authors': [<nomad_user_id>, <nomad_user_id>, <nomad_user_id>] # 'coauthors': [<nomad_user_id>, <nomad_user_id>, <nomad_user_id>]
# these are calc specific metadata that supercede any upload metadata # these are calc specific metadata that supercede any upload metadata
'calculations': calc_metadata} 'calculations': calc_metadata}
......
...@@ -3,16 +3,18 @@ ...@@ -3,16 +3,18 @@
"version": "nomad-gui-version-placeholder", "version": "nomad-gui-version-placeholder",
"private": true, "private": true,
"dependencies": { "dependencies": {
"@material-ui/core": "^3.9.0", "@material-ui/core": "^3.9.3",
"@material-ui/icons": "^3.0.2", "@material-ui/icons": "^3.0.2",
"@navjobs/upload": "^3.1.3", "@navjobs/upload": "^3.1.3",
"autosuggest-highlight": "^3.1.1", "autosuggest-highlight": "^3.1.1",
"base-64": "^0.1.0", "base-64": "^0.1.0",
"chroma-js": "^2.0.3", "chroma-js": "^2.0.3",
"clsx": "^1.0.4",
"d3": "^5.9.1", "d3": "^5.9.1",
"fetch": "^1.1.0", "fetch": "^1.1.0",
"file-saver": "^2.0.0", "file-saver": "^2.0.0",
"html-to-react": "^1.3.3", "html-to-react": "^1.3.3",
"keycloak-js": "^6.0.0",
"marked": "^0.6.0", "marked": "^0.6.0",
"material-ui-chip-input": "^1.0.0-beta.14", "material-ui-chip-input": "^1.0.0-beta.14",
"material-ui-flat-pagination": "^3.2.0", "material-ui-flat-pagination": "^3.2.0",
...@@ -28,7 +30,9 @@ ...@@ -28,7 +30,9 @@
"react-dom": "^16.4.2", "react-dom": "^16.4.2",
"react-dropzone": "^5.0.1", "react-dropzone": "^5.0.1",
"react-highlight": "^0.12.0", "react-highlight": "^0.12.0",
"react-infinite-scroller": "^1.2.4",
"react-json-view": "^1.19.1", "react-json-view": "^1.19.1",
"react-keycloak": "^6.1.0",
"react-router-dom": "^4.3.1", "react-router-dom": "^4.3.1",
"react-router-hash-link": "^1.2.0", "react-router-hash-link": "^1.2.0",
"react-scripts": "1.1.4", "react-scripts": "1.1.4",
......
window.nomadEnv = { window.nomadEnv = {
'keycloakBase': 'https://labdev-nomad.esc.rzg.mpg.de/fairdi/keycloak/auth/',
'keycloakRealm': 'fairdi_nomad_test',
'keycloakClientId': 'nomad_gui_dev',
'appBase': 'http://localhost:8000/fairdi/nomad/latest', 'appBase': 'http://localhost:8000/fairdi/nomad/latest',
'kibanaBase': '/fairdi/kibana', 'kibanaBase': '/fairdi/kibana',
'debug': false 'debug': false
......
...@@ -35,7 +35,7 @@ ...@@ -35,7 +35,7 @@
<link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons"> <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons">
<title>NOMAD upload</title> <title>NOMAD upload</title>
</head> </head>
<body> <body style="overflow-y: hidden;">
<noscript> <noscript>
You need to enable JavaScript to run this app. You need to enable JavaScript to run this app.
</noscript> </noscript>
......
...@@ -36,7 +36,7 @@ img.bg { ...@@ -36,7 +36,7 @@ img.bg {
} }
.pace .pace-progress { .pace .pace-progress {
background: #feb60d; background: #ffb300;
position: fixed; position: fixed;
z-index: 2000; z-index: 2000;
top: 0; top: 0;
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment