README.md 6.46 KB
Newer Older
Markus Scheidgen's avatar
Markus Scheidgen committed
1
2
3
[![pipeline status](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/badges/master/pipeline.svg)](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/commits/master)
[![coverage report](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/badges/master/coverage.svg)](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/commits/master)

4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
This project implements the new *nomad@FAIRDI* infrastructure. It is currently used
to enable users to upload data, process the data, maintain a version of the NOMAD
archive and meta-info, provide search, inspection, and download to all NOMAD raw and
archive data. As a long term strategy, this project will integrate, refactor, and re-write
more and more of the existing NOMAD CoE components.

The overall goal of *nomad@FAIRDI* is to provide common interfaces to the main services of NOMAD:
*Repository*, *Archive*, and *Encyclopedia*. These interfaces comprise a graphical web-based
UI that allows users to upload data, supervise data processing, inspect and download
metadata, raw-files, and archive data, provide visual tools to explore the data, and
to learn more about advanced use modes, like API and Analytics Toolkit. The second interface
is a unified REST API with various endpoints that represent the core NOMAD services.
This will allow users the automated use of NOMAD for managing their data, and using
data on NOMAD for analytics. A specific way of using the API is through the NOMAD
Analytics Toolkit, which is revamped as a
[separate project](https://gitlab.mpcdf.mpg.de/nomad-lab/analytics-jupyterhub).

Furthermore, this projects aims at establishing NOMAD as a distributed platform for
material science data sharing and management. This includes the on-site deployment of
NOMAD as a standalone service (*oasis*), the federated use of NOMAD through a
serious of full and partial *mirrors*, the integration of 3rd party material science
databases (i.e. [Aflow](http://www.aflow.org/), [OQMD](http://oqmd.org/),
[Materials Project](https://materialsproject.org/)), and support for open APIs and
standards like the [Optimade](http://www.optimade.org/) API.
28

29
30
## Getting started

Markus Scheidgen's avatar
Markus Scheidgen committed
31
Read the docs. The documentation is part of the source code. It covers aspects like
32
33
34
35
36
37
introduction, architecture, development setup/deployment, contributing, and API reference.

### Read the docs on the latest deployed version

You can access the running system and its documentation here:

38
[https://repository.nomad-coe.eu/uploads/api/docs](https://repository.nomad-coe.eu/uploads/api/docs/index.html)
39
40

### Generate the docs from the source
41

42
First, clone this repo and init its submodules:
43
```
44
45
git clone git@gitlab.mpcdf.mpg.de:nomad-lab/nomad-FAIR.git
cd nomad-FAIR
Daniel Speckhard's avatar
Daniel Speckhard committed
46
git submodule init --depth 1
47
48
```

49
Second, create and source your own virtual python environment:
50
```
51
pip install virtualenv
52
virtualenv -p `which python3` .pyenv
53
source .pyenv/bin/activate
54
55
```

56
57
Third, install the development dependencies, including the documentation system
[sphinx](http://www.sphinx-doc.org/en/master/index.html):
58
```
59
60
pip install --upgrade pip
pip install --upgrade setuptools
Markus Scheidgen's avatar
Markus Scheidgen committed
61
pip install -r requirements.txt
62
63
```

64
Forth, generate the documentation:
65
```
66
67
cd docs
make html
68
69
```

70
Conintue with reading the documentation for further setup and contribution guidelines:
71
```
72
cd .build/html
73
python -m http.server 8888
74
```
75
76
Open [http://localhost:8888/html/setup.html](http://localhost:8888/html/setup.html) in
your browser.
Markus Scheidgen's avatar
Markus Scheidgen committed
77
78

## Change log
Markus Scheidgen's avatar
Markus Scheidgen committed
79
Omitted versions are plain bugfix releases with only minor changes and fixes.
Markus Scheidgen's avatar
Markus Scheidgen committed
80

Markus Scheidgen's avatar
Markus Scheidgen committed
81
82
83
84
### v0.7.1
- Download of archive files based on search queries
- minor bugfixes

Markus Scheidgen's avatar
Markus Scheidgen committed
85
### v0.7.0
86
87
- User metadata editing and datasets with DOIs
- Revised GUI lists (entries, grouped entries, datasets, uploads)
88
- Keycloak based user management
89
90
- Rawfile preview
- no dependencies with the old NOMAD CoE Repository
91
92
93
94
95
96
97

### v0.6.2
- GUI performance enhancements
- API /raw/query endpoint takes file pattern to further filter download contents and
  strips potential shared path prefixes for a cleaner download .zip
- Stipped common path prefixes in raw file downloads
- minor bugfixes
98

99
### v0.6.0
Markus Scheidgen's avatar
Markus Scheidgen committed
100
- GUI URL, and API endpoint that resolves NOMAD CoE legacy PIDs
101
102
- Support for datasets in the GUI
- more flexible search python module and repo API
Markus Scheidgen's avatar
Markus Scheidgen committed
103
- support for external_id
104
- support for code-based raw_id
105
- Optimade API 0.10.0
106
- GUI supports Optimade filter query and other quantities
107
108
- minor bugfixes

109
110
111
112
113
### v0.5.2
- allows to download large files over longer time period
- streamlined deployment without API+GUI proxy
- minor bugfixes

Markus Scheidgen's avatar
Markus Scheidgen committed
114
### v0.5.1
Markus Scheidgen's avatar
Markus Scheidgen committed
115
- integrated parsers Dmol3, qbox, molcas, fleur, and onetep
116
- API endpoint for query based raw file download
Markus Scheidgen's avatar
Markus Scheidgen committed
117
- improvements to admin cli: e.g. clean staging files, reprocess uploads based on codes
118
- improved error handling in the GUI
Markus Scheidgen's avatar
Markus Scheidgen committed
119
- lots of parser bugfixes
Markus Scheidgen's avatar
Markus Scheidgen committed
120
121
- lots of minor bugfixes

122
### v0.5.0
123
124
The first production version of nomad@fairdi as the upload API and gui for NOMAD
- Production ready software and deployments (term agreements, better GUI docs)
125
126
127
- Raw file API with support to list directories. This replaces the `files` calculation
  metadata key. It was necessary due to arbitrary large lists of *auxfiles* in some
  calculations.
128
- Search interface that contains all features of the CoE Repository GUI.
129
130
131
- Refactored search API that allows to search for entries (paginated + scroll),
  metrics based on quantity aggregations (+ paginated entries), quantity aggregations
  with all values via `after` key (+ paginated entries).
132
133
134
- reprocessing of published results (e.g. after parser/normalizer improvements)
- mirror functionality
- refactored command line interface (CLI)
135
- potential GUI user tracking capabilities
136
- many minor bugfixes
137

138
139
140
141
### v0.4.7
- more migration scripts
- minor bugfixes

142
143
144
145
146
147
### v0.4.6
- admin commands to directly manipulate upload data
- additional migration scripts
- fixed system normalizer to understand indexed atom labels correctly
- many minor bugfixes

148
149
150
151
152
### v0.4.5
- improved uploads view with published uploads
- support for publishing to the existing nomad CoE repository
- many minor bugfixes

153
### v0.4.4
154
- improved GUI navigation
155
156
- support for multiple domains
- info API endpoint
Markus Scheidgen's avatar
Markus Scheidgen committed
157
- metainfo browser
Markus Scheidgen's avatar
Markus Scheidgen committed
158
- support for latest exciting version
Markus Scheidgen's avatar
Markus Scheidgen committed
159
- bugfixes in system normalization
Markus Scheidgen's avatar
Markus Scheidgen committed
160
- many minor bugfixes
161

162
163
164
165
166
167
168
### v0.4.3
- more flexible celery routing
- config via nomad.yml
- repo_db can be disabled
- publishing of calculations with failed processing
- cli for managing running processing tasks

Markus Scheidgen's avatar
Markus Scheidgen committed
169
170
171
### v0.4.2
- bugfixes regarding the migration
- better migration configurability and reproducibility
Markus Scheidgen's avatar
Markus Scheidgen committed
172
- scales to multi node kubernetes deployment