README.md 6.56 KB
Newer Older
Markus Scheidgen's avatar
Markus Scheidgen committed
1
2
3
[![pipeline status](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/badges/master/pipeline.svg)](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/commits/master)
[![coverage report](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/badges/master/coverage.svg)](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/commits/master)

4
5
6
7
8
9
10
11
12
This project implements the new *nomad@FAIRDI* infrastructure. Contrary to its NOMAD CoE
predecessor, it implements the NOMAD Repository and NOMAD Archive functionality within
a single cohesive application. This project provides all necessary artifacts to develop,
test, deploy, and operate the NOMAD Respository and Archive, e.g. at
[https://repository.nomad-coe.eu/app/gui](https://repository.nomad-coe.eu/app/gui).

In the future, this project's aim is to integrate more NOMAD CoE components, like the NOMAD
Encyclopedia and NOMAD Analytics Toolkit, to fully integrate NOMAD with one GUI and consistent
APIs. Furthermore, this projects aims at establishing NOMAD as a distributed platform for
13
14
15
16
17
18
material science data sharing and management. This includes the on-site deployment of
NOMAD as a standalone service (*oasis*), the federated use of NOMAD through a
serious of full and partial *mirrors*, the integration of 3rd party material science
databases (i.e. [Aflow](http://www.aflow.org/), [OQMD](http://oqmd.org/),
[Materials Project](https://materialsproject.org/)), and support for open APIs and
standards like the [Optimade](http://www.optimade.org/) API.
19

20

21
## Getting started
22

Markus Scheidgen's avatar
Markus Scheidgen committed
23
24
25
26
27
28
### Using NOMAD as a Python package

You can install the `nomad` Python package from source distribution with pip. Please
note, that this will only install part of NOMAD's dependencies that will only allow
your to use NOMAD's client library, e.g. to access the NOMAD Archive.
```
Markus Scheidgen's avatar
Markus Scheidgen committed
29
pip install nomad --extra-index-url https://repository.nomad-coe.eu/v0.8/dist
Markus Scheidgen's avatar
Markus Scheidgen committed
30
31
32
33
```

To **use the NOMAD parsers for example**, install the `parsing` extra:
```
Markus Scheidgen's avatar
Markus Scheidgen committed
34
pip install nomad[parsing] --extra-index-url https://repository.nomad-coe.eu/v0.8/dist
Markus Scheidgen's avatar
Markus Scheidgen committed
35
36
37
38
39
nomad parse --show-backend <your-file-to-parse>
```

### For NOMAD developer

40
41
42
Read the [docs](https://repository.nomad-coe.eu/app/docs). The documentation is also part
of the source code. It covers aspects like introduction, architecture, development setup/deployment,
contributing, and API reference.
43

Markus Scheidgen's avatar
Markus Scheidgen committed
44
45

## Change log
46

Markus Scheidgen's avatar
Markus Scheidgen committed
47
Omitted versions are plain bugfix releases with only minor changes and fixes.
Markus Scheidgen's avatar
Markus Scheidgen committed
48

49
50
51
52
### v0.8.0
- new underlying datamodel that allows to maintain multiple domains
- mulitple domains supported the GUI
- new metainfo implementation
53
- API endpoint to access the metainfo
54
55
56
57
- new archive based on new metainfo
- client library that serves archive data as objects (with tab completion) not dictionaries
- properties and user tab in the search GUI
- improved performance on most parsers
Markus Scheidgen's avatar
Markus Scheidgen committed
58
- NOMAD source distribution
59

60
61
62
63
### v0.7.9
- Everything to run a simple NOMAD OASIS based on the central user-management
- minor bugfixes

64
65
66
67
### v0.7.7
- Shows dataset contents with embargo data, but hides the entry details (raw-files, archive)
- minor bugfixes

68
### v0.7.5
Markus Scheidgen's avatar
Markus Scheidgen committed
69
- AFLOWLIB prototypes (archive)
70
71
- primitive label search
- improved search performance based on excluded fields
Markus Scheidgen's avatar
Markus Scheidgen committed
72
- improved logs
73
74
- minor bugfixes

75
76
77
78
79
80
81
82
### v0.7.3
- fixed aborted raw-file downloads
- improved representation of data availability (staging, embargo, public) in GUI
- user data uploads ordered by upload time
- user data shows uploads with name
- configurable embargo period length
- minor bugfixes

83
84
85
86
### v0.7.2
- API curl, Python, and results for entries and search queries shown in the GUI
- minor bugfixes

Markus Scheidgen's avatar
Markus Scheidgen committed
87
88
89
90
### v0.7.1
- Download of archive files based on search queries
- minor bugfixes

Markus Scheidgen's avatar
Markus Scheidgen committed
91
### v0.7.0
92
93
- User metadata editing and datasets with DOIs
- Revised GUI lists (entries, grouped entries, datasets, uploads)
94
- Keycloak based user management
95
96
- Rawfile preview
- no dependencies with the old NOMAD CoE Repository
97
98
99
100
101
102
103

### v0.6.2
- GUI performance enhancements
- API /raw/query endpoint takes file pattern to further filter download contents and
  strips potential shared path prefixes for a cleaner download .zip
- Stipped common path prefixes in raw file downloads
- minor bugfixes
104

105
### v0.6.0
Markus Scheidgen's avatar
Markus Scheidgen committed
106
- GUI URL, and API endpoint that resolves NOMAD CoE legacy PIDs
107
108
- Support for datasets in the GUI
- more flexible search python module and repo API
Markus Scheidgen's avatar
Markus Scheidgen committed
109
- support for external_id
110
- support for code-based raw_id
111
- Optimade API 0.10.0
112
- GUI supports Optimade filter query and other quantities
113
114
- minor bugfixes

115
116
117
118
119
### v0.5.2
- allows to download large files over longer time period
- streamlined deployment without API+GUI proxy
- minor bugfixes

Markus Scheidgen's avatar
Markus Scheidgen committed
120
### v0.5.1
Markus Scheidgen's avatar
Markus Scheidgen committed
121
- integrated parsers Dmol3, qbox, molcas, fleur, and onetep
122
- API endpoint for query based raw file download
Markus Scheidgen's avatar
Markus Scheidgen committed
123
- improvements to admin cli: e.g. clean staging files, reprocess uploads based on codes
124
- improved error handling in the GUI
Markus Scheidgen's avatar
Markus Scheidgen committed
125
- lots of parser bugfixes
Markus Scheidgen's avatar
Markus Scheidgen committed
126
127
- lots of minor bugfixes

128
### v0.5.0
129
130
The first production version of nomad@fairdi as the upload API and gui for NOMAD
- Production ready software and deployments (term agreements, better GUI docs)
131
132
133
- Raw file API with support to list directories. This replaces the `files` calculation
  metadata key. It was necessary due to arbitrary large lists of *auxfiles* in some
  calculations.
134
- Search interface that contains all features of the CoE Repository GUI.
135
136
137
- Refactored search API that allows to search for entries (paginated + scroll),
  metrics based on quantity aggregations (+ paginated entries), quantity aggregations
  with all values via `after` key (+ paginated entries).
138
139
140
- reprocessing of published results (e.g. after parser/normalizer improvements)
- mirror functionality
- refactored command line interface (CLI)
141
- potential GUI user tracking capabilities
142
- many minor bugfixes
143

144
145
146
147
### v0.4.7
- more migration scripts
- minor bugfixes

148
149
150
151
152
153
### v0.4.6
- admin commands to directly manipulate upload data
- additional migration scripts
- fixed system normalizer to understand indexed atom labels correctly
- many minor bugfixes

154
155
156
157
158
### v0.4.5
- improved uploads view with published uploads
- support for publishing to the existing nomad CoE repository
- many minor bugfixes

159
### v0.4.4
160
- improved GUI navigation
161
162
- support for multiple domains
- info API endpoint
Markus Scheidgen's avatar
Markus Scheidgen committed
163
- metainfo browser
Markus Scheidgen's avatar
Markus Scheidgen committed
164
- support for latest exciting version
Markus Scheidgen's avatar
Markus Scheidgen committed
165
- bugfixes in system normalization
Markus Scheidgen's avatar
Markus Scheidgen committed
166
- many minor bugfixes
167

168
169
170
171
172
173
174
### v0.4.3
- more flexible celery routing
- config via nomad.yml
- repo_db can be disabled
- publishing of calculations with failed processing
- cli for managing running processing tasks

Markus Scheidgen's avatar
Markus Scheidgen committed
175
176
177
### v0.4.2
- bugfixes regarding the migration
- better migration configurability and reproducibility
Markus Scheidgen's avatar
Markus Scheidgen committed
178
- scales to multi node kubernetes deployment