Commit 397a607a authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Updated api docs. Updated statistics. Use the right parser version.

parent 121ba3ed
Pipeline #69255 passed with stages
in 17 minutes and 53 seconds
Subproject commit 15d0110cbeda05aaea05e4d30ba3aeb0874dafef
Subproject commit 022a2af6bad45364dbdfac6b6c913f04186ac7d4
Subproject commit b39569c5fa69254c90f91ec430d28a0941efbe95
Subproject commit fe15759f080e8176d88af91447949243608b0d7e
......@@ -17,8 +17,13 @@ We do not assume many specific python packages. Only the *bravado* package (avai
via pipy) is required. It allows us to use the nomad ReST API in a more friendly and
pythonic way. You can simply install it the usual way
Optionally, if you need to access your private data, the package *python-keycloak* is
required to conveniently acquire the necessary tokens to authenticate your self towards
NOMAD.
```
pip install bravado
pip install python-keycloak
```
For the following code snippets, we need the following imports:
......@@ -33,6 +38,12 @@ import os.path
import sys
```
And optionally:
```python
from bravado.requests_client import RequestsClient, Authenticator
from keycloak import KeycloakOpenID
```
### An example file
Lets assume you have an example upload file ready. Its a `.zip` (`.tgz` would also work)
with some *VASP* data from a single run at `/example/AcAg/vasprun.xml`, `/example/AcAg/OUTCAR`, ...
......@@ -55,13 +66,47 @@ password = 'password'
### Using bravado
Bravado reads a ReST API's definition from a `swagger.json` as it is provided by
many APIs, including nomad's of course. Bravado also allows to use authentication,
which makes it even easier. The following would be a typical setup:
many APIs, including nomad's of course.
```python
host = urlparse(nomad_url).netloc.split(':')[0]
http_client = RequestsClient()
http_client.set_basic_auth(host, user, password)
client = SwaggerClient.from_url('%s/swagger.json' % nomad_url, http_client=http_client)
```
Bravado also allows to use authentication, if required. The following would be a typical setup:
```python
class KeycloakAuthenticator(Authenticator):
""" A bravado authenticator for NOMAD's keycloak-based user management. """
def __init__(self, user, password):
super().__init__(host=urlparse(nomad_url).netloc.split(':')[0])
self.user = user
self.password = password
self.token = None
self.__oidc = KeycloakOpenID(
server_url='https://repository.nomad-coe.eu/fairdi/keycloak/auth/',
realm_name='fairdi_nomad_prod',
client_id='nomad_public')
def apply(self, request):
if self.token is None:
self.token = self.__oidc.token(username=self.user, password=self.password)
self.token['time'] = time()
elif self.token['expires_in'] < int(time()) - self.token['time'] + 10:
try:
self.token = self.__oidc.refresh_token(self.token['refresh_token'])
self.token['time'] = time()
except Exception:
self.token = self.__oidc.token(username=self.user, password=self.password)
self.token['time'] = time()
request.headers.setdefault('Authorization', 'Bearer %s' % self.token['access_token'])
return request
http_client = RequestsClient()
http_client.authenticator = KeycloakAuthenticator(user=user, password=password)
client = SwaggerClient.from_url('%s/swagger.json' % nomad_url, http_client=http_client)
```
......@@ -192,7 +237,30 @@ print('%s/raw/%s/%s/*' % (nomad_url, calc['upload_id'], os.path.dirname(calc['ma
There are different options to download individual files, or zips with multiple files.
## Using *curl* to access the API
The shell tool *curl* can be used to call most API endpoints. Most endpoints for searching
or downloading data are only **GET** operations controlled by URL parameters. For example:
Downloading data:
```
curl http://repository.nomad-coe.eu/app/api/raw/query?upload_id=<your_upload_id> -o download.zip
```
It is a litle bit trickier, if you need to authenticate yourself, e.g. to download
not yet published or embargoed data. All endpoints support and most require the use of
an access token. To acquire an access token from our usermanagement system with curl:
```
curl --data 'grant_type=password&client_id=nomad_public&username=<your_username>&password=<your password>' \
https://repository.nomad-coe.eu/fairdi/keycloak/auth/realms/fairdi_nomad_prod/protocol/openid-connect/token
```
You can use the access-token with:
```
curl -H 'Authorization: Bearer <you_access_token>' \
http://repository.nomad-coe.eu/app/api/raw/query?upload_id=<your_upload_id> -o download.zip
```
## Conclusions
This was just a small glimpse into the nomad API. You should checkout our swagger documentation
for more details on all the API endpoints and their parameters. You can explore the
API via swagger-ui and even try it in your browser. Just visit the API url.
This was just a small glimpse into the nomad API. You should checkout our [swagger-ui](https://repository.nomad-coe.eu/app/api/) for more details on all the API endpoints and their parameters. You can explore the
API via the swagger-ui and even try it in your browser.
......@@ -24,7 +24,9 @@ import numpy as np
import click
import json
from datetime import datetime
import subprocess
from nomad import config
from .client import client
......@@ -359,7 +361,8 @@ def statistics_plot(errors, title, x_axis, y_axis, cumulate, total, save, power,
@client.command(help='Generate table with basic statistics summary.')
@click.option('--html', is_flag=True, help='Output HTML instead of plain text table.')
@click.option('--geometries', is_flag=True, help='Use geometries not unique geometries.')
def statistics_table(html, geometries):
@click.option('--public-path', type=str, default=config.fs.public, help='The path to the public data. Default is %s.' % config.fs.public)
def statistics_table(html, geometries, public_path):
# get more stats for files
# uploads: find . -maxdepth 2 | wc -l
# public archive: find . -regex '.*archive.*public.*zip' -type f -print0 | du --files0-from=- -ch | grep total$
......@@ -412,17 +415,43 @@ def statistics_table(html, geometries):
client.repo.search(per_page=1, code_name='Phonopy').response().result,
'total', 'all', 'code_runs')
# files and sized
def run_shell_command(command):
process = subprocess.run(['bash', '-c', command], stdout=subprocess.PIPE)
out = process.stdout.decode('utf-8')
if process.stderr is not None:
err = process.stderr.decode('utf-8')
print('There is an error: %s' % str(err))
return out
archive_data = run_shell_command((
'find %s -regex \'.*archive.*public.*zip\' '
'-type f -print0 | du --files0-from=- -ch | grep total$') % public_path)
raw_data = run_shell_command((
'find %s -regex \'.*raw.*public.*zip\' '
'-type f -print0 | du --files0-from=- -ch | grep total$') % public_path)
n_uploads = run_shell_command(
'find %s -regex \'.*raw.*public.*zip\' -type f | wc -l' % public_path)
try:
n_uploads = '{:,}'.format(int(n_uploads))
except Exception:
pass
if not html:
print('''
Entries: {:,.0f},
Calculations, e.g. total energies: {:,.0f},
Geometries: {:,.0f},
Bulk crystals: {:,.0f},
2D / Surfaces: {:,.0f},
Atoms / Molecules: {:,.0f},
DOS: {:,.0f},
Entries: {:,.0f}
Calculations, e.g. total energies: {:,.0f}
Geometries: {:,.0f}
Bulk crystals: {:,.0f}
2D / Surfaces: {:,.0f}
Atoms / Molecules: {:,.0f}
DOS: {:,.0f}
Band structures: {:,.0f}
Total parsed quantities: {:,.0f}
Public raw data: {}
Public archive data: {}
Number of uploads: {}
'''.format(
entries,
calculations,
......@@ -432,7 +461,10 @@ def statistics_table(html, geometries):
calculations_1d,
dos,
band_structures,
quantities
quantities,
raw_data,
archive_data,
n_uploads
))
else:
......@@ -497,8 +529,8 @@ def statistics_table(html, geometries):
Furthermore:
</p>
<ul>
<li><b>5,053</b> Uploads with <b>41 TB</b> of raw data</li>
<li><b>15 TB</b> of archive data</li>
<li><b>{}</b> Uploads with <b>{}B</b> of raw data</li>
<li><b>{}B</b> of archive data</li>
<li>Data classified using <b>168</b> public metadata of the NOMAD Meta Info and <b>2,360</b> code-specific metadata</li>
</ul>
<p>
......@@ -551,5 +583,8 @@ def statistics_table(html, geometries):
dos,
band_structures,
phonons,
quantities
quantities,
n_uploads,
raw_data,
archive_data
))
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment