Commit 21d41bf1 authored by Markus Scheidgen's avatar Markus Scheidgen
Browse files

Merge branch 'bugfixes' into 'v0.8.1'

Bugfixes

See merge request !119
parents 2fa4fa91 95615c3b
Pipeline #76329 passed with stages
in 19 minutes and 34 seconds
......@@ -107,9 +107,9 @@ install_tests:
- pip install numpy
- python setup.py compile
- python setup.py sdist
- pip install dist/nomad-0.8.0.tar.gz
- pip install dist/nomad-lab-*.tar.gz
- python -c "import nomad.datamodel, nomad.datamodel.metainfo, nomad.client"
- pip install dist/nomad-0.8.0.tar.gz[parsing]
- pip install `echo dist/nomad-lab-*.tar.gz`[parsing]
- python -m nomad.cli parse tests/data/parsers/vasp/vasp.xml
deploy:
......
......@@ -75,6 +75,7 @@ COPY . /app
RUN python setup.py compile
RUN pip install .[all]
RUN python setup.py sdist
RUN cp dist/nomad-lab-*.tar.gz dist/nomad-lab.tar.gz
WORKDIR /app/docs
RUN make html
......
......@@ -46,9 +46,12 @@ contributing, and API reference.
Omitted versions are plain bugfix releases with only minor changes and fixes.
### v0.8.1
- switched to support Python 3.7
### v0.8.0
- new underlying datamodel that allows to maintain multiple domains
- mulitple domains supported the GUI
- multiple domains supported the GUI
- new metainfo implementation
- API endpoint to access the metainfo
- new archive based on new metainfo
......@@ -175,4 +178,4 @@ The first production version of nomad@fairdi as the upload API and gui for NOMAD
### v0.4.2
- bugfixes regarding the migration
- better migration configurability and reproducibility
- scales to multi node kubernetes deployment
- scales to multi node kubernetes deployment
\ No newline at end of file
Subproject commit 5c98d3fc345beb43ecb8c163fb2f1ad9678f45a2
Subproject commit 2e385c1dbf934157d0e533ee709595fd0ccfb742
......@@ -2,11 +2,18 @@ Install the NOMAD client library
================================
We release the NOMAD client library as a Python `distutils <https://docs.python.org/3/library/distutils.html>`_ source distribution.
You can install it the usual way using *pip* (or *conda*).
You can download and install it the usual way using *pip* (or *conda*).
Install from pypi
.. parsed-literal::
pip install nomad --extra-index-url |pypi_url|
pip install nomad-lab
Download and install latest release from nomad
.. parsed-literal::
curl https://repository.nomad-coe.eu/v0.8/dist/nomad-lab.tar.gz -o nomad-lab.tar.gz
pip install ./nomad-lab.tar.gz
There are different layers of dependencies that you have to install, in order to use
certain functions of NOMAD. The base install above, will only install the
......@@ -19,9 +26,9 @@ requirements:
.. parsed-literal::
pip install nomad[parsing] --extra-index-url |pypi_url|
pip install nomad[infrastructure] --extra-index-url |pypi_url|
pip install nomad[dev] --extra-index-url |pypi_url|
pip install nomad-lab[parsing]
pip install nomad-lab[infrastructure]
pip install nomad-lab[dev]
The various *extras* have the following meaning:
......
......@@ -29,10 +29,6 @@ project = 'nomad-FAIRDI'
copyright = '2018, FAIRDI e.V.'
author = 'FAIRDI e.V.'
rst_epilog = '''
.. |pypi_url| replace:: https://repository.nomad-coe.eu/v0.8/dist/
'''
# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
......
{
"name": "nomad-fair-gui",
"version": "0.8.0",
"version": "0.8.1",
"commit": "e98694e",
"private": true,
"dependencies": {
......
......@@ -214,7 +214,7 @@ export default function About() {
There is a [tutorial on how to use the API with plain Python](${appBase}/docs/api_tutorial.html).
Another [tutorial covers how to install and use NOMAD's Python client library](${appBase}/docs/archive_tutorial.html).
The [NOMAD Analytics Toolkit](https://analytic-toolkit.nomad-coe.eu) allows to use
The [NOMAD Analytics Toolkit](https://analytics-toolkit.nomad-coe.eu) allows to use
this without installation and directly on NOMAD servers.
`}</Markdown></InfoCard>
<Grid item xs={12}>
......@@ -272,7 +272,7 @@ export default function About() {
- domains: ${info ? Object.keys(info.domains).map(domain => info.domains[domain].name).join(', ') : 'loading'}
- git: \`${info ? info.git.ref : 'loading'}; ${info ? info.git.version : 'loading'}\`
- last commit message: *${info ? info.git.log : 'loading'}*
- supported codes: ${info ? info.codes.join(', ') : 'loading'}
- supported codes: ${info ? info.codes.map(code => code.code_name).join(', ') : 'loading'}
- parsers: ${info ? info.parsers.join(', ') : 'loading'}
- normalizers: ${info ? info.normalizers.join(', ') : 'loading'}
`}</Markdown>
......
......@@ -6,28 +6,6 @@ import { CircularProgress, InputAdornment, Button, Tooltip } from '@material-ui/
import searchQuantities from '../../searchQuantities'
import { apiContext } from '../api'
/**
* A few helper functions related to format and analyse suggested options
*/
const Options = {
split: (suggestion) => {
let [quantity, value] = suggestion.split('=')
if (value && searchQuantities[quantity] && searchQuantities[quantity].many) {
value = value.split(',')
}
return [quantity, value]
},
join: (quantity, value) => `${quantity}=${value}`,
splitForCompare: (suggestion) => {
const [quantity, value] = suggestion.split('=')
return [quantity ? quantity.toLowerCase() : '', value ? value.toLowerCase() : '']
}
}
function getOptionLabel(option) {
return option.substring(option.indexOf('.') + 1)
}
/**
* This searchbar component shows a searchbar with autocomplete functionality. The
* searchbar also includes a status line about the current results. It uses the
......@@ -35,12 +13,19 @@ function getOptionLabel(option) {
* API calls to provide autocomplete suggestion options.
*/
export default function SearchBar() {
const suggestionsTimerRef = useRef(null)
const currentLoadOptionsConfigRef = useRef({
timer: null,
latestOption: null,
requestedOption: null
})
const {response: {statistics, pagination, error}, domain, query, apiQuery, setQuery} = useContext(searchContext)
const defaultOptions = useMemo(() => {
return Object.keys(searchQuantities)
.map(quantity => searchQuantities[quantity].name)
.filter(quantity => !quantity.includes('.') || quantity.startsWith(domain.key + '.'))
.map(quantity => ({
quantity: quantity,
domain: quantity.includes('.') ? quantity.split('.')[0] : null
}))
.filter(option => !option.domain || option.domain === domain.key)
}, [domain.key])
const [open, setOpen] = useState(false)
......@@ -51,15 +36,17 @@ export default function SearchBar() {
const {api} = useContext(apiContext)
const autocompleteValue = Object.keys(query).map(quantity => Options.join(quantity, query[quantity]))
const autocompleteValue = Object.keys(query).map(quantity => ({
quantity: quantity,
domain: quantity.includes('.') ? quantity.split('.')[0] : null,
value: query[quantity]
}))
const handleSearchTypeClicked = useCallback(() => {
if (searchType === 'nomad') {
setSearchType('optimade')
// handleChange(null, [])
} else {
setSearchType('nomad')
// handleChange(null, [])
}
}, [searchType, setSearchType])
......@@ -75,43 +62,43 @@ export default function SearchBar() {
helperText = <span>There are no more entries matching your criteria.</span>
} else {
helperText = <span>
There {pagination.total === 1 ? 'is' : 'are'} {Object.keys(domain.searchMetrics).filter(key => statistics.total.all[key]).map(key => {
return <span key={key}>
{domain.searchMetrics[key].renderResultString(statistics.total.all[key])}
</span>
})}{Object.keys(query).length ? ' left' : ''}.
There {pagination.total === 1 ? 'is' : 'are'} {
Object.keys(domain.searchMetrics).filter(key => statistics.total.all[key]).map(key => {
return <span key={key}>
{domain.searchMetrics[key].renderResultString(statistics.total.all[key])}
</span>
})
}{Object.keys(query).length ? ' left' : ''}.
</span>
}
}
const filterOptions = useCallback((options, params) => {
let [quantity, value] = Options.splitForCompare(params.inputValue)
const filteredOptions = options.filter(option => {
let [optionQuantity, optionValue] = Options.splitForCompare(option)
if (!value) {
return optionQuantity && (optionQuantity.includes(quantity) || optionQuantity === quantity)
} else {
return optionValue.includes(value) || optionValue === value
}
})
return filteredOptions
}, [])
const loadOptions = useCallback((quantity, value) => {
const size = searchQuantities[quantity].statistic_size
const loadOptions = useCallback(option => {
const config = currentLoadOptionsConfigRef.current
config.latestOption = option
if (suggestionsTimerRef.current !== null) {
clearTimeout(suggestionsTimerRef.current)
if (config.timer !== null) {
clearTimeout(config.timer)
}
if (loading) {
return
}
suggestionsTimerRef.current = setTimeout(() => {
config.timer = setTimeout(() => {
config.requestedOption = option
const size = searchQuantities[option.quantity].statistic_size
setLoading(true)
api.suggestions_search(quantity, apiQuery, size ? null : value, size || 20, true)
api.suggestions_search(option.quantity, apiQuery, size ? null : option.value, size || 20, true)
.then(response => {
setLoading(false)
const options = response.suggestions.map(value => Options.join(quantity, value))
if (!config.latestOption || config.requestedOption.quantity !== config.latestOption.quantity) {
// don't do anything if quantity has changed in the meantime
return
}
const options = response.suggestions.map(value => ({
quantity: option.quantity,
domain: option.domain,
value: value
}))
setOptions(options)
setOpen(true)
})
......@@ -119,37 +106,114 @@ export default function SearchBar() {
setLoading(false)
})
}, 200)
}, [api, suggestionsTimerRef, apiQuery, loading, setLoading])
}, [api, currentLoadOptionsConfigRef, apiQuery, loading, setLoading])
const getOptionLabel = useCallback(option => {
let label = option.quantity + '='
if (option.value) {
if (Array.isArray(option.value)) {
label += option.value.join(',')
} else {
label += option.value
}
}
return label.substring(label.indexOf('.') + 1)
}, [])
const parseOption = useCallback(input => {
const [inputQuantity, inputValue] = input.split('=')
let quantity = inputQuantity
let value = inputValue
if (!searchQuantities[quantity]) {
quantity = domain.key + '.' + quantity
}
if (value && searchQuantities[quantity] && searchQuantities[quantity].many) {
value = value.split(',').map(item => item.trim())
}
return {
inputQuantity: inputQuantity,
inputValue: inputValue,
domain: inputQuantity.includes('.') ? inputQuantity.split('.')[0] : null,
quantity: searchQuantities[quantity] ? quantity : null,
value: value
}
}, [domain.key])
const filterOptions = useCallback((options, params) => {
const inputOption = parseOption(params.inputValue)
const filteredOptions = options.filter(option => {
if (!inputOption.quantity) {
return option.quantity.includes(
inputOption.inputQuantity) && (option.domain === domain.key || !option.domain)
}
if (option.quantity !== inputOption.quantity) {
return false
}
if (!inputOption.value) {
return true
}
const matches = option.value &&
inputOption.inputValue &&
option.value.toLowerCase().includes(inputOption.inputValue.toLowerCase())
if (matches) {
if (option.value === inputOption.inputValue) {
inputOption.exists |= true
}
return true
}
return false
})
// Add the value as option, even if it does not exist to allow search for missing,
// faulty, or not yet loaded options
if (inputOption.quantity && !inputOption.exists) {
filteredOptions.push(inputOption)
}
return filteredOptions
}, [domain.key, parseOption])
const handleInputChange = useCallback((event, value, reason) => {
if (reason === 'input') {
setInputValue(value)
const [quantity, quantityValue] = Options.split(value)
if (searchQuantities[quantity]) {
loadOptions(quantity, quantityValue)
const inputOption = parseOption(value)
if (inputOption.quantity) {
loadOptions(inputOption)
} else {
setOptions(defaultOptions)
}
}
}, [loadOptions, defaultOptions])
}, [loadOptions, defaultOptions, parseOption])
const handleChange = (event, entries) => {
currentLoadOptionsConfigRef.current.latestOption = null
entries = entries.map(entry => {
if (typeof entry === 'string') {
return parseOption(entry)
} else {
return entry
}
})
const newQuery = entries.reduce((query, entry) => {
if (entry) {
const [quantity, value] = Options.split(entry)
if (query[quantity]) {
if (searchQuantities[quantity].many) {
if (Array.isArray(query[quantity])) {
query[quantity].push(value)
if (query[entry.quantity]) {
if (searchQuantities[entry.quantity].many) {
if (Array.isArray(query[entry.quantity])) {
query[entry.quantity].push(entry.value)
} else {
query[quantity] = [query[quantity], value]
query[entry.quantity] = [query[entry.quantity], entry.value]
}
} else {
query[quantity] = value
query[entry.quantity] = entry.value
}
} else {
query[quantity] = value
query[entry.quantity] = entry.value
}
}
return query
......@@ -158,12 +222,11 @@ export default function SearchBar() {
if (entries.length !== 0) {
const entry = entries[entries.length - 1]
const [quantity, value] = Options.split(entry)
if (value) {
if (entry.value) {
setInputValue('')
} else {
setInputValue(`${getOptionLabel(entry)}=`)
loadOptions(quantity)
setInputValue(getOptionLabel(entry))
loadOptions(entry)
}
}
}
......@@ -213,11 +276,14 @@ export default function SearchBar() {
}}
onChange={handleChange}
onInputChange={handleInputChange}
getOptionSelected={(option, value) => option === value}
getOptionSelected={(option, inputOption) => {
return inputOption.quantity === option.quantity && inputOption.value === option.value
}}
getOptionLabel={getOptionLabel}
options={options}
loading={loading}
filterOptions={filterOptions}
// handleHomeEndKeys
renderInput={(params) => (
<TextField
{...commonTextFieldProps(params)}
......@@ -243,7 +309,6 @@ export default function SearchBar() {
}}
defaultValue={query['dft.optimade'] || ''}
onKeyPress={(ev) => {
console.log(`Pressed keyCode ${ev.key}`)
if (ev.key === 'Enter') {
handleOptimadeEntered(ev.target.value)
ev.preventDefault()
......
......@@ -2,7 +2,7 @@ import { createMuiTheme } from '@material-ui/core'
window.nomadEnv = window.nomadEnv || {}
export const appBase = window.nomadEnv.appBase.replace(/\/$/, '')
// export const apiBase = 'http://labdev-nomad.esc.rzg.mpg.de/fairdi/nomad/testing-major/api'
// export const apiBase = 'http://repository.nomad-coe.eu/v0.8/api'
export const apiBase = `${appBase}/api`
export const optimadeBase = `${appBase}/optimade`
export const guiBase = process.env.PUBLIC_URL
......
......@@ -20,5 +20,8 @@ from flask import Blueprint
from flask_restplus import Api
from .api import blueprint, url, api
from .endpoints import CalculationList, Calculation
# TODO ReferenceList, Reference, ReferenceInfo, Links are missing, because the implement
# the wrong thing.
from .endpoints import CalculationList, Calculation, CalculationInfo, Info, Structure, StructuresInfo, StructureList
from .index import Info
from .filterparser import parse_filter
......@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
from flask import Blueprint
from flask import Blueprint, request, abort
from flask_restplus import Api
import urllib.parse
......@@ -25,12 +25,18 @@ base_url = 'http://%s/%s/optimade' % (
config.services.api_base_path.strip('/'))
def url(endpoint: str = None, **kwargs):
def url(endpoint: str = None, version='v0', **kwargs):
''' Returns the full optimade api url (for a given endpoint) including query parameters. '''
if endpoint is None:
url = base_url
if version is not None:
url = '%s/%s' % (base_url, version)
else:
url = base_url
else:
url = '%s/%s' % (base_url, endpoint)
if version is not None:
url = '%s/%s/%s' % (base_url, version, endpoint)
else:
url = '%s/%s' % (base_url, endpoint)
if len(kwargs) > 0:
return '%s?%s' % (url, urllib.parse.urlencode(kwargs))
......@@ -38,6 +44,17 @@ def url(endpoint: str = None, **kwargs):
return url
# TODO replace with decorator that filters response_fields
def base_request_args():
if request.args.get('response_format', 'json') != 'json':
abort(400, 'Response format is not supported.')
properties_str = request.args.get('request_fields', None)
if properties_str is not None:
return properties_str.split(',')
return None
api = Api(
blueprint,
version='1.0', title='NOMAD\'s OPTiMaDe API implementation',
......
......@@ -20,7 +20,7 @@ from elasticsearch_dsl import Q
from nomad import search, files, datamodel
from nomad.datamodel import OptimadeEntry
from .api import api, url
from .api import api, url, base_request_args
from .models import json_api_single_response_model, entry_listing_endpoint_parser, Meta, \
Links as LinksModel, CalculationDataObject, single_entry_endpoint_parser, base_endpoint_parser, \
json_api_info_response_model, json_api_list_response_model, ReferenceObject, StructureObject, \
......@@ -28,19 +28,7 @@ from .models import json_api_single_response_model, entry_listing_endpoint_parse
json_api_structure_response_model, json_api_structures_response_model
from .filterparser import parse_filter, FilterException
ns = api.namespace('', description='The (only) API namespace with all OPTiMaDe endpoints.')
# TODO replace with decorator that filters response_fields
def base_request_args():
if request.args.get('response_format', 'json') != 'json':
abort(400, 'Response format is not supported.')
properties_str = request.args.get('request_fields', None)
if properties_str is not None:
return properties_str.split(',')
return None
ns = api.namespace('v0', description='The version v0 API namespace with all OPTiMaDe endpoints.')
def base_search_request():
......@@ -74,6 +62,10 @@ def to_calc_with_metadata(results: List[Dict[str, Any]]):
return result
# TODO the Entry/ListEntry endpoints for References, Calculations, Structures should
# reuse more code.
# Calculations are identical to structures. Not sure if this is what the optimade
# specification intends.
@ns.route('/calculations')
class CalculationList(Resource):
@api.doc('list_calculations')
......@@ -259,6 +251,9 @@ class References(Resource):
results = to_calc_with_metadata(result['results'])
assert len(results) == len(result['results']), 'Mongodb and elasticsearch are not consistent'
# TODO References are about returning user provided references to paper or web resources.
# The ReferenceObject does not have this kind of information.
# TODO Why is TopLevelLinks different from LinksModel. Any what is "TopLevel" about it.
return dict(
meta=Meta(
query=request.url,
......@@ -312,12 +307,14 @@ class Links(Resource):
page_limit=page_limit,
sort=sort, filter=filter
),
# TODO Links are about links to other optimade databases, e.g. OQMD, MP, AFLOW.
# It is not about links within NOMAD, like LinkObject suggests.
data=[LinkObject(d, page_number=page_number, sort=sort, filter=filter) for d in results]
)
@ns.route('/structures')
class Structures(Resource):
class StructureList(Resource):
@api.doc('structures')
@api.response(400, 'Invalid requests, e.g. bad parameter.')
@api.response(422, 'Validation error')
......@@ -386,3 +383,29 @@ class Structure(Resource):
meta=Meta(query=request.url, returned=1),
data=StructureObject(results[0], request_fields=request_fields)
), 200
@ns.route('/info/structures')
class StructuresInfo(Resource):
@api.doc('structures_info')
@api.response(400, 'Invalid requests, e.g. bad parameter.')
@api.expect(base_endpoint_parser, validate=True)
@api.marshal_with(json_api_info_response_model, skip_none=True, code=200)
def get(self):
''' Returns information relating to the API implementation- '''
base_request_args()
result = {
'description': 'a structure entry',
'properties': {
attr.name: dict(description=attr.description)
for attr in OptimadeEntry.m_def.all_properties.values()},
'formats': ['json'],
'output_fields_by_format': {
'json': list(OptimadeEntry.m_def.all_properties.keys())}
}
return dict(
meta=Meta(query=request.url, returned=1),
data=result
), 200
# Copyright 2018 Markus Scheidgen
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at