Limits of the Nomad Repo API
Currently, the GUI in the Nomad Repository uses both repo/old-api
and repo/search
endpoints in order to get values from the Elasticsearch index1. Per default, the number of values returned is maximum 100. It is however possible to get up to 10000 values by adding a limit parameter (limit
, num_results
, num_groups
). What follows is a list of all typical actions performed by the user on the GUI and the associated API calls. For this issue, I would like to focus on the limits because we had several problems in the past which were related to them. For examples, the user cannot see all of his calculations if he has more than 1000. Moreover, the 'download whole dataset' functionality does not accept more than 5000 calculations as it is currently. In my opinion, it would be good to review some of these limits and at least to be aware at all that they are there.
-
Home page loading
repo/old-api/topics
-
Select facets
repo/old-api/facets?tids=58
-
Searches as an unlogged user
- General search with facets (groups displayed)
repo/old-api/groups?num_groups=300&tids=58
repo/old-api/datasets?num_groups=100&&tids=58
- additional filtering from dataset
repo/old-api/results?limit=1000&datasets=3159373&tids=58
- dataset display
repo/search/datasets?query=dataset_calc_id=3159373
repo/search/calculations_oldformat?query=dataset_calc_id=3159373&num_results=5000
- General search with facets (groups displayed)
-
DOI search
repo/search/dataset_dois?num_results=1000
repo/search/citations?num_results=1000
- DOI internal view
repo/search/dataset_dois?num_results=1000
repo/search/datasets?query=dataset_doi_name=%22http%3A%2F%2Fdx.doi.org%2F10.17172%2FNOMAD%2F2017.03.17-1%22&num_results=5000
repo/search/datasets?query=dataset_calc_id=3329405
repo/search/calculations_oldformat?query=dataset_calc_id=3329405&num_results=5000
- clicking the “show all dataset members“ button
repo/search/datasets?query=dataset_calc_id=3311033
repo/search/calculations_oldformat?query=dataset_calc_id=3311033&num_results=5000
- DOI external view
repo/search/dataset_dois?num_results=1000
repo/search/calculations_oldformat?query=citation_value=%2210.1039%2Fc5cp01236h%22&num_results=5000
- clicking the “show all dataset members“ button
repo/search/datasets?query=dataset_calc_id=3329489
repo/search/calculations_oldformat?query=dataset_calc_id=3329489&num_results=5000
- DOI internal view
-
As a logged-in user
-
Uploaded data
repo/old-api/results?limit=1000&data_source=uploader,68
repo/old-api/datasets?num_groups=100&&data_source=uploader,68
repo/old-api/results?data_source=uploader,68
repo/old-api/uploads?data_source=uploader,68
-
Coauthored data
repo/old-api/results?limit=1000&data_source=coauthor,68
repo/old-api/datasets?num_groups=100&&data_source=coauthor,68
repo/old-api/results?data_source=uploader,68
repo/old-api/uploads?data_source=uploader,68
-
Shared data
repo/old-api/results?limit=1000&data_source=shared,68
repo/old-api/datasets?num_groups=100&&data_source=shared,68
repo/old-api/results?data_source=uploader,68
repo/old-api/uploads?data_source=uploader,68
-
-
Downloads
- Group download
repo/search/calculations_oldformat?query=repository_grouping_checksum="g--DwDBawNvevCUO6dPyHxJd_LbY8"
- Calculation download
repo/search/calculations_oldformat?query=repository_calc_id=2133
- Whole dataset download
repo/search/calculations_oldformat?query=dataset_calc_id=3159373&num_results=5000
- Group download