Skip to content
GitLab
Closed
Issue created by Nicolas Fabas@fnicoDeveloper

Limits of the Nomad Repo API

Currently, the GUI in the Nomad Repository uses both repo/old-api and repo/search endpoints in order to get values from the Elasticsearch index1. Per default, the number of values returned is maximum 100. It is however possible to get up to 10000 values by adding a limit parameter (limit, num_results, num_groups). What follows is a list of all typical actions performed by the user on the GUI and the associated API calls. For this issue, I would like to focus on the limits because we had several problems in the past which were related to them. For examples, the user cannot see all of his calculations if he has more than 1000. Moreover, the 'download whole dataset' functionality does not accept more than 5000 calculations as it is currently. In my opinion, it would be good to review some of these limits and at least to be aware at all that they are there.

  • Home page loading
    repo/old-api/topics

  • Select facets
    repo/old-api/facets?tids=58

  • Searches as an unlogged user

    • General search with facets (groups displayed)
      repo/old-api/groups?num_groups=300&tids=58
      repo/old-api/datasets?num_groups=100&&tids=58
    • additional filtering from dataset
      repo/old-api/results?limit=1000&datasets=3159373&tids=58
    • dataset display
      repo/search/datasets?query=dataset_calc_id=3159373
      repo/search/calculations_oldformat?query=dataset_calc_id=3159373&num_results=5000

  • DOI search
    repo/search/dataset_dois?num_results=1000
    repo/search/citations?num_results=1000

    • DOI internal view
      repo/search/dataset_dois?num_results=1000
      repo/search/datasets?query=dataset_doi_name=%22http%3A%2F%2Fdx.doi.org%2F10.17172%2FNOMAD%2F2017.03.17-1%22&num_results=5000
      repo/search/datasets?query=dataset_calc_id=3329405
      repo/search/calculations_oldformat?query=dataset_calc_id=3329405&num_results=5000
    • clicking the “show all dataset members“ button
      repo/search/datasets?query=dataset_calc_id=3311033
      repo/search/calculations_oldformat?query=dataset_calc_id=3311033&num_results=5000
    • DOI external view
      repo/search/dataset_dois?num_results=1000
      repo/search/calculations_oldformat?query=citation_value=%2210.1039%2Fc5cp01236h%22&num_results=5000
    • clicking the “show all dataset members“ button
      repo/search/datasets?query=dataset_calc_id=3329489
      repo/search/calculations_oldformat?query=dataset_calc_id=3329489&num_results=5000

  • As a logged-in user

    • Uploaded data
      repo/old-api/results?limit=1000&data_source=uploader,68
      repo/old-api/datasets?num_groups=100&&data_source=uploader,68
      repo/old-api/results?data_source=uploader,68
      repo/old-api/uploads?data_source=uploader,68

    • Coauthored data
      repo/old-api/results?limit=1000&data_source=coauthor,68
      repo/old-api/datasets?num_groups=100&&data_source=coauthor,68
      repo/old-api/results?data_source=uploader,68
      repo/old-api/uploads?data_source=uploader,68

    • Shared data
      repo/old-api/results?limit=1000&data_source=shared,68
      repo/old-api/datasets?num_groups=100&&data_source=shared,68
      repo/old-api/results?data_source=uploader,68
      repo/old-api/uploads?data_source=uploader,68

  • Downloads

    • Group download
      repo/search/calculations_oldformat?query=repository_grouping_checksum="g--DwDBawNvevCUO6dPyHxJd_LbY8"
    • Calculation download
      repo/search/calculations_oldformat?query=repository_calc_id=2133
    • Whole dataset download
      repo/search/calculations_oldformat?query=dataset_calc_id=3159373&num_results=5000

  1. [https://labdev-nomad.esc.rzg.mpg.de/repo/docs/index.html#/NQL_(Nomad_Query_Language)_based_search_API] ↩