Skip to content
GitLab
Projects Groups Topics Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • nomad-FAIR nomad-FAIR
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributor statistics
    • Graph
    • Compare revisions
  • Issues 244
    • Issues 244
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 45
    • Merge requests 45
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar

On Monday, June 12, 2023, from 9.00 to 10.00 am there will be a maintenance with a short downtime of the GitLab service.

  • nomad-labnomad-lab
  • nomad-FAIRnomad-FAIR
  • Merge requests
  • !564

Resolve "In-api processing of simple and individual entries"

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged David Sikter requested to merge 745-in-api-processing-of-simple-and-individual-entries into v1.1.0 Feb 17, 2022
  • Overview 3
  • Commits 4
  • Pipelines 4
  • Changes 8

Closes #745 (closed).

The endpoint for PUT uploads/{upload_id}/raw/{path} is extended with two query arguments: wait_for_processing and include_archive. When specifying wait_for_processing, the upload and processing is run locally and the endpoint call blocks until the processing is complete (but only the uploaded file is matched and (if a parser matches) processed, nothing else), and we return an additional processing key in the response. If include_archive is specified, the archive is also included in the response.

The response now has the following format:

{
  "upload_id": ...,
  "data": <upload Proc data>
  "processing": {
    "upload_id": ...,
    "path": <full raw path to the uploaded file>,
    "entry_id": ...,
    "parser_name": ...,
    "entry": <entry proc data>
    "archive": <archive data>
  }
}

Which is identical to before except for the addition of the processing key. The processing key will be null if not specifying wait_for_processing. If we do specify wait_for_processing, but the file does not match any parser, the entry-related values inside the processing dict will be all null.

I introduced a new decorator @process_local, which was maybe a bit ambitious, but I think it was the best way to do it to get a consistent handling of the process attributes, like process_status, sync_counter etc, and unified error boundary. The local process cannot be started (i.e. the endpoint will fail) if something else is running, and nothing else can be run or scheduled to run while a local process is running.

Edited Feb 18, 2022 by David Sikter
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: 745-in-api-processing-of-simple-and-individual-entries