Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • nomad-FAIR nomad-FAIR
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 219
    • Issues 219
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 32
    • Merge requests 32
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Monitor
    • Monitor
    • Incidents
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • nomad-labnomad-lab
  • nomad-FAIRnomad-FAIR
  • Merge requests
  • !479

Refactor proc framework

  • Review changes

  • Download
  • Email patches
  • Plain diff
Merged David Sikter requested to merge refactor-proc-framework into v1.0.0 Dec 10, 2021
  • Overview 0
  • Commits 3
  • Pipelines 1
  • Changes 6

Major refactor of the proc framework. The join functionality is lifted up to the base class and generalized. By marking a process with is_child = True, you tell the framework that we should try to join with (resume) the parent Proc (obtained by calling self.parent on the child Proc) when he child process is done. When the join succeeds (when all children are done), the parent is put to RUNNING (previously it remained in status WAITING_FOR_RESULT) and the new join method is called. The Proc may from there go back to status WAITING_FOR_RESULT again (so that we can implement the idea of "slotted" parsing, see #668 (closed)).

Further, queuing is made possible (#667 (closed)). I.e. when a process is running, it may be possible to invoke another process on the same object. The framework should ensure that the processes are queued up and executed one at a time in sequence. By setting is_blocking = True, you can ensure that certain processes "block" additional queuing until they are done, as it does not make sense/seem safe to queue up calls in every case. I have marked delete_upload, publish_upload, and publish_externally as "blocking". The queue is persisted to the mongodb object. Measures are taken to ensure that scheduling, starting and completing a process are atomic operations.

Edited Dec 13, 2021 by David Sikter
Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: refactor-proc-framework