Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • nomad-FAIR nomad-FAIR
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 115
    • Issues 115
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 13
    • Merge requests 13
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Package Registry
    • Container Registry
    • Infrastructure Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • nomad-lab
  • nomad-FAIRnomad-FAIR
  • Merge requests
  • !479

Merged
Created Dec 10, 2021 by David Sikter@dsikterDeveloper

Refactor proc framework

  • Overview 0
  • Commits 3
  • Pipelines 1
  • Changes 6

Major refactor of the proc framework. The join functionality is lifted up to the base class and generalized. By marking a process with is_child = True, you tell the framework that we should try to join with (resume) the parent Proc (obtained by calling self.parent on the child Proc) when he child process is done. When the join succeeds (when all children are done), the parent is put to RUNNING (previously it remained in status WAITING_FOR_RESULT) and the new join method is called. The Proc may from there go back to status WAITING_FOR_RESULT again (so that we can implement the idea of "slotted" parsing, see #668 (closed)).

Further, queuing is made possible (#667 (closed)). I.e. when a process is running, it may be possible to invoke another process on the same object. The framework should ensure that the processes are queued up and executed one at a time in sequence. By setting is_blocking = True, you can ensure that certain processes "block" additional queuing until they are done, as it does not make sense/seem safe to queue up calls in every case. I have marked delete_upload, publish_upload, and publish_externally as "blocking". The queue is persisted to the mongodb object. Measures are taken to ensure that scheduling, starting and completing a process are atomic operations.

Edited Dec 13, 2021 by David Sikter
Assignee
Assign to
Reviewer
Request review from
Time tracking
Source branch: refactor-proc-framework