Skip to content
Snippets Groups Projects

Refactor proc framework

Merged David Sikter requested to merge refactor-proc-framework into v1.0.0

Major refactor of the proc framework. The join functionality is lifted up to the base class and generalized. By marking a process with is_child = True, you tell the framework that we should try to join with (resume) the parent Proc (obtained by calling self.parent on the child Proc) when he child process is done. When the join succeeds (when all children are done), the parent is put to RUNNING (previously it remained in status WAITING_FOR_RESULT) and the new join method is called. The Proc may from there go back to status WAITING_FOR_RESULT again (so that we can implement the idea of "slotted" parsing, see #668 (closed)).

Further, queuing is made possible (#667 (closed)). I.e. when a process is running, it may be possible to invoke another process on the same object. The framework should ensure that the processes are queued up and executed one at a time in sequence. By setting is_blocking = True, you can ensure that certain processes "block" additional queuing until they are done, as it does not make sense/seem safe to queue up calls in every case. I have marked delete_upload, publish_upload, and publish_externally as "blocking". The queue is persisted to the mongodb object. Measures are taken to ensure that scheduling, starting and completing a process are atomic operations.

Edited by David Sikter

Merge request reports

Pipeline #117533 passed

Pipeline passed for f6ee9529 on refactor-proc-framework

Test coverage 79.00% (0.00%) from 1 job
Approval is optional

Merged by Markus ScheidgenMarkus Scheidgen 3 years ago (Dec 13, 2021 11:40am UTC)

Merge details

  • Changes merged into v1.0.0 with 18b1ba4e.
  • Deleted the source branch.

Pipeline #117646 passed

Pipeline passed for 18b1ba4e on v1.0.0

Test coverage 79.00% (0.00%) from 1 job

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
Please register or sign in to reply