Parallelize Temporal batch processing and add concurrency controls

  • Refactored BatchProcessEntriesWorkflow to use parallel child workflows instead of sequential continue_as_new iterations.
  • Implemented asyncio.Semaphore and gather to process multiple batches concurrently, eliminating the bottleneck of waiting for one batch to fully finish before starting the next.
  • Improved task queue stability by capping peak load at 500 concurrent activities (5 batches × 100 entries), smoothing out the previous 1,000-entry bursts.
Edited by Ahmed Ilyas

Merge request reports

Loading