Camunda batch process

Hi Team ,

Need some help to finalize some design approach whether to go on Camunda Batch process or Java Spring boot batch .

Requirement say : will have millons of records (as an account number list) which we will be trigger via batch on every 24 hrs , so each account record will process with BPMN flow .

Question :
1:Is any provision in camunda batch configuration which splits the list of records across instances in cases of multiple workflows executing (means more then one account workflow microservice running)
Example : let say need to run batch for 2000 records , how this 2000 records will accommodate among multiple work flow instances ?

2:As per my understanding , camunda runs single instance of workflow per one active Job executor .So in this case how the load distribution happen. ?

3:And also how failure or restarting instance works ,when any workflow instance fails ?
How camunda batch automatically fail over for in case of heterogeneous clusters ?

Any help will be much appreciated

Thanks
Prasanjit

HI @prasanjit.deka

I don’t think i fully understand some of the requirements you’re discussing but i’ll try to answer your questions best I can. Please feel free to give me clarifications if I’m on the wrong track.

Generally it just means that you’ll start 2000 instances of a specific definition these by default will run in parallel but you if you want to run them sequentially you can create a parent process which manages the queue

This isn’t the case, a job executor can be running multiple jobs for multiple process instances in parallel. The limitation is that for a single process instance only one thread is active in order to preserve a consistent state.

Depending on how complicated the errors and how complicated the rollback you can either use BPMN to model an asyc rollback using compensation events or you can just let the engine roll back the thread itself when it encounters an error. It depends on the details. There are a lot of options here and there’s bound to be a solution regarding error handling.

1 Like

Thanks Naill for your prominate explanation ,

Based on point 1 : Requirement says , scheduler pick millions of record and process with some definition batch by batch . Now running the batch Service into multiple PODS , then how its balance the load among multiple PODs .

Is there any way in camunda batch configuration will maintain the load balancing based on batch load size ?