We have designed a camunda BPMN from which parent workflow makes a call to sub-workflow using BPMN “Call Activity” object and submits a list of tasks to the sub-workflow as “In-mapping propagation”.
Inside the sub-workflow we make use of “Parallel Multi Instance” sub process and it uses the same ${tasks} Collection. The size of this ${tasks} Collection may go up to 1,00,000 (1 lac).
We need to understand how many of these tasks would run in parallel and is there a way to configure it somewhere using some configuration? How the camunda process engine will handle this situation? What kind of resources (vCPU and memory) are expected to be provided to this camunda process?
Note: We are using Camunda Platform Engine 7 and it’s deployed K8s environment and running inside a POD.
Looking forward to your response and insights on this use-case.
Thanks @jonathan.lukas for your immediate response.
Do you have any reference BPMN diagram that you can share that will help us understand better about your recommendation - to page the entries at your desired size, run the pages in a sequence and inside, the page in parallel.
Wouldn’t this load arise only at the moment when all the sub-instances are created? After that, the instances would all be processed by the usual job executors thus having a bound on the number of really executed threads?
As strategy to control number of elements in parallel, I prefer to divide collection to chunks, for example 100 chunks of 100 elements, setup double subprocess, 1 level iterates as sequence with chunks, 2 level Parallel Multi Instance with chunk size.
Hey @Ishan_Aggarwal!
Sorry for the long delay in answering to your post. I have been on travels and did not manage to get back to open forum threads.
Below you can find the BPMN file we have used to iterate over a list of a million entries. Note, this has been done in Camunda 8. So you might want to convert it to a C7 BPMN model. extreme-multi-instance.bpmn (8.0 KB)
