I am modelling a process which at times will require a very large number of parallel sub-processes (tens of thousands) to be launched. Obviously it’s not possible for these all to run in parallel simultaneously - how will the Camunda process engine handle this? Is it possible to control how many subprocesses will run at a time?
For this particular use case, I’d suggest using Camunda 8 if you can sign up here and it is kind of designed for this kind of use case.
4 Likes
@Niall Thanks, that’s quite a big change as we already use Camunda 7 extensively. I wonder if it would be covered by our existing Enterprise license (probably not)? I’ll dig into it a bit further but in the meantime is there any way to control the number of concurrent processes in this situation in Camunda 7?
Hi @Ettery,
one way to control the amout of parallel multiinstance would be nesting. Maybe a small amount of sequential multiinstance subprocesses start a bigger bunch of parallel multiinstance tasks?
Hope this helps, Ingo
Just to clarify, if I have 10 000 subprocesses and want to run 10 in parallel at a time, I could have a parallel multiinstance with 10 subprocesses, each of which has a sequential multiinstance with 1000 subprocesses?
What do you mean exactly when you write “run in parallel”? If you have an async point somewhere in the subprocess then all of them will be started but only few will be really executed (in the sense that an activity is being executed by the CPU). You can then control the numbers via the job executor settings.
@fml2 That sounds exactly like what I’m trying to understand - so of all the parallel processes queued, only a few will run at a time - is there documentation about controlling this?
Yes, this is controlled by the job executor settings.
Hi @Ettery,
Regarding your question whether your existing Camunda 7 Enterprise Agreement covers Camunda 8:
I wonder if it would be covered by our existing Enterprise license (probably not)?
The answer is “Yes”! given that Camunda 8 is simply the new version of the product that you already bought as a customer, you have access to it under your existing contract. Note that this covers Camunda 8 Self Managed only, if you want to use the SaaS version, am additional agreement is necessary.
I hope that answers that part of your question.
Best,
Daniel
1 Like
Attacking this from a different angle; we run our Camunda engines embedded in containers in Kubernetes. Rather than allocating a container lots of CPU and fiddling with the job executor settings, it would make sense for us to scale horizontally, launching multiple containers with the Camunda engine embedded, each of which can process from the queue. Is this possible and if so is it advisable?
I think this would be a viable approach if you want to increase the number of jobs executed in parallel. But you’ll still have to configure the job executor properly. Also, you have to have a DB that is able to serve that many connections. So you still might reach the limit pretty soon.
Zeebe supposedly does not have the DB limitation and allows for unbounded horizontal scalability. But I have not tried it myself yet.
1 Like