We have a main bpmn file which has 3 call activities. There will be millions of process instances (say a million ) created on main which will in turn 3 more million process instances.
It will definitely be appealing visually and also modularize the bpmn workflows. But, I would like to understand latency of stream processor and downstream impacts in creating so many process instances.
Is my understanding correct on process instances?
Please let me know what you recommend. We want to reduce latency.
Thanks
Janardhan.
no,
we have events coming to kakfa. A kafka consumer will publish messages to camunda. In 1 hr, we can expect to publish 1 to 2M messages. so it is spread across 1 hr.
Hi @JG2023 - that is quite a lot of process instances in an hour. I think the answer to this is to try to properly size and monitor your environment to find the ideal balance. There isn’t a simple answer. There are many variables: how many are created concurrently, how long does each take to run, do you see that load every hour of every day or are there peak times, etc.
Thanks.
Going back to the original question, will the latency improve if I reduce number of call activities as each call activity is going to create a process instance?
When message is published to zeebe, it throttles it, is there a way to improve this and what metrics to look at? I am guessing first step is to look at stream processor metrics.
Hi @JG2023 - the latency between a call activity and any other type of task should be roughly the same. Any performance impact from call activities will likely be from copying process instance variables over.
What message throttling are you referring to? If you publish a message to Zeebe, there is a buffer, but I don’t think that’s the same as what an inbound Kafka Connector would do.