Sandeep Gupta: Hello Community,
We have built a workflow with 8 service tasks. While testing, we see that it takes around 1 second to execute all of them. Each individual task takes only 5-10 msec. Our use case is to implement near real time work flow, is it possible to reduce time between next task is triggered after previous one finishes? We are using pyzeebe to implement the tasks and zeebe 1.3.1
Nicolas: As your workers are external processes, if you have 8 sequential tasks, you end up doing the following:
• Process is created. First task is created
• Worker polls for first task
• Task is completed. Second task is created
• Worker polls for second task
• Task is completed. Third task is created
• …
Between each step, there is a little bit of time. And between each step, there’s at least one request which goes out, potentially two (if the gateway must re-route it). So even if you have a very fast network, you still have to add a bit of time for the communication as well as the processing. So when it comes to straight-through processing, there’s an unfortunate delay here.
However, the nice part is, you could be working on many workflow instances at the same time, and unless you overload your brokers, the time would remain more or less the same, letting you increase throughput.
To decrease the cycle latency, however, you would need to model your process a bit differently.
Sandeep Gupta: Thanks <@UAHNVBW8N>, I understood your reply but what is your suggestion when you say that process should be modeled a bit differently? Do you mean all tasks should be merged into one?
Nicolas: Having less sequential steps will improve the latency overall. Could you maybe make some of them parallel?
Sandeep Gupta: Unfortunately, no. These are sequential only.
zell: I assume you using a self-hosted zeebe version? What you could try is to also increase the resources each broker uses, like cpu, memory, threads etc. and play a bit around with that. Make sure you use ssd’s and that your system is not io-throttled.
Sandeep Gupta: Correct, we are using self hosted version and have deployed on a kubernetes cluster. Will try to implement your suggestions and see the difference in performance.
Note: This post was generated by Slack Archivist from a conversation in the Zeebe Slack, a source of valuable discussions on Zeebe (get an invite). Someone in the Slack thought this was worth sharing!
If this post answered a question for you, hit the Like button - we use that to assess which posts to put into docs.