Hello All. I am new to Camunda and BPM, and trying to determine whether the technology is a good fit for my application. I am looking to orchestrate workflows that comprise tasks implemented as a collection of microservices. The workflows will contain a large number of short duration service tasks punctuated with human tasks typically for status review. Additionally, the workflows will often split into parallel task threads (100s, possibly 1000s) that will execute concurrently.
For me it is very valuable to be able to model the workflow in terms of building blocks, and make customizations based on customer feedback. For this reason and many others a BPMN-based workflow engine like Camunda is an attractive proposition.
I have been experimenting with the Camunda community edition, and have found it to be a very capable tool. My biggest concern at this point is execution overhead. I have been testing a simple workflow that uses a multi-instance subprocess to concurrently execute a small workflow across a collection of 100 instances. The tasks within the sub-workflow will execute in 3 seconds (In this simulation I control the execution time). However, I am finding that the execution time for the multi-instance workflow is 20 sec. The tasks in the subprocess workflow are tagged async_after and not exclusive. The test is somewhat contrived in that the simulated durations of the subtasks are the same, and subprocess instances will complete at close to the same time. As you can imagine I end up with a significant number of optimistic locking exceptions. From the logs it appears that most of the instance workflows complete in about 4 sec, but it takes a further 16 sec for the locking issues to be resolved, and the multi-instance subprocess to complete.
I am wondering how to tune my configuration for this situation, and whether this is the price of the the transactional integrity that the engine provides.
Any help is much appreciated. Thanks. Peter