External Task Consumption Performance Issue

Camunda 7 Topics Discussion & Questions
1

Hi everyone,

I was wondering if I could have some help trying to improve the external task consumption performance in our project. We are having what I suppose to be some concurrency issues that are affecting the performance of the external tasks consumption when we scale up our camunda pod number in our cluster. I’m gonna try to make some schemes to contextualize.

First of all, in our architecture we use camunda as a remote application (Camunda 7.15.1 but used 7.18 on these tests), not embedded, and communicate with camunda exclusively via the camunda rest api. We have a service which has the single responsibility to provide that communication with camunda (Bpm Gateway Service).

arq (1)

We handle our service tasks asynchronously, modeling the bpm processes as the following example:

image

We have simplified the way our Bpm Gateway Service handles the external tasks right now to try to avoid concurrency caused by our side of the equation and being able to test the case on the camunda side. So in this test version we have the following behaviour:

Only 1 Bpm Gateway Service pod is up and is requesting fetch and lock with a list of all topics of our application. We are using the following parameters: asyncTimeoutResponse: 30000 / lockDuration: 5000 / maxTasks: 10

Bpm Gateway Service is running the following infinite loop on its start:

while(true) {

  1. Bpm Gateway Service executes fetchAndLock request to Camunda through rest api
  2. Bpm Gateway Service completes all external tasks retrieved from Camunda through rest api
  3. The bpm process moves to the following correlate message activity waiting for the response the task was handle to move on.
  4. Bpm Gateway Service produces a message to the Service responsible to handle the external task (e.g. Service A)
    }

Asynchronously:
5. Service A consumes the rabbitMq message and does its job
6. Service A sends a rabbitMq message to Bpm Gateway Service after the handling
7. Bpm Gateway Service sends a correlate message request to Camunda through rest api
8. The process moves on

We tested the following bpm process running in a loop and made some queries to Camunda db to obtain some metrics regarding the time external tasks take to be consumed after its creation.

image
image1129×477 21.8 KB

These are the results:

  • 1 Camunda pod up:

The following query result shows the minimum, maximum and average external task consumption time (among the 200 ET looped):

image

No external task took more than a second to be consumed

  • 2 Camunda pods up with shared db:

The following query result shows the minimum, maximum and average external task consumption time (among the 200 ET looped):

image

10 of the 200 external tasks took many time to be consumed, a value close to the asyncTimeoutResponse time used. Next table shows the 10 results with consumption time over 1 second:

image
image796×290 156 KB

Do you have any ideas how to mitigate or minimize this delay?

Thank you

1 Like
2

I used to see the same result with 2 engine in cluster, when used 1 worker, subscribed to engine 1 for example.
If engine 1 is processing tasks and adds new one to external task queue, everything works fine. But if engine2 process tasks and adds new item to external task queue there is great timeout.

This is issue of dispatching new tasks to workers. As I understand it, there is some mechanism of notifying long polling workers. But it works only with a worker on engine 1. Engine2 uses tasks rereading from db, as second mechanism. It takes a lot of time.

To solve this, I used to add worker1 to engine1, worker2 to engine2. In this case everything works ok.

There is also some engine parameters to reread more often.

name=“maxJobsPerAcquisition” 50
name=“waitTimeInMillis” 5
name=“maxWait” 50
name=“backoffTimeInMillis” 30
name=“maxBackoff” 150

2 Likes
3

Hi @MaximMonin,

Thank you for your reply. However I’m a little confused concerning the way I may configure the engines to work with different workers. When you talk about workers are you refering to camunda engine workers or your app workers (pods that execute fetch and lock requests to camunda)? If you meant camunda workers how can I configure them?

1 Like
4

I am using docker compose, not cube pods:
Check:

Worker1 points to engine1, worker2 points to engine2,
This way I always have about 5-30ms delay to get next external task.
In my project minimizing process latency is main priority, so I spent a lot of time to reach it. This demo project accumulates 2 years camunda production expirience with nodejs workers, processing all tasks

2 Likes