Scaling External Task Handlers

Camunda 7 Topics
1

Using Open Source Camunda.

When we try to scale the external task handlers by marking them as @Async (Springboot) and configuring ThreadPoolTaskExecutor with n threads… we notice that the same task gets processed multiple times. We also have a cluster of Camunda engines against the same database.

The process instances get triggered by an S3 event when a file(s) get uploaded. So not sure if we need to queue the responses before we call the Camunda start process endpoint.

We see no evidence that the task level locking is working.

When we scale back to single instance, sync task handlers, it works just fine but its slow.

What is the optimal scaling strategy for such a use case?

2

Hi @Aniruddha_Raje1

You can scale external task workers by launching multiple instances, each assigned a unique worker ID, to ensure that tasks locked by one worker cannot be fetched by others.

2 Likes
3

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.