External Workers Scalability

Hi!
Is there any way to avoid / significantly decrease number of OptimisticLockingExceptions in case of multiple External Workers fetching tasks of the same topic in parallel? Tasks can’t be split between multiple topics (suppose its the same tasks in a single process template in highly-loaded system).

Is it possible to at least make Camunda process OLE inside polling interval?

For now it looks like External Workers can’t be scaled effectively due to massive errors during fetchAndLock (and complete operations too).

*We use https://github.com/camunda/camunda-external-task-client-java.

Hi @JustMormegil,

have a look at this post.
Maybe you can try to set maxTasks=1. That should help you with decreasing the number of exceptions.

Regards
Michael

Hi @MichiDahm, assuming each task is executed very fast, won’t that bring us constant concurrency locks from permanent fetch&lock request spam?

Hi @JustMormegil,

as far as I undesrstood, it is best practice to set maxTasks to 1.
The fetch and lock requests depend on the interval that you specify in which you want to poll. So I guess the engine can handle the parallel polls of multiple externaltasks on it’s own and the locks shouldn’t be a problem for your application.

1 Like

Thanks for advice, will check Camunda behavior for maxTasks = 1.

Regarding second part of the question: is there way to make Camunda autoretry Optimistic locking exceptions which appear inside fetchAndLock & complete external tasks methods?

I see no point in returning control back to requestor as it leads to additional retry implementation on External Workers side for all external workers.

1 Like