Fetch And Lock Limits

Hello,
We are using Camunda with its Rest API. We are using FetchAndLock API request to pull work via topic names.

Since we are getting more and more topics in our different micro services, we decided to change our fetchAndLock logic from creating 1 worker per 1 topic to having 1 worker for all of the topics per micro service with the option of adding more instances of the same worker on each such micro service if there is high traffic on that specific micro service.
One of the reasons for that was getting near to a Camunda Limit -

But before we move to a solution where one worker can pull a (slowly growing) list of topics per micro service, I was wondering if there is a “soft” limit of the amount of topics we can send on every fetchAndLock request? So for example maybe if we send 200 topics in one such request we will begin to feel it in terms of worse performance? Because the logic in Camunda will have to check 200+ topics every time?

Hi @Ivgi,

The topic name is basically a column in the table ACT_RU_EXT_TASK. Each topic would add an additional filter clause of the external task query prior to the fetch and lock. Since it’s a simple filter directly done in the ACT_RU_EXT_TASK, I don’t see it impacting the performance dramatically. In my view, the amount of tasks that are returned for each query is more concerning than this filter.

Your solution might be a good one considering that you have a large number of topics, and those might have a small number of tasks themselves. I would always recommend you doing performance tests to make sure your design is sound.

Regards.

1 Like

Thank you for the quick response! @lucas.silva

In terms of tasks.
For every micro service I will start with 1 worker that pulls 10 tasks.
Then via performance tests I will increase the amount of workers as needed ( I dont think I will increase the amount of tasks exactly because of your concern).

I just want to mention that currently we are running very short term processes with a user who is actually waiting for the response, this is why performance is even more crucial for us.

And currently some of the services already got about 50-100 topics to pull for.
But it sounds that the filter you mentioned should indeed only have a very tiny impact on the process.