Scaling External task workers independently

Camunda 7 Topics Discussion & Questions
1

What is they best way to setup external task client(java) and workers?

Scenario

  • We have 5 client applications(microservices) which has its own process definitions.
  • Each process definition will have around 10+ different topics (taskType=external).

  1. How to setup the workers for each client applications and to scale the workers independently?

  2. Can we create one microservice and create all workers in that microservice itself to handle all the client applications request?

  3. If we embed the external task client in microservice, how can we make workers keep executing or initalizing?

2

Hi @aravindhrs,

there is no best way for this setup. As one worker can subscribe to many topics, you can use all variants between a single worker for all topics and one worker (maybe started multiple times) for each topic.

I would consider:

  • The dependencies of the business logic: What belongs into a single bounded context?
    Have a look into the Domain Driven Design space to cut your business desing into independent pieces.
  • How long will it take to complete each single service call and how many requests do I expect?
    This may give you a number of instances you need to start for each worker.

Technically you can wrap each worker/client application as a spring boot application and execute and control them as all the other microservices around (ansible, kubernetes, …)

Hope this helps, Ingo

1 Like
3

Thanks @Ingo_Richtsmeier it’s a nice solution :slight_smile:

4

Have a followup question here…
How does each remote microservice set up a pool of workers that can be started up during application startup. Lets say we have a pool of “ni” instances of ExternaltaskClient at startup subscribing to the same topic , does it mean that each can process 5 external tasks. These tasks are defined within a multi instance parallel sub-process where potentially 1000’s of external tasks can pile up on the workflow and we want to able to process these as quickly as we can.

How do we set external task clients for such cases…Am looking at something similar to a Sppring JmsTemplate we use to set up topic subscribers etc

1 Like
5