Multiple External Task Client instances

Camunda 7 Topics Discussion & Questions
1

Hi everyone,

I am working with the ExternalTask Client pattern and encountered a problem.

The situation is the following:

I have this example process. In the first task I simply print “start” to the console and in the last task I print “end” to the console, so I can check when the process arrives in which step. The task in the middle should be executed with an ExternalTask Client.

external1
external1756×270 2.89 KB

The code for the ExternalTask client is shown in the following picture. The code prints the ID of the processInstance, sleeps for a few seconds and prints the ID again.

external2
external2785×412 10.1 KB

Now when I start three instances of this process in the Camunda Tasklist all three processes arrive at the middle Task and wait to be completed. If I now execute the main method of this ExternTask client two times (to get two instances of the client) there seems to be a problem

The first instance of the ExternalTask client prints the following:

2019-02-05 10:39:59,662 [INFO ] ExternalTaskExample - start
2019-02-05 10:40:02,008 [INFO ] ExternalTaskExample - end
2019-02-05 10:40:02,572 [DEBUG] ExternalTaskExample - [f5b68cc6-2929-11e9-919d-98b18b5b5347] start
2019-02-05 10:40:10,574 [DEBUG] ExternalTaskExample - [f5b68cc6-2929-11e9-919d-98b18b5b5347] end
2019-02-05 10:40:10,712 [DEBUG] ExternalTaskExample - [f7e309bd-2929-11e9-919d-98b18b5b5347] start
2019-02-05 10:40:18,712 [DEBUG] ExternalTaskExample - [f7e309bd-2929-11e9-919d-98b18b5b5347] end
2019-02-05 10:40:18,791 [DEBUG] ExternalTaskExample - [fa5d7ea4-2929-11e9-919d-98b18b5b5347] start
2019-02-05 10:40:26,792 [DEBUG] ExternalTaskExample - [fa5d7ea4-2929-11e9-919d-98b18b5b5347] end
2019-02-05 10:40:26,812 [ERROR] client - TASK/CLIENT-03004 Exception on external task service method invocation for topic ‘external-test’:
org.camunda.bpm.client.exception.NotAcquiredException: TASK/CLIENT-01007 Exception while completing the external task: The task’s most recent lock could not be acquired

and the second instance prints this:

2019-02-05 10:40:02,438 [INFO ] ExternalTaskExample - start
2019-02-05 10:40:03,951 [INFO ] ExternalTaskExample - end
2019-02-05 10:40:24,301 [DEBUG] ExternalTaskExample - [fa5d7ea4-2929-11e9-919d-98b18b5b5347] start
2019-02-05 10:40:32,304 [DEBUG] ExternalTaskExample - [fa5d7ea4-2929-11e9-919d-98b18b5b5347] end

This behaviour shows that the process instance with the id “fa5d7ea4-2929-11e9-919d-98b18b5b5347” is processed two times. I am wondering why this happens.

On the other hand, the last task in the process that only prints “end” to the console is executed three times (as it should be) but I also get the following error:

External Task fa64d1a9-2929-11e9-919d-98b18b5b5347 cannot be completed by worker ‘062a2cd5-90a7-4ec8-966c-d2d2b9810dba’. It is locked by worker ‘2a23073d-d746-49b9-9035-cb2056585a17’.

I hope someone can explain what the problem with this example is. Is there anything that needs to be configured for starting the ExternalTask client multiple times? Thanks in advance.

Regards
Michael

2

Hi @MichiDahm,

this is expected behavior as you fetch up to 10 tasks. They will be handled sequentially in their handler. Each task takes 8 seconds and some milliseconds. So the 3rd task is unlocked before completion (locktime is 18 seconds) and is picked up in the fetch again.

My proposal to use maxTasks=1 and lockTime long enough to complete the task.

Then you can scale up starting additional workers for your topic.

Hope this helps, Ingo

2 Likes
3

How scaling can be achieved on external task client workers?

Running Multiple ExternalTaskclients(lets say instance1, instance2, instance-n) to fetch single task on same topic by every ExternalTaskclients instances?

or

One externalTaskclient implementation of multithreaded/CompletableFuture execution, thus allowing each thread to fetch single task on same topic?

4

Hi @aravindhrs,

in my windows laptop I start another cmd.exe and run the next worker with java -jar externalTaskWorker.jar ... It’s your option 1.

For production you can use docker and k8s or ansible or another tool of your choice.

As I run into troubles with multi threading several times I would avoid it and let the operation system do the load distribution.

Cheers, Ingo

1 Like
5

Hi @Ingo_Richtsmeier,

thanks for the fast response. When setting maxTasks to 1 the example works perfectly fine. I can now start the ExternalTask client multiple times without problems.

Thanks

Regards
Michael

6

Hi @MichiDahm,

do a mvn clean package once and run java -jar myExternalClient.jar ... in multiple terminal windows, for example.

Maybe set the workerId for each start to a unique value from a command line option.

Cheers, Ingo

2 Likes
7

Hi @Ingo_Richtsmeier,

I have another question regarding the load balancing with multiple ExternalTask clients. So in the Rest interface there is a method “Get External Task Count” to find out how many task are waiting be be completed.

But is there also a way to automatically tell the engine when to start and stop the single ExternalTask client instances. Otherwise I would need to query the number of external tasks on my own and implement the load balancing logic by my own.

Regards Michael

8

Hi @MichiDahm,

there is no way to make the engine to start additional workers out of the box. Both should be independent services.

The counter may be a help to scale the workers, but I have no experience in this field of operation. I assume that the modern elastic environments (kubernetes, docker swarm, …) support these uses cases.

Hope this helps, Ingo