Race Conditioning two workers for a job

Camunda 8 Topics
1

Hi,

I am simulating a scenario where there are two workers polling for a same job type. In one of the worker code I have Thread.sleep(<time_greater_than_time_out>) so that timeout would happen and Zeebe would re-assign the job to the second worker. I see two kind of behaviors depending on no. of instances.
image

image

1. When I create one instance in process
When the instance reaches the service task, if the worker one acquires the job due to Thread.sleep() it would go for sleep and Zeebe would re-assign to second worker, this second worker would complete the job and first worker would come out of sleep and throw NOT_FOUND exception. This is the expected behavior as per [document](https://docs.camunda.io/docs/components/concepts/job-workers/#timeouts)

image
image1920×157 14.3 KB

2. When I create two instances (at the same time) in process
For the first instance I can see the above behavior , but for the second instance the primary worker which has Thread.sleep() sleeps for 20s and completes the job, I don’t see zeebe re-assigning the work to second worker in this case.

image
image1146×65 28.7 KB

What would be the reason for this behavior ?

2

Hi @durairajsekar,

before a worker gets a job, he sends an activateJobs command to the broker to get available jobs. (Zeebe API (gRPC) | Camunda 8 Docs) This command gets a parameter maxJobsToActivate. Did you configure this parameter in your example? The default value is 32: https://github.com/camunda/zeebe/blob/ef16ef3aefea697eace8d3d232f2b2c3ad0e5bc2/clients/java/src/main/java/io/camunda/zeebe/client/impl/ZeebeClientBuilderImpl.java#L69

But as there is only 1 execution thread by default, I think that all jobs are executed in a loop in the client.

Maybe you can enhance your experiment with more output on the worker (for example the job- and processInstanceKey) to check this and let us know further results?

Hope this helps, Ingo

3

@durairajsekar I don’t know if its a best practice to have a job type trying to activate twice because the broker randomly choose and push the message. Instead, you can modify the model and handle through process flow and have two distinct job types.
Hope this helps!