Multi-instance Parallel Service Tasks Hang/Stop Consuming

Camunda 8 Topics Discussion & Questions
1

Hi Camunda Community,

We are experiencing a strange issue where parallel multi-instance Service Tasks stop being consumed by our Java worker after a few batches, leaving the remaining instances stuck forever.

Environment:

  • Zeebe Broker Version: 8.6.6 (Self-managed)

  • Zeebe Java Client: io.camunda:zeebe-client-java:8.2.0-alpha4

Original Scenario

We have a Service Task handler that invokes an asynchronous third-party API. Inside the job handler implementation, we first initiate the external service call, and then poll periodically to get the final result. Once the response is received, we call completeJob and return.

When we configure this Service Task as a parallel multi-instance (with 20 instances), only a portion of the instances complete successfully. The remaining jobs just sit in the queue and are never consumed again.

A Simple Test Case to Reproduce

To rule out external API interference, we created a minimal reproducible example with a simple job handler that just blocks the execution thread by sleeping for 60 seconds.

Worker Registration:

import io.camunda.zeebe.client.ZeebeClient;
import io.camunda.zeebe.client.api.worker.JobWorker;
import java.time.Duration;

JobWorker jobWorker = zeebeClient.newWorker()
    .jobType("TestHandler60S")
    .handler(new TestHandler60S())
    .timeout(Duration.ofDays(365))
    .maxJobsActive(5)
    .open();

Job Handler Implementation:

import io.camunda.zeebe.client.api.response.ActivatedJob;
import io.camunda.zeebe.client.api.worker.JobClient;
import io.camunda.zeebe.client.api.worker.JobHandler;
import java.util.concurrent.TimeUnit;
import lombok.extern.slf4j.Slf4j;

@Slf4j
public class TestHandler60S implements JobHandler {
    @Override
    public void handle(JobClient client, ActivatedJob job) {
        try {
            // Block the thread to simulate long-running/polling behavior
            Thread.sleep(60 * 1000); 
        } catch (Exception e) {
            log.error("testJobWorkerError", e);
        }
        
        client.newCompleteCommand(job.getKey())
            .send()
            .join(10, TimeUnit.SECONDS);
    }
}

The Unexpected Behavior

We deployed a test process containing only this single Service Task, configured as a parallel multi-instance with a cardinality of 20.

We expected the worker to process 5 jobs every 60 seconds (matching maxJobsActive=5), completing all 20 jobs in 4 batches. However, the actual polling pattern we observed was extremely weird:

  1. Batch 1: 5 jobs were processed and completed successfully.

  2. Batch 2: 3 jobs were processed and completed.

  3. Batch 3: 3 jobs were processed and completed.

  4. Batch 4: 3 jobs were processed and completed.

  5. Batch 5: 1 job was processed and completed.

  6. Afterward: Total completed = 15. The remaining 5 jobs are never picked up, and the worker stops consuming entirely.

We have already consulted AI assistants, but the typical answers regarding thread pools do not seem to fit, as we can clearly see the first 15 jobs completed successfully rather than deadlocking immediately.

Is there a flaw in how we implemented this long-running/polling handler? Or could this be a backpressure / long-polling stream timeout issue between the 8.2.0-alpha4 client and the 8.6.6 broker?

Any insights or recommendations on the best practices for handling asynchronous polling tasks in Zeebe would be highly appreciated.

Thanks in advance!

2

This appears to be a known issue with multi-instance parallel service tasks where job workers stop consuming jobs after processing some batches, particularly when using large collections and potentially exacerbated by your client/broker version mismatch. I found the following relevant resources:

Does this help? If not, can anyone from the community jump in? :waving_hand:

:light_bulb: Hints: Use the Ask AI feature in Camunda’s documentation to chat with AI and get fast help. Report bugs and features in Camuda’s GitHub issue tracker. Trust the process. :robot: