Migrate from Camunda7 external-tasks to Camunda8 job/activations

Camunda 8 Topics
1

In Camunda7 API, i can consume many topics in one API call.
But in Camunda8, i can only consume one job type at a time.

I have 100+ topics/jobs types.

In Camunda8, i will have to make many many more calls.
And i have many workers (severals dozens).
It could be something like 10-100 queries/second.

Is there some limitations (performance wise) ?
Is there a efficient way to consume job from the REST API ?

I read Reducing the Job Activation Delay in Zeebe | Camunda which made me anxious.

thank you

2

@faroni, Yeah, your concern is very valid—there is a fundamental difference in how job activation works between Camunda 7 and Camunda 8 (Zeebe). Let’s unpack it and look at what the implications are for performance and how you can mitigate them.

Camunda 7 vs. Camunda 8 - Job Fetching

Camunda 7:

  • The fetchAndLock REST API allows fetching multiple topics in a single request.
  • Very REST-friendly, good for high topic count, simple scaling.

Camunda 8 (Zeebe):
Each worker is tied to a single job type.
You can only activate jobs of one type at a time via the ActivateJobs gRPC or REST Gateway.

This model is designed to scale horizontally using gRPC rather than scaling via API call density. But it can seem more burdensome if you’re used to the batch-like efficiency of Camunda 7.

What Are the Real Limits?

Zeebe Gateway (REST/gRPC):

  • No hard limits like rate throttling on the number of ActivateJobs requests.
  • But:
    • Each request hits the Gateway.
  • The Gateway forwards it to a partition leader.
  • If requests are too frequent, and especially if no jobs are available, it can create pressure and increase activation delay.

In your case (10-100 calls/sec):

  • If most calls result in active jobs being found, you’re fine.
  • If many result in no jobs available, you’re creating load with no benefit — this is what the article you mentioned warns about.

Strategies for Efficient Job Consumption

  1. Use Long Polling (requestTimeout)
    • This is key: set a higher requestTimeout (e.g., 30s).
    • The Gateway holds the request open until a job is available or timeout is reached.
    • Reduces “empty” polling and system load dramatically.
  2. Use gRPC Instead of REST if Possible
    • REST is just a wrapper around gRPC and adds a bit of overhead.
    • If your workers are in Java or Node, prefer the native gRPC clients.
  3. Use Job Worker Libraries
    • Camunda-maintained clients (like Java client) manage polling, backpressure, and exponential backoff for you.
    • They’re far more efficient than DIY polling logic.
  4. Batch Job Activation
    • Set maxJobsToActivate > 1 to get more jobs per call.
    • If a worker can process in parallel or queue jobs, this helps.
  5. Distribute Job Types Across Workers
    • Instead of one worker per job type, you can register multiple types on a single process or thread pool.
    • This requires managing your own pooling logic, but avoids the 1:1 job type to worker constraint.
  6. Use Fewer Job Types (if you can)
    • If your 100+ types can be grouped logically, consider using metadata/variables to differentiate within a smaller number of types.

Recap: What Should You Do Now?
• Use long polling (requestTimeout: 30000) and batch activation (maxJobsToActivate: 10+).
• Avoid frequent empty polling, especially in idle periods.
• Consider gRPC clients or Camunda’s Java client instead of pure REST.
• Monitor Gateway CPU and request rates via Prometheus to detect pressure.

3

Hi @faroni and @aravindhrs,

another option that is missed in the extensive list of options is to use job streaming: Configuration | Camunda 8 Docs

The Camunda 8 engineers told me, that job streaming has less overhead than job polling in Camunda 8.

Hope that helps, Ingo

1 Like
4

Here’s a refined version of the setup for Camunda 8.6.9, using Spring Boot with version-aligned dependencies and configuration for job streaming:

1. Maven Dependency for Camunda 8.6.9

In your pom.xml, use:

<dependency>
  <groupId>io.camunda</groupId>
  <artifactId>spring-boot-starter-camunda-sdk</artifactId>
  <version>8.6.9</version>
</dependency>

2. application.yaml Configuration

camunda:
  client:
    mode: self-managed
    tenant-ids:
      - <default>
    zeebe:
      enabled: true
      base-url: http://localhost:8088
      audience: zeebe-api
      grpc-address: http://localhost:26500
      rest-address: http://localhost:8080
      execution-threads: 10
      message-time-to-live: PT2H
      request-timeout: PT20S

      prefer-rest-over-grpc: false
      defaults:
        max-job-active: 32
        stream-enabled: true
        stream-timeout: 30
      scope:
    identity:
      enabled: false
      audience: identity-api
      base-url:	http://localhost:8090

3. JobWorker Class

@Component
public class MyWorker {

  @JobWorker(type = "payment-service", name = "payment-worker", autoComplete = true)
  public void handlePaymentJob(ActivatedJob job) {
    // Your job handling logic
    System.out.println("Processing job: " + job.getKey());
  }
}

You can customize job handling with retries, manual completion/failure, etc., by injecting ZeebeClient.

4. Optional: Manual Job Completion

If you need more control:

@Autowired
private ZeebeClient zeebeClient;

@JobWorker(type = "invoice-service", autoComplete = false)
public void handleInvoice(ActivatedJob job) {
  try {
    // Your custom logic
    zeebeClient.newCompleteCommand(job.getKey())
               .variables(Map.of("status", "done"))
               .send()
               .join();
  } catch (Exception e) {
    zeebeClient.newFailCommand(job.getKey())
               .retries(job.getRetries() - 1)
               .errorMessage("Processing failed")
               .send()
               .join();
  }
}
5

It seems Job streams is useful when you have regularly jobs to consume.

Anyway, consumer Job from API is less convenient/efficient in Camunda8