Camunda 8.8/ 8.9 deadline issue

Camunda 8 Topics Discussion & Questions
, ,
1

Hello Devs,

I recently shifted from Camunda 8.7 to Camunda 8.8 / 8.9 and I am facing an issue while debugging my Java code locally.

I am using only `c8run` for running Camunda 8 locally. I do not currently have the option to use Docker or Camunda SaaS.

The issue is that while debugging the Java worker/client code, after around 1–2 runs I often start getting `DEADLINE_EXCEEDED` errors. Because of this, I can hardly debug the code properly for more than around 30 minutes.

This was not happening, or at least was much less frequent, when I was using Camunda 8.7.

My setup is roughly:

- Camunda version: 8.8 / 8.9

- Local runtime: `c8run`

- Client/worker: Java

- Environment: local machine, no Docker, no SaaS

- Error: `DEADLINE_EXCEEDED`

- Scenario: happens during debugging, especially when breakpoints pause execution for some time

Has anyone faced a similar issue after moving from 8.7 to 8.8 or 8.9?

I would really appreciate suggestions on how to avoid or reduce this issue while using only `c8run`. For example:

- Are there any timeout settings I should increase for local debugging?

- Is there any recommended configuration for Java workers when using breakpoints?

- Are there any known differences in 8.8 / 8.9 compared to 8.7 that could cause this?

- Is there a better way to run/debug locally with `c8run`?

Any guidance or workaround would be very helpful.

Thanks in advance!

2

This appears to be related to timeout configuration changes and the shift to REST as default protocol in Camunda 8.8+, which can cause DEADLINE_EXCEEDED errors during debugging with breakpoints. I found the following relevant resources:

Does this help? If not, can anyone from the community jump in? :waving_hand:

:light_bulb: Hints: Use the Ask AI feature in Camunda’s documentation to chat with AI and get fast help. Report bugs and features in Camuda’s GitHub issue tracker. Trust the process. :robot:

3

Hi @Bishal072 ,

Problem Summary

After upgrading from Camunda 8.7 to 8.8/8.9, DEADLINE_EXCEEDED errors occur frequently when debugging Java worker/client code locally using c8run. This happens especially when breakpoints pause execution, making debugging sessions barely last 30 minutes.

Root Cause

In Camunda 8.8+, several timeout defaults were tightened and the gRPC gateway behavior changed:

  1. Zeebe Gateway request timeout was reduced in newer versions

  2. Job worker lock duration defaults are shorter

  3. gRPC keep-alive and deadline settings are more aggressive

  4. Long poll timeout for job workers can conflict with breakpoint pauses

When you hit a breakpoint, the JVM pauses entirely — including the job worker’s internal threads that handle heartbeats, polling, and gRPC stream management. This causes the gateway to consider the connection dead and return DEADLINE_EXCEEDED.

Solution 1: Increase Java Client Timeouts (Recommended)

Zeebe Client Configuration

import io.camunda.zeebe.client.ZeebeClient;
import java.time.Duration;

ZeebeClient client = ZeebeClient.newClientBuilder()
    .gatewayAddress("localhost:26500")
    .usePlaintext()
    // Increase the default request timeout (default is 20s in 8.8+)
    .defaultRequestTimeout(Duration.ofMinutes(10))
    // Increase gRPC message size if needed
    .maxMessageSize(4 * 1024 * 1024)
    .build();

Job Worker Configuration

import io.camunda.zeebe.client.api.worker.JobWorker;
import java.time.Duration;

JobWorker worker = client.newWorkerBuilder()
    .jobType("my-job-type")
    .handler(jobHandler)
    // Increase job timeout so Zeebe won't reassign while you're on a breakpoint
    .timeout(Duration.ofMinutes(30))
    // Increase poll interval to reduce frequency of deadline-sensitive calls
    .pollInterval(Duration.ofSeconds(30))
    // Request fewer jobs at a time during debugging
    .maxJobsActive(1)
    // Increase request timeout for this worker
    .requestTimeout(Duration.ofMinutes(10))
    .open();

Spring Boot / application.yaml Configuration

If using Spring Boot with Camunda 8 starter:

camunda:
  client:
    zeebe:
      defaults:
        request-timeout: PT10M
      override:
        request-timeout: PT10M
    worker:
      defaults:
        timeout: PT30M
        poll-interval: PT30S
        request-timeout: PT10M
      override:
        timeout: PT30M

zeebe:
  client:
    broker:
      gatewayAddress: localhost:26500
    security:
      plaintext: true
    requestTimeout: PT10M
    worker:
      defaultJobTimeout: PT30M
      defaultJobWorkerMaxJobsActive: 1
      defaultJobPollInterval: PT30S

application.properties Alternative

# Zeebe client timeout settings for debugging
camunda.client.zeebe.defaults.request-timeout=PT10M
camunda.client.worker.defaults.timeout=PT30M
camunda.client.worker.defaults.poll-interval=PT30S
camunda.client.worker.defaults.request-timeout=PT10M

# Legacy property names (for older Spring starters)
zeebe.client.requestTimeout=600000
zeebe.client.worker.defaultJobTimeout=1800000
zeebe.client.worker.defaultJobWorkerMaxJobsActive=1

Solution 2: Configure c8run Gateway Timeouts

Modify Zeebe Gateway Configuration

Locate or create a configuration file for your c8run instance. Typically you can set environment variables or modify application.yaml in the c8run directory.

Environment Variables (set before running c8run)

REM Windows cmd.exe
set ZEEBE_GATEWAY_LONGPOLLING_TIMEOUT=600000
set ZEEBE_GATEWAY_REQUEST_TIMEOUT=600s
set ZEEBE_BROKER_GATEWAY_LONGPOLLING_TIMEOUT=600000
set ZEEBE_GATEWAY_GRPC_MAXMESSAGESIZE=4194304
set ZEEBE_GATEWAY_KEEPALIVE_INTERVAL=300s
c8run.exe start


# PowerShell
$env:ZEEBE_GATEWAY_LONGPOLLING_TIMEOUT = "600000"
$env:ZEEBE_GATEWAY_REQUEST_TIMEOUT = "600s"
$env:ZEEBE_BROKER_GATEWAY_LONGPOLLING_TIMEOUT = "600000"
$env:ZEEBE_GATEWAY_GRPC_MAXMESSAGESIZE = "4194304"
$env:ZEEBE_GATEWAY_KEEPALIVE_INTERVAL = "300s"
.\c8run.exe start

c8run YAML Configuration Override

If c8run supports a custom config file (check your version’s docs), create or modify config/application.yaml:

zeebe:
  gateway:
    longPolling:
      timeout: 600000  # 10 minutes in ms
    request:
      timeout: 600s
    grpc:
      maxMessageSize: 4MB
    keepAlive:
      interval: 300s
  broker:
    gateway:
      longPolling:
        timeout: 600000

Solution 3: Create a Debug-Specific Profile

Create a separate Spring profile for debugging that maximizes all timeouts:

application-debug.yaml

camunda:
  client:
    zeebe:
      defaults:
        request-timeout: PT30M
      override:
        request-timeout: PT30M
    worker:
      defaults:
        timeout: PT60M
        poll-interval: PT60S
        request-timeout: PT30M
        max-jobs-active: 1
      override:
        timeout: PT60M

logging:
  level:
    io.camunda: DEBUG
    io.grpc: DEBUG

Run with:

java -Dspring.profiles.active=debug -jar your-app.jar

Or in your IDE run configuration, add VM argument: -Dspring.profiles.active=debug

Solution 4: gRPC Channel Configuration (Advanced)

For fine-grained control over gRPC behavior:

import io.camunda.zeebe.client.ZeebeClient;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import java.time.Duration;
import java.util.concurrent.TimeUnit;

// Build a custom gRPC channel with relaxed timeouts
ManagedChannel channel = ManagedChannelBuilder.forAddress("localhost", 26500)
    .usePlaintext()
    .keepAliveTime(5, TimeUnit.MINUTES)
    .keepAliveTimeout(10, TimeUnit.MINUTES)
    .keepAliveWithoutCalls(true)
    .idleTimeout(60, TimeUnit.MINUTES)
    .maxInboundMessageSize(4 * 1024 * 1024)
    .build();

// Use the custom channel (if your client version supports it)
ZeebeClient client = ZeebeClient.newClientBuilder()
    .gatewayAddress("localhost:26500")
    .usePlaintext()
    .defaultRequestTimeout(Duration.ofMinutes(30))
    .build();

Solution 5: Job Completion Pattern for Debugging

Instead of having the worker auto-complete the job inside the handler (which times out while you’re on a breakpoint), consider a manual completion pattern:

import io.camunda.zeebe.client.api.response.ActivatedJob;
import io.camunda.zeebe.client.api.worker.JobClient;
import io.camunda.zeebe.client.api.worker.JobHandler;

public class DebugFriendlyHandler implements JobHandler {
    
    @Override
    public void handle(JobClient client, ActivatedJob job) throws Exception {
        // Set a very long deadline for this specific job completion
        // This gives you time to step through with breakpoints
        
        System.out.println("Processing job: " + job.getKey());
        System.out.println("Variables: " + job.getVariables());
        
        // Your business logic here - set breakpoints below this line
        Object result = doBusinessLogic(job);
        
        // Complete with extended timeout
        client.newCompleteCommand(job.getKey())
            .variables(Map.of("result", result))
            .requestTimeout(Duration.ofMinutes(10))
            .send()
            .join();
    }
    
    private Object doBusinessLogic(ActivatedJob job) {
        // <-- Set your breakpoints here
        return "processed";
    }
}

Solution 6: Workaround — Use StreamEnabled=false

Camunda 8.8+ introduced streaming for job workers by default. Disabling it can help with debugging:

JobWorker worker = client.newWorkerBuilder()
    .jobType("my-job-type")
    .handler(jobHandler)
    .timeout(Duration.ofMinutes(30))
    .streamEnabled(false)  // Disable streaming, use polling only
    .pollInterval(Duration.ofSeconds(30))
    .maxJobsActive(1)
    .requestTimeout(Duration.ofMinutes(10))
    .open();

Spring Boot:

camunda:
  client:
    worker:
      defaults:
        stream-enabled: false
        timeout: PT30M
        poll-interval: PT30S

Key Differences: Camunda 8.7 vs 8.8/8.9

Setting 8.7 Default 8.8/8.9 Default Impact on Debugging
Gateway request timeout 30s+ 20s Shorter window before DEADLINE_EXCEEDED
Job worker streaming Not default Enabled by default Streams break when JVM pauses
Long poll timeout Generous Tighter Polls expire during breakpoint pauses
gRPC keep-alive Relaxed More aggressive Connections dropped sooner
Default job timeout 5 min 5 min (but reassignment faster) Jobs reassigned while debugging

Recommended Debugging Workflow

  1. Start c8run with extended timeouts (Solution 2)

  2. Use a debug Spring profile (Solution 3) with:

    • maxJobsActive: 1

    • timeout: PT30M or more

    • streamEnabled: false

    • pollInterval: PT30S or more

    • requestTimeout: PT10M or more

  3. Set breakpoints only in business logic, not in framework/handler infrastructure code

  4. Use conditional breakpoints to minimize pause frequency

  5. If debugging a specific process instance, activate a single job manually rather than using continuous polling:

// Manual single-job activation for focused debugging
ActivateJobsResponse response = client.newActivateJobsCommand()
    .jobType("my-job-type")
    .maxJobsToActivate(1)
    .timeout(Duration.ofMinutes(60))  // Very long timeout for debugging
    .requestTimeout(Duration.ofMinutes(10))
    .send()
    .join();

for (ActivatedJob job : response.getJobs()) {
    // Set breakpoint here
    processJob(job);
    
    client.newCompleteCommand(job.getKey())
        .requestTimeout(Duration.ofMinutes(10))
        .send()
        .join();
}

Quick-Start Fix (Minimal Changes)

If you want the fastest fix with minimal code changes, add these JVM arguments to your IDE debug configuration:

-Dzeebe.client.requestTimeout=600000
-Dzeebe.client.worker.defaultJobTimeout=1800000  
-Dzeebe.client.worker.defaultJobWorkerMaxJobsActive=1
-Dcamunda.client.zeebe.defaults.request-timeout=PT10M
-Dcamunda.client.worker.defaults.timeout=PT30M
-Dcamunda.client.worker.defaults.stream-enabled=false

And set these environment variables before starting c8run:

set ZEEBE_GATEWAY_LONGPOLLING_TIMEOUT=600000
set ZEEBE_GATEWAY_REQUEST_TIMEOUT=600s
c8run.exe start
4

Thanks @aravindhrs now it is working pretty well marking it as a solution.