My program always reports this error. What's the reason for this

Camunda 8 Topics
1

My program always reports this error. What’s the reason for this
25-09-22 14:37:22.512 WARN 3246849 — [ult-executor-67] io.camunda.client.job.poller : Failed to activate jobs for worker default and job type HB_UserTaskCreatedHook

io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP status code 504
invalid content-type: text/html
headers: Metadata(:status=504,date=Mon, 22 Sep 2025 06:37:22 GMT,content-type=text/html,strict-transport-security=max-age=31536000; includeSubDomains,content-length=160)
DATA-----------------------------

504 Gateway Time-out

504 Gateway Time-out

nginx 
    at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.73.0.jar!/:1.73.0]
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:564) ~[grpc-stub-1.73.0.jar!/:1.73.0]
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:565) ~[grpc-core-1.73.0.jar!/:1.73.0]
    at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72) ~[grpc-core-1.73.0.jar!/:1.73.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:733) ~[grpc-core-1.73.0.jar!/:1.73.0]
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:714) ~[grpc-core-1.73.0.jar!/:1.73.0]
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.73.0.jar!/:1.73.0]
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.73.0.jar!/:1.73.0]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
2

Hi there! :wave:

I can help you understand this error. You’re encountering a 504 Gateway Timeout error, which is a known issue when using Camunda 8 with nginx as a reverse proxy.

Root Cause

The error occurs because:

  • nginx doesn’t support HTTP/2 keepalive pings that are required for long-lived gRPC streams (like job streaming)
  • nginx closes idle connections, causing your worker to receive 504 Gateway Timeout errors when trying to activate jobs
  • This results in the io.grpc.StatusRuntimeException: UNAVAILABLE you’re seeing

Solution

You have two main options to fix this:

Option 1: Configure Stream Timeout (Recommended)

Configure both your Camunda worker and nginx with appropriate timeouts:

In your Camunda worker (Java client):

zeebe:
  client:
    default-job-worker-stream-enabled: true
    # Set stream timeout to 1 hour
    stream-timeout: PT1H

In nginx configuration:

grpc_send_timeout 70m;  # Slightly higher than worker stream timeout

Option 2: Disable Job Streaming

If you can’t modify nginx configuration, you can disable job streaming and use polling instead:

zeebe:
  client:
    default-job-worker-stream-enabled: false

Note: This may increase latency and is not recommended for high-throughput scenarios.

Additional Information

To help you better, could you please provide:

  • Your Camunda version (format: major.minor.patch)
  • Are you using Camunda SaaS or Self-Managed?
  • Are you using nginx as a reverse proxy in your setup?

References

Let me know if you need help implementing any of these solutions!

3

Camunda SaaS

4

It seems that the ingress nginx I am using does not have the grpc_dend_timeout configuration

5
  • Are you using Camunda SaaS or Self-Managed? YES
6

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.