Hi Team,
We are getting below exception continuously at client side (worker code).
INFO [main] io.camunda.zeebe.spring.client.jobhandling.JobWorkerManager - . Starting Zeebe worker: ZeebeWorkerValue{type='MY_WORKER', name='myWorker#myDecisionMatrix', timeout=PT1M, maxJobsActive=10, requestTimeout=PT1M, pollInterval=PT1M, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=MethodInfo{classInfo=ClassInfo{beanName=myWorker}, method=public java.util.Map com.example.MyWorker.myDecisionMatrix(io.camunda.zeebe.client.api.worker.JobClient,io.camunda.zeebe.client.api.response.ActivatedJob)}, tenantIds=[<default>], forceFetchAllVariables=false, streamEnabled=true, streamTimeout=PT1H, maxRetries=0}
WARN [pool-6-thread-1] io.camunda.client.job.worker - Failed to stream jobs of type 'MY_WORKER' to worker 'MyWorker#myDecisionMatrix'
io.grpc.StatusRuntimeException: UNAVAILABLE: unavailable
at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.68.2.jar!/:1.68.2]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.68.2.jar!/:1.68.2]
at io.opentelemetry.javaagent.shaded.instrumentation.grpc.v1_6.TracingClientInterceptor$TracingClientCall$TracingClientCallListener.onClose(TracingClientInterceptor.java:159) ~[javaagent.jar:?]
Worker config are:
camunda:
client:
mode: self-managed
tenant-ids:
- <default>
auth:
client-id: ${CLIENT_ID}
client-secret: ${CLIENT_SECRET}
issuer: ${AUTHZ_ISSUER}
zeebe:
enabled: true
base-url: ${GATEWAY_BASE_URL}
audience: zeebe-api
grpc-address: ${GRPC_URL}
rest-address: ${REST_URL}
prefer-rest-over-grpc: false
defaults:
enabled: true
stream-enabled: true
request-timeout: PT60S
timeout: PT60S
poll-interval: PT60S
max-jobs-active: 10
identity:
enabled: true
audience: identity-api
base-url: ${IDENTITY_BASE_URL}
Depenedencies as below at worker:
<dependency>
<groupId>io.camunda</groupId>
<artifactId>spring-boot-starter-camunda-sdk</artifactId>
<version>8.6.9</version>
<dependency>
<spring-boot.version>3.3.7</spring-boot.version>
<camunda-sdk.version>8.6.9</camunda-sdk.version>
Camunda cluster version (Self-Managed): Version 8.6.6
@nathan.loding Can you please share some insights about this issue? Right now what I’m think of the solution is:
Upgrading camunda cluster version from 8.6.6 to 8.6.9 should solve the issue
Otherwise revisit the worker config to adjust to work
cpbpm
February 24, 2025, 3:49pm
2
Back to your old post from you, Camunda Zeebe Client Error: io.grpc.StatusRuntimeException: UNAVAILABLE: unavailable , you know maintaining the same version, would solve the issue. Did you try it out?
@aravindhrs - please don’t tag people who haven’t already participated in the thread; this isn’t an official support forum. If you need priority support, it is best to open a support ticket.
The error here is different than the error cpbpm linked to. This one looks like an issue with the ingress possibly terminating idle streaming connections? This issue seems related, especially the last comment:
opened 11:34AM - 03 Dec 24 UTC
kind/toil
support
component/zeebe
**Description**
Getting this warning over and over after running a Spring boot … application for a while (40s or so):
**2024-12-03T11:14:00.731+01:00 WARN 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] io.camunda.client.job.worker : Failed to stream jobs of type 'trackOrderStatus' to worker 'orderHandler#handleOrderStatusJob' io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP status code 504**
I tested the same application with Java client and I can't reproduce it (even forcing a job _.streamEnabled(true)_).
It seems that the Spring Client has the job stream enabled by default.
The "warning" stopped after manually adding it to the .yaml file:
```
camunda:
client:
mode: saas
auth:
client-id: ***
client-secret: ***
cluster-id: ***
region: ***
zeebe:
defaults:
stream-enabled: false
```
The code is available here:
* Java: https://github.com/camunda-academy/c8-develop-workers-java-lab
* Spring: https://github.com/camunda-academy/c8-develop-workers-spring-lab
```
2024-12-03T11:13:21.082+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.camunda.academy.handler.OrderHandler : Getting order (9b6cf8d) for job: 6755399515704647
2024-12-03T11:13:21.082+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.camunda.academy.handler.OrderHandler : Order status tracked successfully for job: 6755399515704647
2024-12-03T11:13:21.094+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.academy.handler.PackItemsHandler : Handling job: 6755399515704699 for packing items
2024-12-03T11:13:21.098+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.a.services.TrackingOrderService : Packing items for job: 6755399515704699
2024-12-03T11:13:21.098+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.academy.handler.PackItemsHandler : Items packed successfully for job: 6755399515704699
2024-12-03T11:13:21.297+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.a.handler.ProcessPaymentHandler : Handling job: 6755399515704697 for process payment
2024-12-03T11:13:21.297+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.a.services.TrackingOrderService : Processing payment for job: 6755399515704697
2024-12-03T11:13:21.297+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.a.handler.ProcessPaymentHandler : Process payment successfully for job: 6755399515704697
2024-12-03T11:13:21.509+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.academy.handler.PackItemsHandler : Handling job: 6755399515704733 for packing items
2024-12-03T11:13:21.511+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.a.services.TrackingOrderService : Packing items for job: 6755399515704733
2024-12-03T11:13:21.511+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.academy.handler.PackItemsHandler : Items packed successfully for job: 6755399515704733
2024-12-03T11:13:21.713+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.a.handler.ProcessPaymentHandler : Handling job: 6755399515704731 for process payment
2024-12-03T11:13:21.713+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.a.services.TrackingOrderService : Processing payment for job: 6755399515704731
2024-12-03T11:13:21.720+01:00 INFO 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] c.c.a.handler.ProcessPaymentHandler : Process payment successfully for job: 6755399515704731
2024-12-03T11:14:00.731+01:00 WARN 26060 --- [c8-develop-workers-spring-lab] [pool-1-thread-1] io.camunda.client.job.worker : Failed to stream jobs of type 'trackOrderStatus' to worker 'orderHandler#handleOrderStatusJob'
io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP status code 504
invalid content-type: text/html
headers: Metadata(:status=504,date=Tue, 03 Dec 2024 10:14:00 GMT,content-type=text/html,strict-transport-security=max-age=63072000; includeSubDomains,content-length=160)
DATA-----------------------------
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx</center>
</body>
</html>
at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.68.1.jar:1.68.1]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.68.1.jar:1.68.1]
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.68.1.jar:1.68.1]
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.68.1.jar:1.68.1]
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.68.1.jar:1.68.1]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564) ~[grpc-core-1.68.1.jar:1.68.1]
at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72) ~[grpc-core-1.68.1.jar:1.68.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729) ~[grpc-core-1.68.1.jar:1.68.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710) ~[grpc-core-1.68.1.jar:1.68.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.68.1.jar:1.68.1]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.68.1.jar:1.68.1]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]`
```
**Related**
[SUPPORT-25369](https://jira.camunda.com/browse/SUPPORT-25369)
1 Like
I tried decreasing the default streamTimeout=PT1H
to streamTimeout=PT5M
.
But still it throws same warning the continuously.
Disabling streamEnabled=false
property works and stops emitting the warnings in the logs.
system
Closed
March 3, 2025, 9:53pm
5
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.