io.grpc.StatusRuntimeException: UNAVAILABLE: unavailable [io.camunda.client.job.worker - Failed to stream jobs]

[aravindhrs]

Hi Team,

We are getting below exception continuously at client side (worker code).

INFO [main] io.camunda.zeebe.spring.client.jobhandling.JobWorkerManager - . Starting Zeebe worker: ZeebeWorkerValue{type='MY_WORKER', name='myWorker#myDecisionMatrix', timeout=PT1M, maxJobsActive=10, requestTimeout=PT1M, pollInterval=PT1M, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=MethodInfo{classInfo=ClassInfo{beanName=myWorker}, method=public java.util.Map com.example.MyWorker.myDecisionMatrix(io.camunda.zeebe.client.api.worker.JobClient,io.camunda.zeebe.client.api.response.ActivatedJob)}, tenantIds=[<default>], forceFetchAllVariables=false, streamEnabled=true, streamTimeout=PT1H, maxRetries=0}

WARN [pool-6-thread-1] io.camunda.client.job.worker - Failed to stream jobs of type 'MY_WORKER' to worker 'MyWorker#myDecisionMatrix'
io.grpc.StatusRuntimeException: UNAVAILABLE: unavailable
at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.68.2.jar!/:1.68.2]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.68.2.jar!/:1.68.2]
at io.opentelemetry.javaagent.shaded.instrumentation.grpc.v1_6.TracingClientInterceptor$TracingClientCall$TracingClientCallListener.onClose(TracingClientInterceptor.java:159) ~[javaagent.jar:?]

Worker config are:

camunda:
  client:
    mode: self-managed
    tenant-ids:
      - <default>
    auth:
      client-id: ${CLIENT_ID}
      client-secret: ${CLIENT_SECRET}
      issuer: ${AUTHZ_ISSUER}
    zeebe:
      enabled: true
      base-url: ${GATEWAY_BASE_URL}
      audience: zeebe-api
      grpc-address: ${GRPC_URL}
      rest-address: ${REST_URL}
      prefer-rest-over-grpc: false
      defaults:
        enabled: true
        stream-enabled: true
        request-timeout: PT60S
        timeout: PT60S
        poll-interval: PT60S
        max-jobs-active: 10
    identity:
      enabled: true
      audience: identity-api
      base-url: ${IDENTITY_BASE_URL}

Depenedencies as below at worker:

 <dependency>
     <groupId>io.camunda</groupId>
     <artifactId>spring-boot-starter-camunda-sdk</artifactId>
     <version>8.6.9</version>
 <dependency>

<spring-boot.version>3.3.7</spring-boot.version>
<camunda-sdk.version>8.6.9</camunda-sdk.version>

Camunda cluster version (Self-Managed): Version 8.6.6

@nathan.loding Can you please share some insights about this issue? Right now what I’m think of the solution is:

• Upgrading camunda cluster version from 8.6.6 to 8.6.9 should solve the issue
• Otherwise revisit the worker config to adjust to work

[cpbpm]

Back to your old post from you, Camunda Zeebe Client Error: io.grpc.StatusRuntimeException: UNAVAILABLE: unavailable , you know maintaining the same version, would solve the issue. Did you try it out?

[nathan.loding]

@aravindhrs - please don’t tag people who haven’t already participated in the thread; this isn’t an official support forum. If you need priority support, it is best to open a support ticket.

The error here is different than the error cpbpm linked to. This one looks like an issue with the ingress possibly terminating idle streaming connections? This issue seems related, especially the last comment:

Spring Client stream warning

[aravindhrs]

I tried decreasing the default streamTimeout=PT1H to streamTimeout=PT5M.

But still it throws same warning the continuously.

Disabling streamEnabled=false property works and stops emitting the warnings in the logs.