AWS DEV to Camunda SAAS - Getting below error

We are using Camunda SAAS. we stuck with below error last 2days. Please help ASAP. in local spring boot to SAAS DEV working fine. the only issue is AWS DEV to Camunda DEV.
DEPLOYMENT_TRACK: stable
MANAGED_BY: cloudplatform
SERVICE: xxxxxx-camunda-demo-application
SFLY_ENV: DEV
aws_account_name: sfly-aws-dev
category: io.camunda.zeebe.spring.client.actuator.ZeebeClientHealthIndicator
container_id: 208d64f1ccd5
container_name: ecs-xxxxxx-camunda-demo-application-dev-28-xxxxxx-camunda-demo-application-a6f1d687b7b3fda9c701
level: WARN
message: Health check failed
stack_trace: io.camunda.zeebe.client.api.command.ClientException: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Unexpected partition broker health DEAD, should be one of [HEALTHY, UNHEALTHY]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:95)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:50)
at io.camunda.zeebe.spring.client.actuator.ZeebeClientHealthIndicator.doHealthCheck(ZeebeClientHealthIndicator.java:18)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
at java.base/java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1448)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.util.concurrent.ExecutionException: java.lang.RuntimeException: Unexpected partition broker health DEAD, should be one of [HEALTHY, UNHEALTHY]
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:48)
… 8 common frames omitted
Caused by: java.lang.RuntimeException: Unexpected partition broker health DEAD, should be one of [HEALTHY, UNHEALTHY]
at io.camunda.zeebe.client.impl.response.PartitionInfoImpl.(PartitionInfoImpl.java:53)
at io.camunda.zeebe.client.impl.response.BrokerInfoImpl.(BrokerInfoImpl.java:41)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at io.camunda.zeebe.client.impl.response.TopologyImpl.(TopologyImpl.java:34)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.onNext(ZeebeClientFutureImpl.java:72)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:465)
at io.opentelemetry.javaagent.shaded.instrumentation.grpc.v1_6.TracingClientInterceptor$TracingClientCall$TracingClientCallListener.onMessage(TracingClientInterceptor.java:153)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:661)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:646)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

task_id: 8d64239c421c497cbe74d509fb97896a
threadName: HealthCheck2
type: application
}

At a guess, you are using a Zeebe client version that cannot parse the return status DEAD.

Searching the Zeebe source code for DEAD, I find this commit: fix: java client accepts `DEAD` partitions in topology · camunda/zeebe@eaf3cda · GitHub.

So, you have potentially two issues, that I can see:

  1. One of your partitions is DEAD.
  2. Your client application can’t parse that response.

So, I would make sure that the client library that you are using is the latest version, and includes this commit, as a first step to trouble-shooting this.

HTH,
Josh

Thanks a lot Josh. with latest version the issue got resolved.

1 Like