Troubleshooting Unhealthy Partition Status in Zeebe-Broker (v8.7.13, K8s 3-Node Cluster)

jamesxql · December 11, 2025, 10:36am

Recently, I’ve noticed that some partitions in my Zeebe-broker engine have started to show an unhealthy status. Upon checking the logs, I found the following error:

[2025-12-11 18:22:48.818] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 starting
[2025-12-11 18:22:48.818] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning Metrics
[2025-12-11 18:22:48.818] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning LogStorage
[2025-12-11 18:22:48.819] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning LogStream
[2025-12-11 18:22:48.821] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning ZeebeDb
[2025-12-11 18:22:48.821] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning Migration
[2025-12-11 18:22:48.827] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning QueryService
[2025-12-11 18:22:48.827] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning BackupStore
[2025-12-11 18:22:48.827] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning BackupManager
[2025-12-11 18:22:48.828] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning InterPartitionCommandService
[2025-12-11 18:22:48.835] [zb-actors-1] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] INFO 
	io.camunda.zeebe.broker.system - Transition to LEADER on term 15 - transitioning StreamProcessor
[2025-12-11 18:22:48.837] [zb-actors-1] [{actor-name=StreamProcessor-2, actor-scheduler=Broker-1, partitionId=2}] ERROR
	io.camunda.zeebe.logstreams - Actor StreamProcessor-2 failed in phase STARTED.
java.lang.IllegalStateException: Expected to find event with the snapshot position 27720389 in log stream, but nothing was found. Failed to recover 'StreamProcessor-2'.
	at io.camunda.zeebe.stream.impl.StreamProcessor.recoverFromSnapshot(StreamProcessor.java:384) ~[zeebe-stream-platform-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.stream.impl.StreamProcessor.onActorStarted(StreamProcessor.java:167) ~[zeebe-stream-platform-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:84) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:43) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:125) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:127) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:104) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:224) [zeebe-scheduler-8.7.13.jar:8.7.13]
[2025-12-11 18:22:48.838] [zb-actors-1] [{actor-name=StreamProcessor-2, actor-scheduler=Broker-1, partitionId=2}] ERROR
	io.camunda.zeebe.logstreams - Actor StreamProcessor-2 failed in phase STARTED.
java.lang.NullPointerException: Cannot invoke "io.camunda.zeebe.stream.impl.StreamProcessor$AsyncProcessingScheduleServiceActor.closeAsync()" because "this.asyncActor" is null
	at io.camunda.zeebe.stream.impl.StreamProcessor.onFailure(StreamProcessor.java:412) ~[zeebe-stream-platform-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.stream.impl.StreamProcessor.onActorStarted(StreamProcessor.java:238) ~[zeebe-stream-platform-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:84) ~[zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:43) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:125) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:127) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:104) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:224) [zeebe-scheduler-8.7.13.jar:8.7.13]
[2025-12-11 18:22:48.843] [zb-actors-1] [{actor-name=StreamProcessor-2, actor-scheduler=Broker-1, partitionId=2}] ERROR
	io.camunda.zeebe.util.actor - Unexpected error occurred in task StreamProcessor-2 ACTIVE phase: STARTED
java.lang.NullPointerException: Cannot invoke "io.camunda.zeebe.stream.impl.StreamProcessor$AsyncProcessingScheduleServiceActor.closeAsync()" because "this.asyncActor" is null
	at io.camunda.zeebe.stream.impl.StreamProcessor.onFailure(StreamProcessor.java:412) ~[zeebe-stream-platform-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.stream.impl.StreamProcessor.handleFailure(StreamProcessor.java:271) ~[zeebe-stream-platform-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorTask.onFailure(ActorTask.java:305) ~[zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:46) ~[zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:125) ~[zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:127) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:104) [zeebe-scheduler-8.7.13.jar:8.7.13]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:224) [zeebe-scheduler-8.7.13.jar:8.7.13]
[2025-12-11 18:24:23.128] [zb-actors-0] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] WARN 
	io.camunda.zeebe.broker.system - ZeebePartitionHealth-2 failed, marking it as unhealthy: HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT1M34.292S, throwable=null, cause=null, since=2025-12-11T10:24:23.127Z], children={}]
[2025-12-11 18:24:23.128] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT1M34.292S, throwable=null, cause=null, since=2025-12-11T10:24:23.127Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT1M34.292S, throwable=null, cause=null, since=2025-12-11T10:24:23.127Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:24:23.129] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT1M34.292S, throwable=null, cause=null, since=2025-12-11T10:24:23.127Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT1M34.292S, throwable=null, cause=null, since=2025-12-11T10:24:23.127Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:25:23.234] [zb-actors-0] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] WARN 
	io.camunda.zeebe.broker.system - ZeebePartitionHealth-2 failed, marking it as unhealthy: HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT2M34.399S, throwable=null, cause=null, since=2025-12-11T10:25:23.234Z], children={}]
[2025-12-11 18:25:23.235] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT2M34.399S, throwable=null, cause=null, since=2025-12-11T10:25:23.234Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT2M34.399S, throwable=null, cause=null, since=2025-12-11T10:25:23.234Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:25:23.235] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT2M34.399S, throwable=null, cause=null, since=2025-12-11T10:25:23.234Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT2M34.399S, throwable=null, cause=null, since=2025-12-11T10:25:23.234Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:26:23.613] [zb-actors-0] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] WARN 
	io.camunda.zeebe.broker.system - ZeebePartitionHealth-2 failed, marking it as unhealthy: HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT3M34.777S, throwable=null, cause=null, since=2025-12-11T10:26:23.612Z], children={}]
[2025-12-11 18:26:23.613] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT3M34.777S, throwable=null, cause=null, since=2025-12-11T10:26:23.612Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT3M34.777S, throwable=null, cause=null, since=2025-12-11T10:26:23.612Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:26:23.613] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT3M34.777S, throwable=null, cause=null, since=2025-12-11T10:26:23.612Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT3M34.777S, throwable=null, cause=null, since=2025-12-11T10:26:23.612Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:27:23.821] [zb-actors-0] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] WARN 
	io.camunda.zeebe.broker.system - ZeebePartitionHealth-2 failed, marking it as unhealthy: HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT4M34.986S, throwable=null, cause=null, since=2025-12-11T10:27:23.821Z], children={}]
[2025-12-11 18:27:23.822] [zb-actors-1] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT4M34.986S, throwable=null, cause=null, since=2025-12-11T10:27:23.821Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT4M34.986S, throwable=null, cause=null, since=2025-12-11T10:27:23.821Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:27:23.822] [zb-actors-1] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT4M34.986S, throwable=null, cause=null, since=2025-12-11T10:27:23.821Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT4M34.986S, throwable=null, cause=null, since=2025-12-11T10:27:23.821Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:28:24.384] [zb-actors-0] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] WARN 
	io.camunda.zeebe.broker.system - ZeebePartitionHealth-2 failed, marking it as unhealthy: HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT5M35.549S, throwable=null, cause=null, since=2025-12-11T10:28:24.384Z], children={}]
[2025-12-11 18:28:24.385] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT5M35.549S, throwable=null, cause=null, since=2025-12-11T10:28:24.384Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT5M35.549S, throwable=null, cause=null, since=2025-12-11T10:28:24.384Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:28:24.385] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT5M35.549S, throwable=null, cause=null, since=2025-12-11T10:28:24.384Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT5M35.549S, throwable=null, cause=null, since=2025-12-11T10:28:24.384Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:29:11.096] [zb-fs-workers-0] [{actor-name=SnapshotStore-3, actor-scheduler=Broker-1, partitionId=3}] WARN 
	io.camunda.zeebe.logstreams.snapshot - No log entry for next snapshot position 27723595, using index and term from previous snapshot
[2025-12-11 18:29:24.422] [zb-actors-0] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] WARN 
	io.camunda.zeebe.broker.system - ZeebePartitionHealth-2 failed, marking it as unhealthy: HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT6M35.586S, throwable=null, cause=null, since=2025-12-11T10:29:24.421Z], children={}]
[2025-12-11 18:29:24.423] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT6M35.586S, throwable=null, cause=null, since=2025-12-11T10:29:24.421Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT6M35.586S, throwable=null, cause=null, since=2025-12-11T10:29:24.421Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:29:24.423] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT6M35.586S, throwable=null, cause=null, since=2025-12-11T10:29:24.421Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT6M35.586S, throwable=null, cause=null, since=2025-12-11T10:29:24.421Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:30:24.480] [zb-actors-0] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] WARN 
	io.camunda.zeebe.broker.system - ZeebePartitionHealth-2 failed, marking it as unhealthy: HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT7M35.645S, throwable=null, cause=null, since=2025-12-11T10:30:24.480Z], children={}]
[2025-12-11 18:30:24.481] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT7M35.645S, throwable=null, cause=null, since=2025-12-11T10:30:24.480Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT7M35.645S, throwable=null, cause=null, since=2025-12-11T10:30:24.480Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:30:24.481] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT7M35.645S, throwable=null, cause=null, since=2025-12-11T10:30:24.480Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT7M35.645S, throwable=null, cause=null, since=2025-12-11T10:30:24.480Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:31:24.725] [zb-actors-0] [{actor-name=ZeebePartition-2, actor-scheduler=Broker-1, partitionId=2}] WARN 
	io.camunda.zeebe.broker.system - ZeebePartitionHealth-2 failed, marking it as unhealthy: HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT8M35.89S, throwable=null, cause=null, since=2025-12-11T10:31:24.725Z], children={}]
[2025-12-11 18:31:24.726] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT8M35.89S, throwable=null, cause=null, since=2025-12-11T10:31:24.725Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT8M35.89S, throwable=null, cause=null, since=2025-12-11T10:31:24.725Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]
[2025-12-11 18:31:24.726] [zb-actors-0] [{actor-name=HealthCheckService, actor-scheduler=Broker-1}] WARN 
	io.camunda.zeebe.broker.system - Partition-2 failed, marking it as unhealthy: HealthReport[componentName=Partition-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT8M35.89S, throwable=null, cause=null, since=2025-12-11T10:31:24.725Z], children={ZeebePartitionHealth-2=HealthReport[componentName=ZeebePartitionHealth-2, status=UNHEALTHY, issue=HealthIssue[message=Transition from FOLLOWER on term 15 appears blocked, step StreamProcessor has been running for PT8M35.89S, throwable=null, cause=null, since=2025-12-11T10:31:24.725Z], children={}], RaftPartition-2=HealthReport[componentName=RaftPartition-2, status=HEALTHY, issue=null, children={}]}]

What could be the possible cause of this error, and how can I resolve it? My Zeebe version is 8.7.13, and it is deployed in a K8s container with 3 nodes and 3 partitions.

Thanks!

cpbpm · December 11, 2025, 2:27pm

Similar issues are reported in the past for earlier versions,but not for 8.7.13.

github.com/camunda/camunda

IllegalStateException: Failed to recover 'Broker-0-StreamProcessor-2'

opened 02:17PM - 25 Feb 20 UTC

closed 12:44PM - 11 Mar 20 UTC

strawhat5

kind/bug

**Describe the bug** After redeploying the cluster, it is falling into an Illeg…alStateException state. The broker is still up, but rejecting all gRPC requests with `INTERNAL` error. It is happening on my beta environment, but not on alpha environment. My dev cluster was fine till few hours back, but now I am seeing same gRPC `INTERNAL` errors on dev as well. Alpha is still fine by the way. **Expected behavior** The cluster should accept all gRPC requests. <details><summary>Full Stacktrace</summary> <p> ``` 2020-02-25 13:39:27.859 [Broker-0-StreamProcessor-2] [Broker-1-zb-actors-0] DEBUG io.zeebe.logstreams - Recovering state of partition 2 from snapshot 2020-02-25 13:39:28.608 [Broker-0-StreamProcessor-2] [Broker-1-zb-actors-0] ERROR io.zeebe.util.actor - Actor failed in phase 'STARTED'. Continue with next job. java.lang.IllegalStateException: Expected to find event with the snapshot position 64424590432 in log stream, but nothing was found. Failed to recover 'Broker-0-StreamProcessor-2'. at io.zeebe.engine.processor.StreamProcessor.recoverFromSnapshot(StreamProcessor.java:209) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.engine.processor.StreamProcessor.onActorStarted(StreamProcessor.java:121) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) ~[zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.22.1.jar:0.22.1] 2020-02-25 13:39:28.608 [Broker-1-ZeebePartition-2] [Broker-1-zb-actors-0] ERROR io.zeebe.broker.system - Unexpected error on stream processor installation! java.lang.IllegalStateException: Expected to find event with the snapshot position 64424590432 in log stream, but nothing was found. Failed to recover 'Broker-0-StreamProcessor-2'. at io.zeebe.engine.processor.StreamProcessor.recoverFromSnapshot(StreamProcessor.java:209) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.engine.processor.StreamProcessor.onActorStarted(StreamProcessor.java:121) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.22.1.jar:0.22.1] 2020-02-25 13:39:28.609 [Broker-1-ZeebePartition-2] [Broker-1-zb-actors-0] ERROR io.zeebe.broker.system - Failed to install leader partition 2 java.lang.IllegalStateException: Expected to find event with the snapshot position 64424590432 in log stream, but nothing was found. Failed to recover 'Broker-0-StreamProcessor-2'. at io.zeebe.engine.processor.StreamProcessor.recoverFromSnapshot(StreamProcessor.java:209) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.engine.processor.StreamProcessor.onActorStarted(StreamProcessor.java:121) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.22.1.jar:0.22.1] 2020-02-25 13:39:28.609 [Broker-0-LogStream-2] [Broker-1-zb-actors-0] INFO io.zeebe.logstreams - Close appender for log stream raft-partition-partition-2 2020-02-25 13:39:28.609 [Broker-0-StreamProcessor-2] [Broker-1-zb-actors-0] DEBUG io.zeebe.logstreams - Closed stream processor controller Broker-0-StreamProcessor-2. 2020-02-25 13:39:28.614 [Broker-0-StreamProcessor-3] [Broker-1-zb-actors-0] DEBUG io.zeebe.logstreams - Recovering state of partition 3 from snapshot 2020-02-25 13:39:29.185 [Broker-0-StreamProcessor-3] [Broker-1-zb-actors-0] ERROR io.zeebe.util.actor - Actor failed in phase 'STARTED'. Continue with next job. java.lang.IllegalStateException: Expected to find event with the snapshot position 60130044736 in log stream, but nothing was found. Failed to recover 'Broker-0-StreamProcessor-3'. at io.zeebe.engine.processor.StreamProcessor.recoverFromSnapshot(StreamProcessor.java:209) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.engine.processor.StreamProcessor.onActorStarted(StreamProcessor.java:121) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) ~[zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.22.1.jar:0.22.1] 2020-02-25 13:39:29.186 [Broker-1-ZeebePartition-3] [Broker-1-zb-actors-0] ERROR io.zeebe.broker.system - Unexpected error on stream processor installation! java.lang.IllegalStateException: Expected to find event with the snapshot position 60130044736 in log stream, but nothing was found. Failed to recover 'Broker-0-StreamProcessor-3'. at io.zeebe.engine.processor.StreamProcessor.recoverFromSnapshot(StreamProcessor.java:209) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.engine.processor.StreamProcessor.onActorStarted(StreamProcessor.java:121) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.22.1.jar:0.22.1] 2020-02-25 13:39:29.186 [Broker-1-ZeebePartition-3] [Broker-1-zb-actors-0] ERROR io.zeebe.broker.system - Failed to install leader partition 3 java.lang.IllegalStateException: Expected to find event with the snapshot position 60130044736 in log stream, but nothing was found. Failed to recover 'Broker-0-StreamProcessor-3'. at io.zeebe.engine.processor.StreamProcessor.recoverFromSnapshot(StreamProcessor.java:209) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.engine.processor.StreamProcessor.onActorStarted(StreamProcessor.java:121) ~[zeebe-workflow-engine-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.22.1.jar:0.22.1] 2020-02-25 13:39:29.187 [Broker-0-LogStream-3] [Broker-1-zb-actors-0] INFO io.zeebe.logstreams - Close appender for log stream raft-partition-partition-3 2020-02-25 13:39:29.187 [Broker-0-StreamProcessor-3] [Broker-1-zb-actors-0] DEBUG io.zeebe.logstreams - Closed stream processor controller Broker-0-StreamProcessor-3. 2020-02-25 13:39:29.187 [raft-partition-partition-2-write-buffer] [Broker-1-zb-actors-0] DEBUG io.zeebe.dispatcher - Dispatcher closed 2020-02-25 13:39:29.187 [raft-partition-partition-3-write-buffer] [Broker-1-zb-actors-0] DEBUG io.zeebe.dispatcher - Dispatcher closed 2020-02-25 13:39:32.638 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received broker was removed BrokerInfo{nodeId=2, partitionsCount=3, clusterSize=3, replicationFactor=3, partitionRoles={1=FOLLOWER, 2=FOLLOWER, 3=FOLLOWER}, partitionLeaderTerms={}}. 2020-02-25 13:39:32.639 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received member removed BrokerInfo{nodeId=2, partitionsCount=3, clusterSize=3, replicationFactor=3, partitionRoles={1=FOLLOWER, 2=FOLLOWER, 3=FOLLOWER}, partitionLeaderTerms={}} 2020-02-25 13:40:03.405 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received metadata change from Broker 2, partitions {} and terms {}. 2020-02-25 13:40:03.405 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 2, partitions {} terms {} 2020-02-25 13:40:04.404 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received metadata change from Broker 2, partitions {3=FOLLOWER} and terms {}. 2020-02-25 13:40:04.405 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 2, partitions {3=FOLLOWER} terms {} 2020-02-25 13:40:05.154 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received metadata change from Broker 2, partitions {2=FOLLOWER, 3=FOLLOWER} and terms {}. 2020-02-25 13:40:05.155 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 2, partitions {2=FOLLOWER, 3=FOLLOWER} terms {} 2020-02-25 13:40:05.904 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received metadata change from Broker 2, partitions {1=FOLLOWER, 2=FOLLOWER, 3=FOLLOWER} and terms {}. 2020-02-25 13:40:05.905 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 2, partitions {1=FOLLOWER, 2=FOLLOWER, 3=FOLLOWER} terms {} 2020-02-25 13:43:49.394 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received REACHABILITY_CHANGED for broker 0, do nothing. 2020-02-25 13:43:49.394 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received REACHABILITY_CHANGED from member 0, was not handled. 2020-02-25 13:43:59.638 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received broker was removed BrokerInfo{nodeId=0, partitionsCount=3, clusterSize=3, replicationFactor=3, partitionRoles={1=FOLLOWER, 2=FOLLOWER, 3=FOLLOWER}, partitionLeaderTerms={}}. 2020-02-25 13:43:59.639 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received member removed BrokerInfo{nodeId=0, partitionsCount=3, clusterSize=3, replicationFactor=3, partitionRoles={1=FOLLOWER, 2=FOLLOWER, 3=FOLLOWER}, partitionLeaderTerms={}} 2020-02-25 13:44:25.471 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.system - Removing leader partition services for partition PartitionId{id=1, group=raft-partition} 2020-02-25 13:44:25.471 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.system - Closing Broker-0-StreamProcessor-1 2020-02-25 13:44:25.471 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.system - Partition role transitioning from LEADER to FOLLOWER 2020-02-25 13:44:25.471 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.system - Closed Broker-0-StreamProcessor-1 successfully 2020-02-25 13:44:25.471 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.system - Closing io.zeebe.broker.logstreams.LogStreamDeletionService 2020-02-25 13:44:25.472 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.system - Closed io.zeebe.broker.logstreams.LogStreamDeletionService successfully 2020-02-25 13:44:25.473 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.logstreams.snapshot - Closed database from '/usr/local/zeebe/data/raft-partition/partitions/1/runtime'. 2020-02-25 13:44:25.473 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.system - Installing follower partition service for partition PartitionId{id=1, group=raft-partition} 2020-02-25 13:44:25.479 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.logstreams.snapshot - Available snapshots: [SnapshotImpl{position=68719979328, path=/usr/local/zeebe/data/raft-partition/partitions/1/snapshots/314340-19-1581687776238-68719979328}, SnapshotImpl{position=68719943328, path=/usr/local/zeebe/data/raft-partition/partitions/1/snapshots/314201-19-1581686876225-68719943328}, SnapshotImpl{position=68719907328, path=/usr/local/zeebe/data/raft-partition/partitions/1/snapshots/314064-19-1581685976198-68719907328}] 2020-02-25 13:44:25.516 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.logstreams.snapshot - Opened database from '/usr/local/zeebe/data/raft-partition/partitions/1/runtime'. 2020-02-25 13:44:25.517 [Broker-1-ZeebePartition-1] [Broker-1-zb-actors-0] DEBUG io.zeebe.logstreams.snapshot - Recovered state from snapshot 'SnapshotImpl{position=68719979328, path=/usr/local/zeebe/data/raft-partition/partitions/1/snapshots/314340-19-1581687776238-68719979328}' 2020-02-25 13:44:26.523 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received metadata change from Broker 0, partitions {} and terms {}. 2020-02-25 13:44:26.523 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 0, partitions {} terms {} 2020-02-25 13:44:26.773 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received metadata change from Broker 0, partitions {3=FOLLOWER} and terms {}. 2020-02-25 13:44:26.773 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 0, partitions {3=FOLLOWER} terms {} 2020-02-25 13:44:27.023 [GatewayTopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.gateway - Received metadata change from Broker 0, partitions {2=FOLLOWER, 3=FOLLOWER} and terms {}. 2020-02-25 13:44:27.023 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 0, partitions {2=FOLLOWER, 3=FOLLOWER} terms {} 2020-02-25 13:50:22.051 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [Broker-2-zb-actors-0] ERROR io.zeebe.gateway - Error handling gRPC request io.grpc.StatusRuntimeException: INTERNAL: Unexpected error occurred during the request processing at io.grpc.Status.asRuntimeException(Status.java:524) ~[grpc-api-1.26.0.jar:1.26.0] at io.zeebe.gateway.EndpointManager.convertThrowable(EndpointManager.java:372) ~[zeebe-gateway-0.22.1.jar:0.22.1] at io.zeebe.gateway.impl.job.ActivateJobsHandler.logErrorResponse(ActivateJobsHandler.java:109) ~[zeebe-gateway-0.22.1.jar:0.22.1] at io.zeebe.gateway.impl.job.ActivateJobsHandler.lambda$activateJobs$1(ActivateJobsHandler.java:96) ~[zeebe-gateway-0.22.1.jar:0.22.1] at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequest$3(BrokerRequestManager.java:147) ~[zeebe-gateway-0.22.1.jar:0.22.1] at io.zeebe.gateway.impl.broker.BrokerRequestManager.lambda$sendRequestInternal$5(BrokerRequestManager.java:183) ~[zeebe-gateway-0.22.1.jar:0.22.1] at io.zeebe.util.sched.future.FutureContinuationRunnable.run(FutureContinuationRunnable.java:33) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.22.1.jar:0.22.1] Caused by: java.util.concurrent.TimeoutException: Request timed out after PT15S at io.zeebe.transport.impl.RequestContext.timeout(RequestContext.java:109) ~[zeebe-transport-0.22.1.jar:0.22.1] at io.zeebe.transport.impl.AtomixClientTransportAdapter.timeoutFuture(AtomixClientTransportAdapter.java:128) ~[zeebe-transport-0.22.1.jar:0.22.1] at io.zeebe.transport.impl.AtomixClientTransportAdapter.lambda$sendRequestWithRetry$0(AtomixClientTransportAdapter.java:60) ~[zeebe-transport-0.22.1.jar:0.22.1] ... 6 more 2020-02-25 13:50:22.052 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [Broker-2-zb-actors-0] WARN io.zeebe.gateway - Failed to activate jobs for type http from partition 2 java.util.concurrent.TimeoutException: Request timed out after PT15S at io.zeebe.transport.impl.RequestContext.timeout(RequestContext.java:109) ~[zeebe-transport-0.22.1.jar:0.22.1] at io.zeebe.transport.impl.AtomixClientTransportAdapter.timeoutFuture(AtomixClientTransportAdapter.java:128) ~[zeebe-transport-0.22.1.jar:0.22.1] at io.zeebe.transport.impl.AtomixClientTransportAdapter.lambda$sendRequestWithRetry$0(AtomixClientTransportAdapter.java:60) ~[zeebe-transport-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:76) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.22.1.jar:0.22.1] at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.22.1.jar:0.22.1] ``` </p> </details> **Environment:** ``` 2020-02-25 13:53:58.999 [] [main] DEBUG io.zeebe.broker.system - Initializing configuration with base path /usr/local/zeebe 2020-02-25 13:53:59.065 [] [main] INFO io.zeebe.broker.system - Version: 0.22.1 2020-02-25 13:53:58.459 [] [main] INFO io.zeebe.broker.system - Starting broker 0 with configuration { "network": { "host": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local", "portOffset": 0, "maxMessageSize": "4M", "commandApi": { "defaultPort": 26501, "host": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local", "port": 26501, "advertisedHost": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local", "advertisedPort": 26501 }, "internalApi": { "defaultPort": 26502, "host": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local", "port": 26502, "advertisedHost": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local", "advertisedPort": 26502 }, "monitoringApi": { "defaultPort": 9600, "host": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local", "port": 9600, "advertisedHost": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local", "advertisedPort": 9600 } }, "cluster": { "initialContactPoints": [ "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local:26502", "beta-zeebe-zeebe-1.zeebe.beta.svc.cluster.local:26502", "beta-zeebe-zeebe-2.zeebe.beta.svc.cluster.local:26502" ], "partitionIds": [ 1, 2, 3 ], "nodeId": 0, "partitionsCount": 3, "replicationFactor": 3, "clusterSize": 3, "clusterName": "zeebe-cluster", "gossipFailureTimeout": 10000, "gossipInterval": 250, "gossipProbeInterval": 1000 }, "threads": { "cpuThreadCount": 1, "ioThreadCount": 1 }, "data": { "directories": [ "/usr/local/zeebe/data" ], "logSegmentSize": "512M", "snapshotPeriod": "15m", "raftSegmentSize": "512M", "maxSnapshots": 3 }, "exporters": [ { "id": "elasticsearch", "className": "io.zeebe.exporter.ElasticsearchExporter", "args": { "index": { "workflowInstance": true, "messageSubscription": true, "prefix": "zeebe-beta", "jobBatch": true, "error": true, "message": true, "createTemplate": true, "command": true, "rejection": true, "workflowInstanceCreation": true, "variable": true, "variableDocument": true, "workflowInstanceSubscription": true, "event": true, "job": true, "incident": true, "deployment": true }, "bulk": { "delay": 10.0, "size": 1000.0 }, "url": "my_url" } }, { "id": "hazelcast", "jarPath": "/usr/local/zeebe/lib/zeebe-hazelcast-exporter.jar", "className": "io.zeebe.hazelcast.exporter.HazelcastExporter", "args": { "updatePosition": true, "port": 5701.0, "topicPrefix": "zeebe-", "enabledRecordTypes": "EVENT", "enabledValueTypes": "DEPLOYMENT,JOB,JOB_BATCH,WORKFLOW_INSTANCE,WORKFLOW_INSTANCE_CREATION,WORKFLOW_INSTANCE_SUBSCRIPTION,INCIDENT,TIMER,VARIABLE,MESSAGE,MESSAGE_SUBSCRIPTION,MESSAGE_START_EVENT_SUBSCRIPTION" } } ], "gateway": { "enable": true, "network": { "host": "0.0.0.0", "port": 26500, "minKeepAliveInterval": "30s" }, "cluster": { "contactPoint": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local:26502", "maxMessageSize": "4M", "requestTimeout": "15s", "clusterName": "zeebe-cluster", "memberId": "gateway", "host": "0.0.0.0", "port": 26502 }, "threads": { "managementThreads": 1 }, "monitoring": { "enabled": false, "host": "beta-zeebe-zeebe-0.zeebe.beta.svc.cluster.local", "port": 9600 }, "security": { "enabled": false } }, "backpressure": { "enabled": true, "useWindowed": true, "algorithm": "vegas" }, "stepTimeout": "5m" } ```

I would recommend to open a ticket with Camunda support team.

jamesxql · December 15, 2025, 6:00am

Sorry, my previous description contained some inaccuracies. The error actually occurred in version 8.7.7 of Camunda, and we discovered this issue while upgrading the cluster to version 8.7.13. It appears that there is an inconsistency in the raft-log. During the cluster’s operation, we did not restart any brokers, and the operating system did not experience any downtime. It is highly likely that this was caused by an issue within Camunda’s own program. Please help analyze the cause of this problem, as it should be a serious issue. Thank you!

GotnOGuts · December 15, 2025, 4:23pm

I would suggest opening a support ticket on the support site as this site is a user forum (a place to bounce ideas off other users), rather than a support channel.

system · December 23, 2025, 11:45am

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.