Sequencer is closed warning blocks all deployments and job activations

Camunda 8 Topics
, ,
1

I have a setup with:

  1. Zeebe
  2. Zeebe-gateway
  3. Elasticsearch
  4. A Spring Boot application implementing Jobs and deploying processes at startup.

All deployed on Kubernetes.

I notice that after some days of uptime, the Spring Boot application keeps logging "logger":"io.camunda.zeebe.client.job.poller","thread":"grpc-default-executor-111","message":"Failed to activate jobs for worker ...

Looking at the zeebe-gateway logs:

2023-12-27 05:29:52.648 [ActivateJobsHandler] [gateway-scheduler-zb-actors-0] WARN 
      io.camunda.zeebe.gateway - Failed to activate jobs for type create_mission from partition 2
java.util.concurrent.TimeoutException: Request ProtocolRequest{id=29213, subject=command-api-2, sender=10.42.5.103:26502, payload=byte[]{length=153, hash=-606723541}} to zeebe-0.zeebe.backend.svc:26501 timed out in PT15S
	at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$sendAndReceive$4(NettyMessagingService.java:230) ~[zeebe-atomix-cluster-8.1.2.jar:8.1.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at java.lang.Thread.run(Unknown Source) ~[?:?]

And finally in the zeebe logs:

2023-12-27 08:26:57.504 [io.camunda.zeebe.broker.transport.commandapi.CommandApiRequestHandler] [Broker-0-zb-actors-1] WARN 
      io.camunda.zeebe.logstreams.impl.log.Sequencer - Rejecting write of [LogAppendEntryImpl[key=-1, sourceIndex=-1, recordMetadata=RecordMetadata{recordType=COMMAND, valueType=DEPLOYMENT, intent=CREATE}, recordValue={"resources":[{"resourceName":"execute-mission.bpmn","resource":"..."}],"processesMetadata":[],"decisionRequirementsMetadata":[],"decisionsMetadata":[]}]], sequencer is closed

Not sure what to make of it or how reproduce it even. Thought it’s about restarting Zeebe, but that does not seem to trigger it consistently.

Any help / guidance is very much appreciated.

2

Hi @adrianAzoitei,

I personally haven’t seen these exceptions before, but my wild guess is that the disk for writing Zeebe logs is either full or not available.

Hope this helps, Ingo