Guys,
I have a Zeebe Broker, with only one Node and Partition.
It was working fine, but last week it starts to return a lot of error, disconnect the workers, and mark the partition as unhealthy.
I think that it is overloaded and have a lot of instances queued to be processe. Here is the error:
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: 2021-10-28 20:51:51.379 [io.zeebe.gateway.impl.broker.BrokerRequestManager] [Broker-0-zb-actors-1] WARN io.zeebe.gateway - Failed to activate jobs for type setar-email-enviado from partition 1
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: java.util.concurrent.TimeoutException: Request type command-api-1 timed out in 14999 milliseconds
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: at io.atomix.cluster.messaging.impl.AbstractClientConnection$Callback.timeout(AbstractClientConnection.java:163) ~[atomix-cluster-0.25.3.jar:0.25.3]
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
Oct 28 20:51:51 ip-10-0-3-30 broker[31105]: at java.lang.Thread.run(Thread.java:829) ~[?:?]
Oct 28 20:52:03 ip-10-0-3-30 broker[31105]: 2021-10-28 20:52:03.277 [Broker-0-HealthCheckService] [Broker-0-zb-actors-1] INFO io.zeebe.broker.system - Partition-1 recovered, marking it as healthy
Oct 28 20:53:04 ip-10-0-3-30 broker[31105]: 2021-10-28 20:53:04.683 [Broker-0-HealthCheckService] [Broker-0-zb-actors-1] ERROR io.zeebe.broker.system - Partition-1 failed, marking it as unhealthy
The Zeebe is the 0.25.3
Thanks