Hi team, Our application processes, 2000 records at a time. What we have observed it after processing some process instances, the records are stuck at service task at different places. We spin up around 6 application pods and 3 zeebe cluster and 3 partition and 3 replication factor. We are fine with slow processing but what have observed that after certain time none of the process instances are processed further. In zeebe gateway I can see warnings as “Failed to activate jobs for type WorkerTaskDelegate from partition 3 ava.net.ConnectException: Connection RemoteClientConnection camunda-platform-zeebe-1.camunda-platform-zeebe.app-dev.svc/xxx.xx.xxx.xxx:26501]} was closed
at io.atomix.cluster.messaging.impl.AbstractClientConnection.close(AbstractClientConnection.java:76) ~[zeebe-atomix-cluster-8.2.11.jar:8.2.11]”
To resolve this issue , I had to restart zeebe and zeebe gateway and then stuck processes were processed.
Please advise.