About
We are seeing the Zeebe pods on Kubernetes restart frequently after increase the number of workflows instances creation
Infrastructure
We are using Zeebe Self-Managed using Helm and Kubernetes
Zeeve Version: 8.5.3
Helm Chart version: 10.0.5
Custer size: 5
Number of partitions: 10
Replication factor: 3
Gateways: 3
PVC size: 64Gi
Behavior
With this configuration, we are trying to support a rate of 15~20 workflow instances creations per second. We are seeing Zeebe handle this flow but its pods are restarting frequently in Kubernetes, in a period of 2 days running this rate, all Zeebe pods restarted at least 8 times.
We checked the logs of the terminated pods to check the errors that could be the reason of the restarts, but we only found WARNS:
2024-07-01 10:13:28.721 [] [atomix-cluster-heartbeat-sender] [] WARN
io.atomix.cluster.protocol.swim.probe - 3 - Failed to probe zeebe-cluster-zeebe-gateway-5958f977f-4glvb
java.util.concurrent.TimeoutException: Request atomix-membership-probe to 192.168.35.199:26502 timed out in PT0.1S
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$sendAndReceive$4(NettyMessagingService.java:261) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-07-01 10:13:29.493 [Broker-3] [zb-actors-2] [SnapshotDirector-4] INFO
io.camunda.zeebe.logstreams.snapshot - Finished taking temporary snapshot, need to wait until last written event position 7791661 is committed, current commit position is 7791660. After that snapshot will be committed.
2024-07-01 10:13:29.644 [Broker-3] [zb-fs-workers-0] [SnapshotStore-4] INFO
io.camunda.zeebe.snapshots.impl.FileBasedSnapshotStore - Committed new snapshot 7416095-12-7791644-9223372036854775807
2024-07-01 10:15:02.048 [] [atomix-cluster-heartbeat-sender] [] WARN
io.atomix.cluster.protocol.swim.probe - 3 - Failed to probe 2
java.util.concurrent.TimeoutException: Request atomix-membership-probe to zeebe-cluster-zeebe-2.zeebe-cluster-zeebe.xxx.svc:26502 timed out in PT0.1S
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$sendAndReceive$4(NettyMessagingService.java:261) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
2024-07-01 10:15:06.826 [] [atomix-cluster-heartbeat-sender] [] INFO
io.atomix.cluster.protocol.swim - 3 - Member unreachable Member{id=1, address=zeebe-cluster-zeebe-1.zeebe-cluster-zeebe.xxx.svc:26502, properties={brokerInfo=EADJAAAABAABAAAACgAAAAUAAAADAAAAAAABCgAAAGNvbW1hbmRBcGlGAAAAemVlYmUtY2x1c3Rlci16ZWViZS0xLnplZWJlLWNsdXN0ZXItemVlYmUub25laW50ZXJmYWNlLXN0YWdlLnN2YzoyNjUwMQUABgEAAAAAAgAAAAAFAAAAAQYAAAAABwAAAAEKAAAAAQwAAwEAAAATAAAAAAAAAAIAAAANAAAAAAAAAAYAAAAOAAAAAAAAAAUAAAA4LjUuMwUABgEAAAABAgAAAAEFAAAAAQYAAAABBwAAAAEKAAAAAQ==}}
2024-07-01 10:15:10.689 [] [atomix-cluster-heartbeat-sender] [] INFO
io.atomix.cluster.protocol.swim - 3 - Member reachable Member{id=1, address=zeebe-cluster-zeebe-1.zeebe-cluster-zeebe.xxx.svc:26502, properties={brokerInfo=EADJAAAABAABAAAACgAAAAUAAAADAAAAAAABCgAAAGNvbW1hbmRBcGlGAAAAemVlYmUtY2x1c3Rlci16ZWViZS0xLnplZWJlLWNsdXN0ZXItemVlYmUub25laW50ZXJmYWNlLXN0YWdlLnN2YzoyNjUwMQUABgEAAAAAAgAAAAAFAAAAAQYAAAAABwAAAAEKAAAAAQwAAwEAAAATAAAAAAAAAAIAAAANAAAAAAAAAAYAAAAOAAAAAAAAAAUAAAA4LjUuMwUABgEAAAABAgAAAAEFAAAAAQYAAAABBwAAAAEKAAAAAQ==}}
---
2024-07-01 10:25:42.580 [Broker-3] [raft-server-3-9] [raft-server-9] WARN
io.atomix.raft.roles.LeaderAppender - RaftServer{raft-partition-partition-9} - VersionedAppendRequest{version=2, term=18, leader=3, prevLogIndex=7445493, prevLogTerm=18, entries=1, commitIndex=7445493} to 0 failed
java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException: Request raft-partition-partition-9-append-versioned to zeebe-cluster-zeebe-0.zeebe-cluster-zeebe.xxx.svc:26502 timed out in PT2.5S
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$sendAndReceive$4(NettyMessagingService.java:259) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.util.concurrent.TimeoutException: Request raft-partition-partition-9-append-versioned to zeebe-cluster-zeebe-0.zeebe-cluster-zeebe.xxx.svc:26502 timed out in PT2.5S
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$sendAndReceive$4(NettyMessagingService.java:261) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
... 7 more
2024-07-01 10:25:42.621 [Broker-3] [raft-server-3-9] [raft-server-9] WARN
io.atomix.raft.roles.LeaderAppender - RaftServer{raft-partition-partition-9} - VersionedAppendRequest{version=2, term=18, leader=3, prevLogIndex=7445494, prevLogTerm=18, entries=16, commitIndex=7445510} to 0 failed
java.util.concurrent.CompletionException: io.atomix.cluster.messaging.MessagingException$ConnectionClosed: Connection RemoteClientConnection{channel=[id: 0x5d66a77d, L:/192.168.xx.xxx:59996 ! R:zeebe-cluster-zeebe-0.zeebe-cluster-zeebe.xxx.svc.cluster.local/192.168.xx.xxx:26502]} was closed
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$executeOnPooledConnection$25(NettyMessagingService.java:626) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) ~[guava-33.1.0-jre.jar:?]
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$executeOnPooledConnection$26(NettyMessagingService.java:624) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at io.atomix.cluster.messaging.impl.AbstractClientConnection.close(AbstractClientConnection.java:74) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$getOrCreateClientConnection$38(NettyMessagingService.java:702) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1161) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:753) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:729) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:619) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1349) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:755) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$1200(AbstractChannelHandlerContext.java:61) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext$11.run(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[netty-transport-classes-epoll-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: io.atomix.cluster.messaging.MessagingException$ConnectionClosed: Connection RemoteClientConnection{channel=[id: 0x5d66a77d, L:/192.168.xx.xxx:59996 ! R:zeebe-cluster-zeebe-0.zeebe-cluster-zeebe.xxx.svc.cluster.local/192.168.xx.xxx:26502]} was closed
at io.atomix.cluster.messaging.impl.AbstractClientConnection.close(AbstractClientConnection.java:76) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
... 24 more
2024-07-01 10:25:43.607 [Broker-3] [raft-server-3-4] [raft-server-4] WARN
io.atomix.raft.roles.LeaderAppender - RaftServer{raft-partition-partition-4} - VersionedAppendRequest{version=2, term=12, leader=3, prevLogIndex=7435702, prevLogTerm=12, entries=1, commitIndex=7435702} to 0 failed
java.util.concurrent.CompletionException: java.util.concurrent.TimeoutException: Request raft-partition-partition-4-append-versioned to zeebe-cluster-zeebe-0.zeebe-cluster-zeebe.xxx.svc:26502 timed out in PT2.5S
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$sendAndReceive$4(NettyMessagingService.java:259) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.util.concurrent.TimeoutException: Request raft-partition-partition-4-append-versioned to zeebe-cluster-zeebe-0.zeebe-cluster-zeebe.xxx.svc:26502 timed out in PT2.5S
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$sendAndReceive$4(NettyMessagingService.java:261) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
... 7 more
2024-07-01 10:25:43.651 [Broker-3] [raft-server-3-4] [raft-server-4] WARN
io.atomix.raft.roles.LeaderAppender - RaftServer{raft-partition-partition-4} - VersionedAppendRequest{version=2, term=12, leader=3, prevLogIndex=7435703, prevLogTerm=12, entries=1, commitIndex=7435703} to 0 failed
java.util.concurrent.CompletionException: io.atomix.cluster.messaging.MessagingException$ConnectionClosed: Connection RemoteClientConnection{channel=[id: 0x6eb39ccd, L:/192.168.xx.xxx:59990 ! R:zeebe-cluster-zeebe-0.zeebe-cluster-zeebe.xxx.svc.cluster.local/192.168.xx.xxx:26502]} was closed
at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeThrowable(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$executeOnPooledConnection$25(NettyMessagingService.java:626) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) ~[guava-33.1.0-jre.jar:?]
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$executeOnPooledConnection$26(NettyMessagingService.java:624) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(Unknown Source) ~[?:?]
at io.atomix.cluster.messaging.impl.AbstractClientConnection.close(AbstractClientConnection.java:74) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$getOrCreateClientConnection$38(NettyMessagingService.java:702) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1161) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:753) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:729) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:619) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1349) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:755) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$1200(AbstractChannelHandlerContext.java:61) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.AbstractChannelHandlerContext$11.run(AbstractChannelHandlerContext.java:738) ~[netty-transport-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:405) ~[netty-transport-classes-epoll-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.110.Final.jar:4.1.110.Final]
at java.base/java.lang.Thread.run(Unknown Source) [?:?]
Caused by: io.atomix.cluster.messaging.MessagingException$ConnectionClosed: Connection RemoteClientConnection{channel=[id: 0x6eb39ccd, L:/192.168.xx.xxx:59990 ! R:zeebe-cluster-zeebe-0.zeebe-cluster-zeebe.xxx.svc.cluster.local/192.168.xx.xxx:26502]} was closed
at io.atomix.cluster.messaging.impl.AbstractClientConnection.close(AbstractClientConnection.java:76) ~[zeebe-atomix-cluster-8.5.3.jar:8.5.3]
What we need
We need to understand if this is a normal and safely behavior of Zeebe or if we are facing a issue that must be fixed before move to production (could Zeebe break if keep restarting?).