Huge working load in partition 3 and throw warnning called failed to activate jobs

hey Team~

env

zeebe 8.2.0
only one zeebe node
centos 7; 8vcpu 64G

questions

I saw a huge working load in partition 3 about 2.4M but only less then 100 in other paritions.

In my service, I have about 1000+(less then 1500) process instance to create every day and about 10000 job to activated.

May I kindly inquire if this situation is considered normal, and if so, could you please advise me on how to address the warning?

stacktrace

zeebe-1  | 2023-09-05 02:13:20.616 [ActivateJobsHandler] [Broker-0-zb-actors-4] WARN 
dev-zeebe-1  |       io.camunda.zeebe.gateway - Failed to activate jobs for type py-script from partition 3
dev-zeebe-1  | java.net.ConnectException: Connection RemoteClientConnection{channel=[id: 0xfefda59e, L:/192.168.80.8:44532 ! R:192.168.80.8/192.168.80.8:26501]} was closed
dev-zeebe-1  | 	at io.atomix.cluster.messaging.impl.AbstractClientConnection.close(AbstractClientConnection.java:76) ~[zeebe-atomix-cluster-8.2.0.jar:8.2.0]
dev-zeebe-1  | 	at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$getOrCreateClientConnection$36(NettyMessagingService.java:665) ~[zeebe-atomix-cluster-8.2.0.jar:8.2.0]
dev-zeebe-1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1164) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:755) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:731) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:620) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.DefaultChannelPipeline$HeadContext.close(DefaultChannelPipeline.java:1352) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.AbstractChannelHandlerContext.invokeClose(AbstractChannelHandlerContext.java:749) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.AbstractChannelHandlerContext.access$1200(AbstractChannelHandlerContext.java:61) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.AbstractChannelHandlerContext$11.run(AbstractChannelHandlerContext.java:732) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403) ~[netty-transport-classes-epoll-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
dev-zeebe-1  | 	at java.lang.Thread.run(Unknown Source) ~[?:?]

hi, now i meet the same question, Have you resolved this question?

Hi @1015428318

I don’t know exactly why huge working load happening in one partition. You can see I have 10000 job to activate, that’s a little bit much. I reduced the number of job to activate by changing my program, and then the working load begin to decline.