Failed to activated jobs for worker warnings after upgrading Zeebe to 1.x

Hello,

After upgrading Zeebe from 0.x to 1.x, I had to make a change in the Zeebe client build,

Earlier,

ZeebeClient.newClientBuilder()
                 .brokerContactPoint(zeebeBrokerContactpoint)
                 .usePlaintext()
                 .numJobWorkerExecutionThreads(zeebeClientMaxThreads)
                 .build();

After Zeebe 1.x.

return ZeebeClient.newClientBuilder()
                 .gatewayAddress(zeebeBrokerContactpoint)
                 .usePlaintext()
                 .numJobWorkerExecutionThreads(zeebeClientMaxThreads)
                 .build();

now the logs are full of warning stating:

03:51:58.627  WARN [          grpc-default-executor-486] i.c.zeebe.client.job.poller  : Failed to activated jobs for worker notify-ams-failure and job │
│ io.grpc.StatusRuntimeException: INTERNAL: Panic! This is a bug!                                                                                        │
│     at io.grpc.Status.asRuntimeException(Status.java:535)                                                                                              │
│     at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478)                                                      │
│     at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:553)                                                                          │
│     at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:68)                                                                              │
│     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:739)                                     │
│     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:718)                                    │
│     at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)                                                                                   │
│     at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)                                                                          │
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)                                                       │
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)                                                       │
│     at java.base/java.lang.Thread.run(Thread.java:830)                                                                                                 │
│ Caused by: java.lang.NoSuchMethodError: 'void io.netty.buffer.PooledByteBufAllocator.<init>(boolean, int, int, int, int, int, int, boolean)'           │
│     at io.grpc.netty.Utils.createByteBufAllocator(Utils.java:172)                                                                                      │
│     at io.grpc.netty.Utils.access$000(Utils.java:71)                                                                                                   │
│     at io.grpc.netty.Utils$ByteBufAllocatorPreferDirectHolder.<clinit>(Utils.java:93)                                                                  │
│     at io.grpc.netty.Utils.getByteBufAllocator(Utils.java:140)                                                                                         │
│     at io.grpc.netty.NettyClientTransport.start(NettyClientTransport.java:231)                                                                         │
│     at io.grpc.internal.ForwardingConnectionClientTransport.start(ForwardingConnectionClientTransport.java:33)                                         │
│     at io.grpc.internal.ForwardingConnectionClientTransport.start(ForwardingConnectionClientTransport.java:33)                                         │
│     at io.grpc.internal.InternalSubchannel.startNewTransport(InternalSubchannel.java:258)                                                              │
│     at io.grpc.internal.InternalSubchannel.access$400(InternalSubchannel.java:65)                                                                      │
│     at io.grpc.internal.InternalSubchannel$2.run(InternalSubchannel.java:200)                                                                          │
│     at io.grpc.SynchronizationContext.drain(SynchronizationContext.java:95) 

Any insight on what could be going wrong?

Thanks,

Hi @SubhamPramanik,

this sounds like a problem in the dependencies with Netty. Please share the build descriptor (i.e. pom.xml) of your project.

Best regards,
Philipp

Hi @philipp.ossler,

I thought so as well but we’re not really using Netty in our services.
Here’s the pom.xml where the worker connection is failing: channel-pom - Pastebin.com
And another pom.xml where things are working fine after the Zeebe 1.x upgrade (at least we see no warnings): bulk-processor-pom - Pastebin.com

Additional note: we’re using Apache Camel with undertow.

Thanks,
Subham

Thanks for sharing. I recommend generating the Maven dependency tree. It should show if there is a conflict in the Netty dependency.

I took a look into dependency tree and I don’t see a conflict in the dependency but I can see the service which is not working is running Netty v4.1.43 and the working service is using v4.1.65.

Would the patch version difference matter?

Here’s the dependency tree for reference,
Failing service: channel-dep-tree - Pastebin.com
Working service: bulk-processor-dep-tree - Pastebin.com

Would the patch version difference matter?

Usually, it is not expected but we saw this kind of dependency conflict with Netty before (here).

You could either update all dependencies to the latest versions. It seems that the old version comes from com.azure*.

Or, import a fixed Netty version.

      <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-bom</artifactId>
        <version>4.1.65.Final</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>

Thanks for sharing. Last night I added the Netty v4.1.65 in dependencyManagement and it seems to have fixed the issue. Error was indeed due to the lower version.

In another service where we’re not using Azure/AWS dependency, Zeebe and gRPC are the only ones using Netty, it was using v4.1.43 as well. So, I’m not entirely what is causing the em to use the lower version.
Here’s the dep tree for that service: gsma-dep-tree - Pastebin.com

Thank you!

:+1:

It seems that the Netty version comes together with

org.apache.camel.springboot:camel-spring-boot-starter:jar:3.4.0
--> org.springframework.boot:spring-boot-starter:2.3.0.RELEASE
--> org.springframework.boot:spring-boot-dependencies:2.3.0.RELEASE
--> io.netty:netty-bom:4.1.49.Final

Ah nice catch. Completely missed that. Thanks for your help!