Description
Used spring-zeebe-starter: 0.23.0.
We have 2 simple zeebe tests for processing input and output files, each test has its own process flow as well. The idea of tests is simple:
- get a client via ZeebeTestRule;
- register a flow using client.newDeployCommand();
- mock required objects using Mockito;
- create WorkflowInstanceResult and register corresponding workers;
- make assertions abiut changed variables.
So we have 2 tests in different Java classes, both tests pass ok locally in Maven. The jobs activated step-by-step and processed accordingly without errors.
Problems start when we run it in CI/CD. According to log zeebe tests fails randomly because it can pass some of the jobs, e.g:
io.zeebe.client.api.command.ClientStatusException: Time out between gateway and broker: Request type command-api-1 timed out in 2000 milliseconds
at com.epam.vtbbulpm.lfcworker.zeebe.InboundFileHandlerTest.successInboundFileScenario_test(InboundFileHandlerTest.java:87)
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Time out between gateway and broker: Request type command-api-1 timed out in 2000 milliseconds
Line 17296: 11:02:16.141 [Broker-0-zb-actors-0] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"file-formatcheck-task","worker":"default","timeout":300000,"maxJobsToActivate":32,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type file-formatcheck-task after timeout of PT2S
Line 17297: 11:02:16.154 [Broker-0-zb-actors-0] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"file-unzip-task","worker":"default","timeout":300000,"maxJobsToActivate":32,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type file-unzip-task after timeout of PT2S
Line 17429: 11:02:16.740 [Broker-0-zb-actors-1] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"signature-check-task","worker":"default","timeout":300000,"maxJobsToActivate":31,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type signature-check-task after timeout of PT2S
Line 17467: 11:02:17.865 [Broker-0-zb-actors-0] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"file-decrypt-task","worker":"default","timeout":300000,"maxJobsToActivate":32,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type file-decrypt-task after timeout of PT2S
Line 17470: 11:02:18.361 [Broker-0-zb-actors-0] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"file-unzip-task","worker":"default","timeout":300000,"maxJobsToActivate":32,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type file-unzip-task after timeout of PT2S
Line 17471: 11:02:18.442 [Broker-0-zb-actors-0] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"file-formatcheck-task","worker":"default","timeout":300000,"maxJobsToActivate":32,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type file-formatcheck-task after timeout of PT2S
Line 17936: 11:02:26.493 [Broker-0-zb-actors-0] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"file-zip-task","worker":"default","timeout":300000,"maxJobsToActivate":32,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type file-zip-task after timeout of PT2S
Line 17937: 11:02:26.830 [Broker-0-zb-actors-0] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"file-encrypt-task","worker":"default","timeout":300000,"maxJobsToActivate":32,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type file-encrypt-task after timeout of PT2S
Line 17940: 11:02:27.209 [Broker-0-zb-actors-0] DEBUG io.zeebe.gateway - Remove blocking request BrokerActivateJobsRequest{requestDto={"type":"signature-attach-task","worker":"default","timeout":300000,"maxJobsToActivate":32,"jobKeys":[],"jobs":[],"variables":[],"truncated":false}} for job type signature-attach-task after timeout of PT2S
Between starting a zeebe broker and zeebe gateway there is some strange error with netty:
15:01:39.431 [Time-limited test] DEBUG io.netty.util.internal.PlatformDependent0 - direct buffer constructor: unavailable
java.lang.UnsupportedOperationException: Reflective setAccessible(true) disabled
at io.netty.util.internal.ReflectionUtil.trySetAccessible(ReflectionUtil.java:31)
at io.netty.util.internal.PlatformDependent0$4.run(PlatformDependent0.java:225)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:219)
at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:273)
at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:92)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:228)
at io.netty.channel.epoll.Native.<clinit>(Native.java:58)
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:51)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:150)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:112)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:99)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:76)
at io.atomix.cluster.messaging.impl.NettyMessagingService.initEventLoopGroup(NettyMessagingService.java:387)
at io.atomix.cluster.messaging.impl.NettyMessagingService.start(NettyMessagingService.java:255)
at io.zeebe.broker.Broker.commandApiTransportStep(Broker.java:226)
at io.zeebe.broker.Broker.lambda$initStart$1(Broker.java:164)
at io.zeebe.broker.bootstrap.StartProcess.lambda$startStepByStep$2(StartProcess.java:60)
at io.zeebe.broker.bootstrap.StartProcess.takeDuration(StartProcess.java:88)
at io.zeebe.broker.bootstrap.StartProcess.startStepByStep(StartProcess.java:58)
at io.zeebe.broker.bootstrap.StartProcess.takeDuration(StartProcess.java:88)
at io.zeebe.broker.bootstrap.StartProcess.start(StartProcess.java:43)
at io.zeebe.broker.Broker.internalStart(Broker.java:135)
at io.zeebe.util.LogUtil.doWithMDC(LogUtil.java:21)
at io.zeebe.broker.Broker.start(Broker.java:115)
at io.zeebe.test.EmbeddedBrokerRule.startBroker(EmbeddedBrokerRule.java:191)
at io.zeebe.test.EmbeddedBrokerRule.before(EmbeddedBrokerRule.java:132)
at io.zeebe.test.ZeebeTestRule.before(ZeebeTestRule.java:70)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
15:01:39.433 [Time-limited test] DEBUG io.netty.util.internal.PlatformDependent0 - java.nio.Bits.unaligned: available, true
15:01:39.458 [Time-limited test] DEBUG io.netty.util.internal.PlatformDependent0 - jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable
java.lang.IllegalAccessException: class io.netty.util.internal.PlatformDependent0$6 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @eb21112
at java.base/jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:361)
at java.base/java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:591)
at java.base/java.lang.reflect.Method.invoke(Method.java:558)
at io.netty.util.internal.PlatformDependent0$6.run(PlatformDependent0.java:335)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.PlatformDependent0.<clinit>(PlatformDependent0.java:326)
at io.netty.util.internal.PlatformDependent.isAndroid(PlatformDependent.java:273)
at io.netty.util.internal.PlatformDependent.<clinit>(PlatformDependent.java:92)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:228)
at io.netty.channel.epoll.Native.<clinit>(Native.java:58)
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:51)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:150)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:112)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:99)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:76)
at io.atomix.cluster.messaging.impl.NettyMessagingService.initEventLoopGroup(NettyMessagingService.java:387)
at io.atomix.cluster.messaging.impl.NettyMessagingService.start(NettyMessagingService.java:255)
at io.zeebe.broker.Broker.commandApiTransportStep(Broker.java:226)
at io.zeebe.broker.Broker.lambda$initStart$1(Broker.java:164)
at io.zeebe.broker.bootstrap.StartProcess.lambda$startStepByStep$2(StartProcess.java:60)
at io.zeebe.broker.bootstrap.StartProcess.takeDuration(StartProcess.java:88)
at io.zeebe.broker.bootstrap.StartProcess.startStepByStep(StartProcess.java:58)
at io.zeebe.broker.bootstrap.StartProcess.takeDuration(StartProcess.java:88)
at io.zeebe.broker.bootstrap.StartProcess.start(StartProcess.java:43)
at io.zeebe.broker.Broker.internalStart(Broker.java:135)
at io.zeebe.util.LogUtil.doWithMDC(LogUtil.java:21)
at io.zeebe.broker.Broker.start(Broker.java:115)
at io.zeebe.test.EmbeddedBrokerRule.startBroker(EmbeddedBrokerRule.java:191)
at io.zeebe.test.EmbeddedBrokerRule.before(EmbeddedBrokerRule.java:132)
at io.zeebe.test.ZeebeTestRule.before(ZeebeTestRule.java:70)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
and:
15:01:39.529 [Time-limited test] DEBUG io.netty.util.internal.NativeLibraryLoader - Unable to load the library 'netty_transport_native_epoll_x86_64', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2670)
at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:830)
at java.base/java.lang.System.loadLibrary(System.java:1870)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:231)
at io.netty.channel.epoll.Native.<clinit>(Native.java:58)
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:51)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:150)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:112)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:99)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:76)
at io.atomix.cluster.messaging.impl.NettyMessagingService.initEventLoopGroup(NettyMessagingService.java:387)
at io.atomix.cluster.messaging.impl.NettyMessagingService.start(NettyMessagingService.java:255)
at io.zeebe.broker.Broker.commandApiTransportStep(Broker.java:226)
at io.zeebe.broker.Broker.lambda$initStart$1(Broker.java:164)
at io.zeebe.broker.bootstrap.StartProcess.lambda$startStepByStep$2(StartProcess.java:60)
at io.zeebe.broker.bootstrap.StartProcess.takeDuration(StartProcess.java:88)
at io.zeebe.broker.bootstrap.StartProcess.startStepByStep(StartProcess.java:58)
at io.zeebe.broker.bootstrap.StartProcess.takeDuration(StartProcess.java:88)
at io.zeebe.broker.bootstrap.StartProcess.start(StartProcess.java:43)
at io.zeebe.broker.Broker.internalStart(Broker.java:135)
at io.zeebe.util.LogUtil.doWithMDC(LogUtil.java:21)
at io.zeebe.broker.Broker.start(Broker.java:115)
at io.zeebe.test.EmbeddedBrokerRule.startBroker(EmbeddedBrokerRule.java:191)
at io.zeebe.test.EmbeddedBrokerRule.before(EmbeddedBrokerRule.java:132)
at io.zeebe.test.ZeebeTestRule.before(ZeebeTestRule.java:70)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
15:01:39.530 [Time-limited test] DEBUG io.netty.util.internal.NativeLibraryLoader - netty_transport_native_epoll_x86_64 cannot be loaded from java.library.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2670)
at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:830)
at java.base/java.lang.System.loadLibrary(System.java:1870)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:231)
at io.netty.channel.epoll.Native.<clinit>(Native.java:58)
at io.netty.channel.epoll.Epoll.<clinit>(Epoll.java:39)
at io.netty.channel.epoll.EpollEventLoop.<clinit>(EpollEventLoop.java:51)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:150)
at io.netty.channel.epoll.EpollEventLoopGroup.newChild(EpollEventLoopGroup.java:35)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47)
at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:112)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:99)
at io.netty.channel.epoll.EpollEventLoopGroup.<init>(EpollEventLoopGroup.java:76)
at io.atomix.cluster.messaging.impl.NettyMessagingService.initEventLoopGroup(NettyMessagingService.java:387)
at io.atomix.cluster.messaging.impl.NettyMessagingService.start(NettyMessagingService.java:255)
at io.zeebe.broker.Broker.commandApiTransportStep(Broker.java:226)
at io.zeebe.broker.Broker.lambda$initStart$1(Broker.java:164)
at io.zeebe.broker.bootstrap.StartProcess.lambda$startStepByStep$2(StartProcess.java:60)
at io.zeebe.broker.bootstrap.StartProcess.takeDuration(StartProcess.java:88)
at io.zeebe.broker.bootstrap.StartProcess.startStepByStep(StartProcess.java:58)
at io.zeebe.broker.bootstrap.StartProcess.takeDuration(StartProcess.java:88)
at io.zeebe.broker.bootstrap.StartProcess.start(StartProcess.java:43)
at io.zeebe.broker.Broker.internalStart(Broker.java:135)
at io.zeebe.util.LogUtil.doWithMDC(LogUtil.java:21)
at io.zeebe.broker.Broker.start(Broker.java:115)
at io.zeebe.test.EmbeddedBrokerRule.startBroker(EmbeddedBrokerRule.java:191)
at io.zeebe.test.EmbeddedBrokerRule.before(EmbeddedBrokerRule.java:132)
at io.zeebe.test.ZeebeTestRule.before(ZeebeTestRule.java:70)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path: [/usr/java/packages/lib, /usr/lib64, /lib64, /lib, /usr/lib]
at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2670)
at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:830)
at java.base/java.lang.System.loadLibrary(System.java:1870)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 34 common frames omitted
15:01:39.607 [Time-limited test] DEBUG io.netty.util.internal.NativeLibraryLoader - Successfully loaded the library /tmp/libnetty_transport_native_epoll_x86_648715979242851843071.so
15:01:39.625 [Time-limited test] DEBUG io.netty.util.NetUtil - -Djava.net.preferIPv4Stack: false
But it seems that required library is loaded at all.
Do you have any ideas why is that and how to solve it?