Incident not being raised with newFailCommand

Camunda 8 Topics
1

Hi, I am using the newFailComand to send a technical error to Zeebe from a job worker, but an incident is not being raised, and the task is being completed and the process token moves to the next task.

The sample code is below.

  private void finalizeWithTechnicalError(
      JobClient client, ActivatedJob job, Exception exception) {

    client.newFailCommand(job.getKey()).retries(0).errorMessage(exception.getMessage()).send().join();

  }

Some aditional comments on this situation:

  • I have one Zeebe Cluster running, and when I execute the job worker code in my local machine it generates the incident (as expected)
  • But when I deploy the job worker in the same EKS that Zeebe is running, I get this behavior of an incident not being raised.

I have searched the Gateway and Broker logs to try to find something that could point to the problem, but on the Broker logs there is no errors, on the Gateway the message below is printed every second, but it does not seem to be related.

Dec 11, 2023 6:41:46 PM io.grpc.netty.NettyServerHandler onDataRead
WARNING: Exception in onDataRead()
java.lang.NullPointerException

Dec 11, 2023 6:41:46 PM io.grpc.netty.NettyServerHandler onStreamError
WARNING: Stream Error
io.netty.handler.codec.http2.Http2Exception$StreamException: 
	at io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:173)
	at io.grpc.netty.NettyServerHandler.newStreamException(NettyServerHandler.java:827)
	at io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:525)
	at io.grpc.netty.NettyServerHandler.access$900(NettyServerHandler.java:110)
	at io.grpc.netty.NettyServerHandler$FrameListener.onHeadersRead(NettyServerHandler.java:868)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:409)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException

Is there anything that I am missing here to have this situation ?

Regards,

Fernando Taniguti

2

Hi @ftaniguti,

do you use spring-zeebe as the framework for your worker?

If yes, it could be that autocomplete of your worker is faster than the fail command if you haven’t explicitly disabled it.

With autocomplete enabled, it is sufficient to throw an exception in the worker code to send the fail command.

Hope this helps, Ingo

3

@Ingo_Richtsmeier, adding the configuration of autocomplete to false solved the issue of incident not being raised.

Is there any reference that explains in what scenarios this unexpected sequence of Zeebe events can take place ?

Thanks!

Fernando

4

Hi @ftaniguti,

you can find more details here: GitHub - camunda-community-hub/spring-zeebe: Easily use the Zeebe Java Client in your Spring or Spring Boot projects

The last sentence of this section describes your situation.

Hope this helps, Ingo