o.grpc.StatusRuntimeException: UNKNOWN

Prashik_Hingaspure · July 27, 2022, 2:45pm

As Camund 8 platform SaaS uses grpc for communication but it throwing RuntimeException unknown
I have checked grpc side and their replay is as follows:
“It sounds like the server returned UNKNOWN with no additional information. If that is a gRPC Java server, that could be caused by the server throwing an exception. I’d suggest looking at the server logs and debugging on server-side. The client has no information as to what went wrong.”

So Can you Check on your side?

23:38:21.135 [grpc-default-executor-0] WARN  io.camunda.zeebe.client.job.poller - Failed to activate jobs for worker task1Worker#task1Service and job type task1-service-worker
io.grpc.StatusRuntimeException: UNKNOWN
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:460)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
	at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.UnsupportedOperationException: null
	at io.grpc.netty.AbstractHttp2Headers.setLong(AbstractHttp2Headers.java:465)
	at io.grpc.netty.AbstractHttp2Headers.setLong(AbstractHttp2Headers.java:26)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:396)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onHeadersRead(DefaultHttp2ConnectionDecoder.java:337)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onHeadersRead(Http2InboundFrameLogger.java:56)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:476)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:484)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1284)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	... 1 common frames omitted

jwulf · July 27, 2022, 7:49pm

Hi @Prashik_Hingaspure

I haven’t seen this error before.

Could you do something to help debug this please. Would you have that worker code do a call to Topology to get the cluster status.

You can add some code like this:

final Topology topology = client.newTopologyRequest().send().join();

System.out.println("Topology:");
topology
 .getBrokers()
 .forEach(
   b -> {
    System.out.println("    " + b.getAddress());
    b.getPartitions()
      .forEach(
        p ->
          System.out.println(
            "      " + p.getPartitionId() + " - " + p.getRole()));
   });

Josh

Prashik_Hingaspure · August 1, 2022, 12:29pm

Topology:
zeebe-2.zeebe-broker-service.ac4f9d58-958b-4e9c-9555-0c6cb45701b5-zeebe.svc.cluster.local:26501
1 - FOLLOWER
2 - LEADER
zeebe-0.zeebe-broker-service.ac4f9d58-958b-4e9c-9555-0c6cb45701b5-zeebe.svc.cluster.local:26501
1 - LEADER
2 - FOLLOWER
zeebe-1.zeebe-broker-service.ac4f9d58-958b-4e9c-9555-0c6cb45701b5-zeebe.svc.cluster.local:26501
1 - FOLLOWER
2 - FOLLOWER

Prashik_Hingaspure · August 1, 2022, 12:30pm

Got any clue?

jwulf · August 2, 2022, 1:46am

Hi @Prashik_Hingaspure

This demonstrates that the connection is viable and working.

Does your worker always throw this error, or sometimes throw this error? Is it able to activate any jobs at all?

What happens if you use the client.activateJobs command? Does this throw every time?

Josh

Zelldon · August 5, 2022, 9:50am

Which version is your client?

Prashik_Hingaspure · August 9, 2022, 5:49am

Yeah its working fine. But some other part of code which belongs to PUB-SUB messaging support was causing such error as it also uses gRPC framework. But its shocking that some other part is throwing warnings for camunda workers (those marked with @ZeebeWorker annotations) may be that’s because zeebe client also use gRPC framework.