ZeebeClient failing with Http2Exception when trying to connect to zeebecluster in aws eks environment

venu.bora · March 30, 2023, 2:21am

We deployed Camunda 8 self managed (8.1.6) in aws eks environment and able to get the topology information with following
grpcurl -insecure -v -import-path /tmp -proto /tmp/gateway.proto <> gateway_protocol.Gateway/Topology

Create a spring application with:
Java 12,
implementation ‘io.camunda:spring-zeebe-starter:8.1.6’
implementation ‘io.camunda:zeebe-client-java:8.1.6’

Getting http2exception when trying to use ZeebeClient to communicate with zeebe cluster to get the same topology.

io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109) ~[netty-codec-http2-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:353) ~[netty-codec-http2-4.1.87.Final.jar:4.1.87.Final]
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:247) ~[netty-codec-http2-4.1.87.Final.jar:4.1.87.Final]

Could Please help me in identify the issue.

StephanHaarmann · March 30, 2023, 11:53am

Hi @venu.bora,

After searching for your error online, I believe it occurs if you try to communicate to HTTP/1.1 server. gRPC requires HTTP/2. How did you expose your Camunda 8 installation? Do you use Camunda’s Helm charts for deployments? Do you run grpcurl and the ZeebeClient from the same machine?

venu.bora · March 30, 2023, 1:09pm

Camunda 8 is installed using helm charts. Yes. I am running the grpcurl and zeebeclient from the same machine. From grpcurl and zeebeclient we are hitting the same ALB

Response from grpcurl command:
Response headers received:
content-type: application/grpc
date: Thu, 30 Mar 2023 02:15:33 GMT
grpc-accept-encoding: gzip
server: awselb/2.0

Response contents:
{
“brokers”: [
{
“nodeId”: 1,
“host”: “c8-zeebe-1.c8-zeebe.c8.svc”,
“port”: 26501,
“partitions”: [
{
“partitionId”: 1
},
{
“partitionId”: 2
},
{
“partitionId”: 3,
“role”: “FOLLOWER”
}
],
“version”: “8.1.6”
},
{
“nodeId”: 2,
“host”: “c8-zeebe-2.c8-zeebe.c8.svc”,
“port”: 26501,
“partitions”: [
{
“partitionId”: 1,
“role”: “FOLLOWER”
},
{
“partitionId”: 2,
“role”: “FOLLOWER”
},
{
“partitionId”: 3
}
],
“version”: “8.1.6”
},
{
“host”: “c8-zeebe-0.c8-zeebe.c8.svc”,
“port”: 26501,
“partitions”: [
{
“partitionId”: 1,
“role”: “FOLLOWER”
},
{
“partitionId”: 2,
“role”: “FOLLOWER”
},
{
“partitionId”: 3,
“role”: “FOLLOWER”
}
],
“version”: “8.1.6”
}
],
“clusterSize”: 3,
“partitionsCount”: 3,
“replicationFactor”: 3,
“gatewayVersion”: “8.1.6”
}

StephanHaarmann · March 30, 2023, 1:30pm

Well, I can only guess. The error message indicates that the client receives an HTTP/1.1 response for an HTTP/2 request. There may be some misconfiguration in the ALB or the Ingress. If you haven’t already, check out this guide:

venu.bora · March 30, 2023, 4:22pm

Will get that checked. I turned on debug logs at package io.grpc.netty and saw these debug message while starting the spring application. Anything of this would have effected ?

2023-03-30 11:17:36.959 DEBUG 11151 — [ 2023-03-30 11:17:36.962 DEBUG 11151 — [ 2023-03-30 11:17:36.962 DEBUG 11151 — [ 2023-03-30 11:17:36.963 DEBUG 11151 — [ 2023-03-30 11:17:36.964 DEBUG 11151 — [ 2023-03-30 11:17:36.964 DEBUG 11151 — [ 2023-03-30 11:17:36.965 DEBUG 11151 — [ 2023-03-30 11:17:36.965 DEBUG 11151 — [ 2023-03-30 11:17:36.966 DEBUG 11151 — [ 2023-03-30 11:17:36.967 DEBUG 11151 — [ 2023-03-30 11:17:36.968 DEBUG 11151 — [ 2023-03-30 11:17:36.968 DEBUG 11151 — [ 2023-03-30 11:17:36.969 DEBUG 11151 — [ 2023-03-30 11:17:36.969 DEBUG 11151 — [ 2023-03-30 11:17:36.970 DEBUG 11151 — [ 2023-03-30 11:17:36.970 DEBUG 11151 — [ 2023-03-30 11:17:36.971 DEBUG 11151 — [ 2023-03-30 11:17:36.971 DEBUG 11151 — [ 2023-03-30 11:17:36.972 DEBUG 11151 — [ 2023-03-30 11:17:36.972 DEBUG 11151 — [ 2023-03-30 11:17:37.018 DEBUG 11151 — [ main] i.n.u.i.logging.InternalLoggerFactory : Using SLF4J as the default logging framework
main] i.n.util.internal.PlatformDependent0 : -Dio.netty.noUnsafe: false
main] i.n.util.internal.PlatformDependent0 : Java version: 12
main] i.n.util.internal.PlatformDependent0 : sun.misc.Unsafe.theUnsafe: available
main] i.n.util.internal.PlatformDependent0 : sun.misc.Unsafe.copyMemory: available
main] i.n.util.internal.PlatformDependent0 : sun.misc.Unsafe.storeFence: available
main] i.n.util.internal.PlatformDependent0 : java.nio.Buffer.address: available
main] i.n.util.internal.PlatformDependent0 : direct buffer constructor: unavailable: Reflective setAccessible(true) disabled
main] i.n.util.internal.PlatformDependent0 : java.nio.Bits.unaligned: available, true
main] i.n.util.internal.PlatformDependent0 : jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable: class io.netty.util.internal.PlatformDependent0$7 cannot access class jdk.internal.misc.Unsafe (in module java.base) because module java.base does not export jdk.internal.misc to unnamed module @6e28bb87
main] i.n.util.internal.PlatformDependent0 : java.nio.DirectByteBuffer.(long, int): unavailable
main] i.netty.util.internal.PlatformDependent : sun.misc.Unsafe: available
main] i.netty.util.internal.PlatformDependent : maxDirectMemory: 4294967296 bytes (maybe)
main] i.netty.util.internal.PlatformDependent : -Dio.netty.tmpdir: /var/folders/yx/9fq035697blfc17dt1n1st1h0000gn/T (java.io.tmpdir)
main] i.netty.util.internal.PlatformDependent : -Dio.netty.bitMode: 64 (sun.arch.data.model)
main] i.netty.util.internal.PlatformDependent : Platform: MacOS
main] i.netty.util.internal.PlatformDependent : -Dio.netty.maxDirectMemory: -1 bytes
main] i.netty.util.internal.PlatformDependent : -Dio.netty.uninitializedArrayAllocationThreshold: -1
main] io.netty.util.internal.CleanerJava9 : java.nio.ByteBuffer.cleaner(): available
main] i.netty.util.internal.PlatformDependent : -Dio.netty.noPreferDirect: false
main] io.grpc.netty.Utils : Epoll is not available, using Nio.

java.lang.ClassNotFoundException: io.netty.channel.epoll.Epoll
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583) ~[na:na]
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) ~[na:na]
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ~[na:na]
at java.base/java.lang.Class.forName0(Native Method) ~[na:na]
at java.base/java.lang.Class.forName(Class.java:332) ~[na:na]
at io.grpc.netty.Utils.getEpollUnavailabilityCause(Utils.java:314) ~[grpc-netty-1.49.1.jar:1.49.1]
at io.grpc.netty.Utils.(Utils.java:125) ~[grpc-netty-1.49.1.jar:1.49.1]
at io.grpc.netty.NettyChannelBuilder.(NettyChannelBuilder.java:83) ~[grpc-netty-1.49.1.jar:1.49.1]
at io.camunda.zeebe.client.impl.ZeebeClientImpl.buildChannel(ZeebeClientImpl.java:129) ~[zeebe-client-java-8.1.6.jar:8.1.6]
at io.camunda.zeebe.client.impl.ZeebeClientImpl.(ZeebeClientImpl.java:85) ~[zeebe-client-java-8.1.6.jar:8.1.6]
at io.camunda.zeebe.client.impl.ZeebeClientBuilderImpl.build(ZeebeClientBuilderImpl.java:314) ~[zeebe-client-java-8.1.6.jar:8.1.6]
at io.camunda.zeebe.spring.client.ZeebeClientSpringConfiguration$1.getObject(ZeebeClientSpringConfiguration.java:39) ~[spring-zeebe-8.1.6.jar:8.1.6]
at io.camunda.zeebe.spring.client.ZeebeClientSpringConfiguration$1.getObject(ZeebeClientSpringConfiguration.java:35) ~[spring-zeebe-8.1.6.jar:8.1.6]
at io.camunda.zeebe.spring.client.lifecycle.ZeebeClientLifecycle.start(ZeebeClientLifecycle.java:42) ~[spring-zeebe-8.1.6.jar:8.1.6]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-5.3.25.jar:5.3.25]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.25.jar:5.3.25]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.25.jar:5.3.25]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.25.jar:5.3.25]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.25.jar:5.3.25]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring-context-5.3.25.jar:5.3.25]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring-context-5.3.25.jar:5.3.25]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.8.jar:2.7.8]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:731) ~[spring-boot-2.7.8.jar:2.7.8]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) ~[spring-boot-2.7.8.jar:2.7.8]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) ~[spring-boot-2.7.8.jar:2.7.8]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303) ~[spring-boot-2.7.8.jar:2.7.8]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292) ~[spring-boot-2.7.8.jar:2.7.8]
at com.epsilon.test.TestZeebeClientApplication.main(TestZeebeClientApplication.java:16) ~[main/:na]

2023-03-30 11:17:37.086 DEBUG 11151 — [ main] i.n.channel.MultithreadEventLoopGroup : -Dio.netty.eventLoopThreads: 24
2023-03-30 11:17:37.103 DEBUG 11151 — [ main] i.n.u.internal.InternalThreadLocalMap : -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2023-03-30 11:17:37.103 DEBUG 11151 — [ main] i.n.u.internal.InternalThreadLocalMap : -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2023-03-30 11:17:37.109 DEBUG 11151 — [ main] io.netty.channel.nio.NioEventLoop : -Dio.netty.noKeySetOptimization: false
2023-03-30 11:17:37.109 DEBUG 11151 — [ main] io.netty.channel.nio.NioEventLoop : -Dio.netty.selectorAutoRebuildThreshold: 512
2023-03-30 11:17:37.117 DEBUG 11151 — [ main] i.netty.util.internal.PlatformDependent : org.jctools-core.MpscChunkedArrayQueue: available
2023-03-30 11:17:37.165 INFO 11151 — [ main] a.p.OutboundConnectorAnnotationProcessor : Reading environment variables or parsing SPI to find connectors that are not Spring beans
2023-03-30 11:17:37.167 INFO 11151 — [ main] .u.o.OutboundConnectorRegistrationHelper : Parsing SPI to find connectors that are not Spring beans
2023-03-30 11:17:37.168 WARN 11151 — [ main] a.p.OutboundConnectorAnnotationProcessor : No outbound connectors configured or found in current Connector Runtime’s classpath
2023-03-30 11:17:37.190 INFO 11151 — [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8092 (http) with context path ‘/test-zeebe-client’