Http2 exception

Camunda 8 Topics Discussion & Questions
1

Hi , i have java client connect to gateway zeebe (Self-Managed - built on my cloud server) over load balancer of k8s ingress, it throw exception INTERNAL: http2 exception:

org.springframework.context.ApplicationContextException: Failed to start bean 'zeebeClientLifecycle'
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155)
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:934)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:587)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:310)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1304)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1293)
at io.camunda.zeebe.spring.example.ExampleApplication.main(ExampleApplication.java:16)
Caused by: io.camunda.zeebe.client.api.command.ClientStatusException: http2 exception
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:93)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:50)
at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.lambda$start$7(ZeebeDeploymentAnnotationProcessor.java:119)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.start(ZeebeDeploymentAnnotationProcessor.java:100)
at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.lambda$startAll$0(ZeebeAnnotationProcessorRegistry.java:38)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.startAll(ZeebeAnnotationProcessorRegistry.java:38)
at io.camunda.zeebe.spring.client.lifecycle.ZeebeClientLifecycle.start(ZeebeClientLifecycle.java:49)
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178)
... 12 common frames omitted
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: INTERNAL: http2 exception
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:48)
... 20 common frames omitted
Caused by: io.grpc.StatusRuntimeException: INTERNAL: http2 exception
at io.grpc.Status.asRuntimeException(Status.java:535)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:487)
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:470)
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:434)
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:467)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:563)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:744)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:353)
at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:247)
at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted

Spring boot version: 2.7.7
Zeebe version: 8.1.5

This is my config ingress annotation k8s:

Screen Shot 2023-04-04 at 10.42.38
Screen Shot 2023-04-04 at 10.42.381916×540 36.9 KB

Screen Shot 2023-04-04 at 10.42.46
Screen Shot 2023-04-04 at 10.42.461898×832 63 KB

Anyone know resolve this issuses, thanks!!

2

This is my config of zeebe client:

zeebe.client.security.plaintext=true
zeebe.client.broker.gateway-address=bpm.gogroup.vn/zeebe

Iam using plaintext, to solve this problem do I need to use SSL?

3

The error indicates that the load balancer/ingress is returning an HTTP1 response. However, gRPC requires HTTP2. This being said, I cannot see any error in your configuration.
Have you tried to access the gateway via another client, for example, zbctl? If so, did it work?

4

When i tried zbctl command, it’ throw error:
MacBook-Pro-2:~ MinhDV$ export ZEEBE_ADDRESS=bpm.gogroup.vn/zeebe
MacBook-Pro-2:~ MinhDV$ zbctl status
Error: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp: address bpm.gogroup.vn/zeebe: missing port in address”

5

‘’’
upstream zeebe_cluster {
least_conn;
server gateway:26500;
server gateway2:26500;
}

Zeebe loadbalancer

server {
listen 443 http2 ssl;

proxy_connect_timeout 30;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
keepalive_timeout 600;

ssl_certificate /ssl/zeebe.crt;
ssl_certificate_key /ssl/zeebe.key;
ssl_dhparam /ssl/dhparam.pem;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_protocols TLSv1.3 TLSv1.2;
add_header Strict-Transport-Security ‘max-age=31536000; includeSubDomains’;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 24h;
proxy_ssl_session_reuse on;

location / {
#grpc_pass grpcs://zeebe_cluster;
grpc_pass grpc://zeebe_cluster;
}
}
‘’’
Nginx manual setup for loadbalancing gprc grpcs traffic

1 Like
6