Internal Http2 Exception from netty codec: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f

Jason7 · November 21, 2023, 10:07am

I am doing a POC on spring boot and for local development have created a new local setup using docker compose as detailed here.

I have created a spring boot project with spring-zeebe client through camunda starter. However I am getting following error stack strace.

org.springframework.context.ApplicationContextException: Failed to start bean 'zeebeLifecycleEventProducer'
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-6.0.8.jar:6.0.8]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:934) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:587) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732) [spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) [spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:310) [spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1304) [spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1293) [spring-boot-3.0.5.jar:3.0.5]
	at com.wu.compliance.pharos.workflowengine.ZeebeJobWorker.main(ZeebeJobWorker.java:12) [classes/:?]
Caused by: io.camunda.zeebe.client.api.command.ClientStatusException: http2 exception
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.lambda$start$7(ZeebeDeploymentAnnotationProcessor.java:119) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.start(ZeebeDeploymentAnnotationProcessor.java:100) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.lambda$startAll$0(ZeebeAnnotationProcessorRegistry.java:38) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.startAll(ZeebeAnnotationProcessorRegistry.java:38) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeClientEventListener.handleStart(ZeebeClientEventListener.java:17) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:343) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:228) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:413) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) ~[spring-context-6.0.8.jar:6.0.8]
	at io.camunda.zeebe.spring.client.event.ZeebeLifecycleEventProducer.start(ZeebeLifecycleEventProducer.java:23) ~[spring-boot-starter-camunda-8.3.1.jar:8.3.1]
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-6.0.8.jar:6.0.8]
	... 13 more
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: INTERNAL: http2 exception
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.lambda$start$7(ZeebeDeploymentAnnotationProcessor.java:119) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.start(ZeebeDeploymentAnnotationProcessor.java:100) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.lambda$startAll$0(ZeebeAnnotationProcessorRegistry.java:38) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.startAll(ZeebeAnnotationProcessorRegistry.java:38) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeClientEventListener.handleStart(ZeebeClientEventListener.java:17) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:343) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:228) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:413) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) ~[spring-context-6.0.8.jar:6.0.8]
	at io.camunda.zeebe.spring.client.event.ZeebeLifecycleEventProducer.start(ZeebeLifecycleEventProducer.java:23) ~[spring-boot-starter-camunda-8.3.1.jar:8.3.1]
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-6.0.8.jar:6.0.8]
	... 13 more
Caused by: io.grpc.StatusRuntimeException: INTERNAL: http2 exception
	at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.58.0.jar:1.58.0]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.58.0.jar:1.58.0]
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.58.0.jar:1.58.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109) ~[netty-codec-http2-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:353) ~[netty-codec-http2-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:247) ~[netty-codec-http2-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) ~[netty-codec-http2-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[netty-codec-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[netty-codec-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.90.Final.jar:4.1.90.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

Versions at my end:
Spring-Client-Zeebe : 8.3.1
JDK : 17
Spring boot : 3.0.5

Based on some analysis it seems that because HTTP version 1 is being used rather than HTTP 2 to establish communication.

Spring Boot Zeebe client configuration:

zeebe:
  client:
    cloud:
      clusterId: xxx # Cluster Id
      clientId: xxx
      clientSecret: xxx
      region: xxx
    broker:
      gateway-address: 127.0.0.1:26500
    security:
        plaintext: true

Similar threads do not offer any solution

Thread 1

Note: Its a local setup and am not using Nginx. Using default credentials as per documentation

nathan.loding · November 22, 2023, 5:22pm

Hi @Jason7 - my first thought is that spring-zeebe might be confused about whether you are connecting to a local or SaaS instance. The zeebe.client.cloud.* properties are inferred to mean a SaaS account. You may need to force right right connection mode using zeebe.client.connection-mode=ADDRESS (see documentation here).

If that doesn’t work, I would try using a different tool such as zbctl to try to connect to Zeebe. That way we can determine if there’s something strange with spring-zeebe, or if there’s a different issue.

Jason7 · November 23, 2023, 5:05am

Hey @nathan.loding,

Thanks for the response.

Tried the changes but still getting SSL exception, because of invalid certificate path in local setup

org.springframework.context.ApplicationContextException: Failed to start bean 'zeebeLifecycleEventProducer'
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-6.0.8.jar:6.0.8]
	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
	at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:934) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:587) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:732) [spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434) [spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:310) [spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1304) [spring-boot-3.0.5.jar:3.0.5]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1293) [spring-boot-3.0.5.jar:3.0.5]
	at com.wu.compliance.pharos.workflowengine.ZeebeJobWorker.main(ZeebeJobWorker.java:12) [classes/:?]
Caused by: io.camunda.zeebe.client.api.command.ClientStatusException
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.lambda$start$7(ZeebeDeploymentAnnotationProcessor.java:119) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.start(ZeebeDeploymentAnnotationProcessor.java:100) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.lambda$startAll$0(ZeebeAnnotationProcessorRegistry.java:38) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.startAll(ZeebeAnnotationProcessorRegistry.java:38) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeClientEventListener.handleStart(ZeebeClientEventListener.java:17) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:343) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:228) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:413) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) ~[spring-context-6.0.8.jar:6.0.8]
	at io.camunda.zeebe.spring.client.event.ZeebeLifecycleEventProducer.start(ZeebeLifecycleEventProducer.java:23) ~[spring-boot-starter-camunda-8.3.1.jar:8.3.1]
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-6.0.8.jar:6.0.8]
	... 13 more
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: CANCELLED
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.lambda$start$7(ZeebeDeploymentAnnotationProcessor.java:119) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.start(ZeebeDeploymentAnnotationProcessor.java:100) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.lambda$startAll$0(ZeebeAnnotationProcessorRegistry.java:38) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at java.util.ArrayList.forEach(ArrayList.java:1511) ~[?:?]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.startAll(ZeebeAnnotationProcessorRegistry.java:38) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at io.camunda.zeebe.spring.client.annotation.processor.ZeebeClientEventListener.handleStart(ZeebeClientEventListener.java:17) ~[spring-client-zeebe-8.3.1.jar:8.3.1]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:343) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:228) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:165) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:413) ~[spring-context-6.0.8.jar:6.0.8]
	at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:370) ~[spring-context-6.0.8.jar:6.0.8]
	at io.camunda.zeebe.spring.client.event.ZeebeLifecycleEventProducer.start(ZeebeLifecycleEventProducer.java:23) ~[spring-boot-starter-camunda-8.3.1.jar:8.3.1]
	at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-6.0.8.jar:6.0.8]
	... 13 more
Caused by: io.grpc.StatusRuntimeException: CANCELLED
	at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.58.0.jar:1.58.0]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.58.0.jar:1.58.0]
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.58.0.jar:1.58.0]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.58.0.jar:1.58.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:370) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:313) ~[?:?]
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:308) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:200) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1430) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1401) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) ~[?:?]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.fetchCredentials(OAuthCredentialsProvider.java:151) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.computeIfMissingOrInvalid(OAuthCredentialsCache.java:100) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.applyCredentials(OAuthCredentialsProvider.java:84) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.ZeebeCallCredentials.lambda$applyRequestMetadata$0(ZeebeCallCredentials.java:48) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:200) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1430) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1401) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) ~[?:?]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.fetchCredentials(OAuthCredentialsProvider.java:151) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.computeIfMissingOrInvalid(OAuthCredentialsCache.java:100) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.applyCredentials(OAuthCredentialsProvider.java:84) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.ZeebeCallCredentials.lambda$applyRequestMetadata$0(ZeebeCallCredentials.java:48) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1341) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232) ~[?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175) ~[?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:200) ~[?:?]
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1500) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1415) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:450) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:421) ~[?:?]
	at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580) ~[?:?]
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1430) ~[?:?]
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1401) ~[?:?]
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:220) ~[?:?]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.fetchCredentials(OAuthCredentialsProvider.java:151) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.computeIfMissingOrInvalid(OAuthCredentialsCache.java:100) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.applyCredentials(OAuthCredentialsProvider.java:84) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at io.camunda.zeebe.client.impl.ZeebeCallCredentials.lambda$applyRequestMetadata$0(ZeebeCallCredentials.java:48) ~[zeebe-client-java-8.3.0.jar:8.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]

nathan.loding · November 27, 2023, 9:41pm

@Jason7 - this at least forward progress and a new error! Did you sign your self-managed installation with a certificate? Was it a commercial certificate or a self-signed one?

Jason7 · December 5, 2023, 9:48am

Hey @nathan.loding ,

No, did not use any certificate while doing self-managed installation.

Ryan_Waszak · March 10, 2024, 5:12am

@nathan.loding @Jason7

Hey guys, I’m experiencing a similar issue. Curious if any progress was made on your journey here?

We have a standalone zeebe cluster deployed with an embedded gateway via docker and behind a k8 ingress.

When I build and run the zeebe docker image locally, I can connect with ZeebeClient just fine on localhost:26500 and successfully execute all commands against the zeebe cluster.

After deploying to k8 however, I get an error with ZeebeClient as well as zbctl.

The part that sticks out though is I can use grpcurl to hit the remote server address and get back a valid response.

For example the following command:

grpcurl -insecure -import-path . -proto ./gateway.proto my-k8-ingress:443 gateway_protocol.Gateway/Topology

results in the following response:

{
  "brokers": [
    {
      "host": "xxx.xxx.xx.xxx",
      "port": 26501,
      "partitions": [
        {
          "partitionId": 1
        }
      ],
      "version": "8.4.4"
    }
  ],
  "clusterSize": 1,
  "partitionsCount": 1,
  "replicationFactor": 1,
  "gatewayVersion": "8.4.4"
}

However ZeebeClient throws the error:

First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f

which corresponds to this post A misleading error when using gRPC with Go and nginx - Kenneth Jenkins

and the following zbctl command

zbctl --insecure --address my-k8-ingress:443 status

results in the below error

Error: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: http2: frame too large"

which seems similar to the first.

Again, the fact that the grpcurl command can connect to the zeebe server and return the topology makes me think its something with the way the zeebe tools (zbctl and ZeebeClient) are communicating and not the deployment in k8 itself.

perrin_nj · April 22, 2025, 3:02pm

@Jason7 If this is for testing purpose, I would advice setting security plaintext to false(see code below). It worked for me.

    security:
        plaintext: false

aravindhrs · April 24, 2025, 11:23am

@Jason7, That error is a classic indicator that your Zeebe client tried to speak gRPC to something that doesn’t understand gRPC, like an HTTP endpoint.

“First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f”

485454502f is the hex for HTTP/ — meaning the response started with HTTP/…, like an HTTP 404/500 page.
But gRPC (used by Zeebe clients) expects the very first frame to be a SETTINGS frame in binary.

Root Cause

Your Zeebe client is connecting to something that’s:

Not a Zeebe Gateway, or
Not using gRPC, or
Behind a reverse proxy (like NGINX or ALB) that doesn’t support gRPC properly

Checklist to Fix It

1. Check the Gateway Address

Make sure your client is pointing to the Zeebe Gateway, not a REST endpoint, Operate, Tasklist, etc.

Example config for a self-managed cluster:

zeebe:
  client:
    broker:
      gateway-address: zeebe-gateway:26500

26500 is the gRPC port — make sure it’s not an HTTP port like 8080
Avoid URLs like http://localhost:8080 — that’s not valid for Zeebe gRPC

2. Avoid Connecting to Web Proxies

Are you going through an Application Load Balancer (ALB), NGINX, or API Gateway?

Default ALB or NGINX does not support gRPC properly without special config
You’ll get this exact error if an ALB forwards requests as HTTP instead of raw gRPC

Fix:

Use a Network Load Balancer (NLB) in AWS
Or configure NGINX with gRPC proxying (grpc_pass not proxy_pass)

3. Try a Manual gRPC Test

From your client machine:

grpcurl -plaintext your-zeebe-host:26500 list

If this fails, you’re not hitting a real gRPC Gateway
If it works, your client config may just have a typo or is hitting the wrong host internally

4. If Using Spring Boot

Double check application.yml or application.properties:

zeebe:
  client:
    broker:
      gateway-address: your-zeebe-gateway-host:26500

And confirm:

You’re using the correct DNS or IP
There’s no http:// in the value

Fix Summary

This is a connection problem, not a Zeebe bug
Make sure you’re connecting to the Zeebe Gateway on port 26500
Avoid HTTP reverse proxies
Use a gRPC-capable load balancer if needed
Use grpcurl to debug the connection