All components of Camunda8 (helm v11.1.1) self managed gke instance not running

Rushikesh_Khedkar · February 12, 2025, 7:03am

I have deployed camunda helm v 11.1.1 self managed instance with separate ingress on gke cluster - My operate, identity and tasklist components are not running -
We are getting error like below -

      io.camunda.operate.archiver.Archiver - INIT: Start archiving data...
2025-02-11 18:27:23.826 [] [main] [] WARN 
      io.camunda.operate.zeebe.PartitionHolder - Error occurred when requesting partition ids from Zeebe client: Network closed for unknown reason
io.camunda.zeebe.client.api.command.ClientStatusException: Network closed for unknown reason
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.6.7.jar:8.6.7]
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.6.7.jar:8.6.7]
	at io.camunda.webapps.zeebe.StandalonePartitionSupplier.getPartitionsCount(StandalonePartitionSupplier.java:23) ~[camunda-zeebe-8.6.7.jar:8.6.7]
	at io.camunda.operate.zeebe.PartitionHolder.getPartitionIdsFromZeebe(PartitionHolder.java:109) ~[operate-common-8.6.7.jar:8.6.7]
	at io.camunda.operate.zeebe.PartitionHolder.getPartitionIdsWithWaitingTimeAndRetries(PartitionHolder.java:67) ~[operate-common-8.6.7.jar:8.6.7]
	at io.camunda.operate.zeebe.PartitionHolder.getPartitionIds(PartitionHolder.java:50) ~[operate-common-8.6.7.jar:8.6.7]
	at io.camunda.operate.archiver.Archiver.startArchiving(Archiver.java:49) ~[operate-archiver-8.6.7.jar:8.6.7]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) ~[spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1798) ~[spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:600) ~[spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:337) ~[spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) [spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:335) [spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) [spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:975) [spring-beans-6.1.14.jar:6.1.14]
	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:971) [spring-context-6.1.14.jar:6.1.14]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:625) [spring-context-6.1.14.jar:6.1.14]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) [spring-boot-3.3.7.jar:3.3.7]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) [spring-boot-3.3.7.jar:3.3.7]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) [spring-boot-3.3.7.jar:3.3.7]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:335) [spring-boot-3.3.7.jar:3.3.7]
	at io.camunda.application.StandaloneOperate.main(StandaloneOperate.java:53) [camunda-zeebe-8.6.7.jar:8.6.7]
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.6.7.jar:8.6.7]
	... 26 more
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.68.2.jar:1.68.2]
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.68.2.jar:1.68.2]
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.68.2.jar:1.68.2]
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.68.2.jar:1.68.2]
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.68.2.jar:1.68.2]
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564) ~[grpc-core-1.68.2.jar:1.68.2]
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72) ~[grpc-core-1.68.2.jar:1.68.2]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729) ~[grpc-core-1.68.2.jar:1.68.2]
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710) ~[grpc-core-1.68.2.jar:1.68.2]
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.68.2.jar:1.68.2]
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.68.2.jar:1.68.2]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]
2025-02-11 18:27:23.844 [] [main] [] INFO 
      io.camunda.operate.zeebe.PartitionHolder - Partition ids can't be fetched from Zeebe. Try next round (1).
2025-02-11 18:27:24.845 [] [main] [] WARN 
      io.camunda.operate.zeebe.PartitionHolder - Error occurred when requesting partition ids from Zeebe client: Network closed for unknown reason
io.camunda.zeebe.client.api.command.ClientStatusException: Network closed for unknown reason

Not able to find the root cause of the issue.

Alex_Voloshyn · February 14, 2025, 9:04am

Hi @Rushikesh_Khedkar
Can you share the statuses of all pods in a namespace where you install the Camunda 8 platform components? It would be also beneficial to see your values.yaml file.

Regards,
Alex

Raghavendra_Chari · February 17, 2025, 1:33pm

Hi Rushikesh,

I am seeing similar error (not exactly the same error message though) in the self managed env.

I will be watching updates to your ticket to see if I can get some tips as well.

io.camunda.tasklist.zeebe.PartitionHolder - Partition ids can’t be fetched from Zeebe. Return empty partition ids list. io.camunda.tasklist.zeebe.PartitionHolder - Tasklist needs an established connection to Zeebe in order to retrieve Zeebe Partitions. Shutting down… io.atomix.cluster.protocol.swim - camunda-tasklist-cf6d9967c-6fvzg - Member deactivated: Member{id=camunda-tasklist-cf6d9967c-6fvzg, address=10.244.0.6:26502, properties={event-service-topics-subscribed=KIIDAGpvYnNBdmFpbGFibOU=}}

Raghavendra_Chari · February 18, 2025, 5:25pm

Hi @Rushikesh_Khedkar - I was able to overcome the issue by adding nodes to the cluster.

Here is my working env - self managed. Just thought of sharing with you.
Basically you need to avoid situation where Zeebe node goes into Pending status. Hope this helps.

clusterSize: 3
partitionCount: 5
replicationFactor: 3

Rushikesh_Khedkar · February 19, 2025, 7:18am

Hi Raghvendra, Thank you for reply. I have 3 node cluster already. I made changes in the env varibale - ZEEBE_GATEWAY_SECURITY_AUTHENTICATION_MODE and enable the ingress for component as i was using seperate ingress.

My all pods are working but I am not able to deploy BPMN via modeler. I am getting below error in zeebe-gateway-pod -


Feb 19, 2025 7:11:00 AM io.grpc.netty.NettyServerTransport notifyTerminated
INFO: Transport failed
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: POST /gateway_protocol.Gateway/Topology 
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:315)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:245)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)

Feb 19, 2025 7:11:01 AM io.grpc.netty.NettyServerTransport notifyTerminated
INFO: Transport failed
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: POST /gateway_protocol.Gateway/Topology 
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:315)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:245)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)

Raghavendra_Chari · February 19, 2025, 2:14pm

Hi @Rushikesh_Khedkar - Can you share the details you are passing while invoking deploy command via Modeler?

Alternately, you may consider deploying bpmn via zbctl CLI - just to see if this method works.

Also, if I may, just to make sure your env is running properly, can you run the “get topology” REST API? Swagger link below.

http://localhost:8082/tasklist/swagger-ui/index.html#/topology-controller/get

Hope this helps !

Rushikesh_Khedkar · February 19, 2025, 2:37pm

This is the configuration i am using while deploying BPMN.

GotnOGuts · February 19, 2025, 4:25pm

This error often means that you have a proxy or a HTTP/1 ingress between the modeller and the gateway. Since the modeller is expecting a GRPC connection, which requires a HTTP/2 connection, a proxy server has to be able to speak HTTP/2 for it to work.