Camunda 8.8 - grpc service is not reachable

Cris_Ron · February 6, 2026, 3:33pm

Hi,

I cannot reach the grpc service endpoint in my Camunda 8.8 instance, it is throwing 500 internal erros, these are the logs generated in the Kubernetes pod:

[2026-02-06 13:43:49.301] [grpc-default-worker-ELG-5-2] DEBUG
	io.grpc.netty.NettyServerHandler - [id: 0xb7c9d7e9, L:/10.244.1.157:26500 - R:/10.244.3.158:34078] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
[2026-02-06 13:43:49.302] [grpc-default-worker-ELG-5-2] DEBUG
	io.grpc.netty.NettyServerHandler - Connection Error
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: POST /gateway_protocol.Gateway/CreateProcessInstance 
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:315)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:245)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:545)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:484)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:805)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)
[2026-02-06 13:43:49.302] [grpc-default-worker-ELG-5-2] DEBUG
	io.grpc.netty.NettyServerHandler - [id: 0xb7c9d7e9, L:/10.244.1.157:26500 - R:/10.244.3.158:34078] OUTBOUND GO_AWAY: lastStreamId=2147483647 errorCode=1 length=82 bytes=556e657

My grpc ingress config is this, I also have a global ingress definition for REST endpoint, which is working fine:

orchestration:
  enabled: true
  contextPath: "/"
  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/"

  camunda:
    persistent:
      sessions:
        enabled: true
    rest:
      query:
        enabled: true
 
  ingress:
    grpc:
      enabled: true
      className: ${ingress_controller}
      annotations:
        cert-manager.io/cluster-issuer: ${clusterissuer_name}
        traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
        traefik.ingress.kubernetes.io/router.tls: "true"
        acme.cert-manager.io/http01-edit-in-place: "false"
        acme.cert-manager.io/http01-ingress-class-name: ${ingress_controller}
        cert-manager.io/common-name: "grpc.${namespace_name}.${service_uri}${stage_uri}${domain}"
        cert-manager.io/duration: "720h"
        cert-manager.io/renew-before: "24h"
        cert-manager.io/subject-organizations: ${subject_organizations}
        cert-manager.io/subject-organizationalunits: ${subject_organizationalunits}
        cert-manager.io/subject-localities: ${subject_localities}
        cert-manager.io/subject-provinces: ${subject_provinces}
        cert-manager.io/subject-countries: ${subject_countries}
        cert-manager.io/private-key-size: "4096"
      host: "grpc.${namespace_name}.${service_uri}${stage_uri}${domain}"
      tls:
        enabled: true
        secretName: "${namespace_name}-${stage}-grpc-tls"

Kind Regards,

Julian

zee-bot · February 7, 2026, 3:34pm

Hi @Cris_Ron,

This is a Problem classification, and I can help you resolve this gRPC connectivity issue.

The error Unexpected HTTP/1.x request: POST /gateway_protocol.Gateway/CreateProcessInstance indicates that your Zeebe gateway is receiving HTTP/1.1 traffic, but gRPC requires HTTP/2 end-to-end. This is typically caused by a misconfigured ingress/proxy in front of Zeebe.

Root Cause

Your Traefik ingress is likely terminating TLS and then forwarding HTTP/1.1 to Zeebe instead of HTTP/2, or the backend protocol is not correctly configured as gRPC.

Troubleshooting Steps

1. Verify Zeebe is healthy (bypass Traefik)

First, let’s confirm Zeebe itself is working by bypassing the ingress:

kubectl port-forward svc/camunda-zeebe-gateway 26500:26500

Then test with grpcurl:

grpcurl -plaintext localhost:26500 list

If this works, the issue is definitely in your Traefik configuration.

2. Test HTTP/2 negotiation through Traefik

Check if your Traefik ingress properly negotiates HTTP/2:

curl -I --http2 https://grpc.${namespace_name}.${service_uri}${stage_uri}${domain}

And test gRPC through the ingress:

grpcurl -insecure grpc.${namespace_name}.${service_uri}${stage_uri}${domain}:443 list

3. Fix your Traefik configuration

Your current configuration is missing crucial gRPC-specific annotations. Here’s what you need to add to your ingress configuration:

ingress:
  grpc:
    enabled: true
    className: ${ingress_controller}
    annotations:
      cert-manager.io/cluster-issuer: ${clusterissuer_name}
      traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
      traefik.ingress.kubernetes.io/router.tls: "true"
      # Add these gRPC-specific annotations:
      traefik.ingress.kubernetes.io/service.serversscheme: h2c
      traefik.ingress.kubernetes.io/router.middlewares: default-grpc-web@kubernetescrd
      # Your existing cert-manager annotations...
    host: "grpc.${namespace_name}.${service_uri}${stage_uri}${domain}"
    tls:
      enabled: true
      secretName: "${namespace_name}-${stage}-grpc-tls"

4. Ensure proper Traefik entrypoint configuration

Make sure your Traefik websecure entrypoint supports HTTP/2. In your Traefik configuration, ensure:

entryPoints:
  websecure:
    address: ":443"
    http:
      tls:
        options: default
    # Ensure HTTP/2 is enabled
    transport:
      respondingTimeouts:
        readTimeout: 0s

Key Requirements for gRPC with Traefik:

HTTP/2 must be enabled on the Traefik entrypoint
Backend protocol must be configured as gRPC/h2c (not plain HTTP/1)
TLS with ALPN must properly negotiate HTTP/2
No HTTP/1 downgrade should occur anywhere in the chain

References:

Please try the port-forward test first to confirm Zeebe is healthy, then work on the Traefik configuration. Let me know the results of your testing!

Best regards,
Zee

Cris_Ron · February 11, 2026, 9:07am

@zee-bot - I have edited my config, see here:

ingress:
    grpc:
      enabled: true
      className: ${ingress_controller}
      annotations:
        cert-manager.io/cluster-issuer: ${clusterissuer_name}
        traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
        traefik.ingress.kubernetes.io/router.tls: "true"
        traefik.ingress.kubernetes.io/service.serversscheme: h2c
        traefik.ingress.kubernetes.io/router.middlewares: default-grpc-web@kubernetescrd
        # traefik.ingress.kubernetes.io/router.middlewares: "${namespace_name}-${stage}-strip-connectors@kubernetescrd"
        acme.cert-manager.io/http01-edit-in-place: "false"
        acme.cert-manager.io/http01-ingress-class-name: ${ingress_controller}
        cert-manager.io/common-name: "grpc.${namespace_name}.${service_uri}${stage_uri}${domain}"
        cert-manager.io/duration: "720h"
        cert-manager.io/renew-before: "24h"
        cert-manager.io/subject-organizations: ${subject_organizations}
        cert-manager.io/subject-organizationalunits: ${subject_organizationalunits}
        cert-manager.io/subject-localities: ${subject_localities}
        cert-manager.io/subject-provinces: ${subject_provinces}
        cert-manager.io/subject-countries: ${subject_countries}
        cert-manager.io/private-key-size: "4096"
      host: "grpc.${namespace_name}.${service_uri}${stage_uri}${domain}"
      tls:
        enabled: true
        secretName: "${namespace_name}-${stage}-grpc-tls"

but I still get the same error:

[2026-02-11 08:47:51.447] [grpc-default-worker-ELG-5-1] DEBUG
	io.grpc.netty.NettyServerHandler - [id: 0x463285ee, L:/10.244.2.158:26500 - R:/10.244.3.158:40984] OUTBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}
[2026-02-11 08:47:51.448] [grpc-default-worker-ELG-5-1] DEBUG
	io.grpc.netty.NettyServerHandler - [id: 0x463285ee, L:/10.244.2.158:26500 - R:/10.244.3.158:40984] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
[2026-02-11 08:47:51.451] [grpc-default-worker-ELG-5-1] DEBUG
	io.grpc.netty.NettyServerHandler - Connection Error
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: POST /gateway_protocol.Gateway/CreateProcessInstance 
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:315)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:245)
	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:451)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:545)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:484)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:805)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Unknown Source)
[2026-02-11 08:47:51.453] [grpc-default-worker-ELG-5-1] DEBUG
	io.grpc.netty.NettyServerHandler - [id: 0x463285ee, L:/10.244.2.158:26500 - R:/10.244.3.158:40984] OUTBOUND GO_AWAY: lastStreamId=2147483647 errorCode=1 length=82 bytes=556e657870656374656420485454502f312e7820726571756573743a20504f5354202f676174657761795f70726f746f636f6c2e476174657761792f43726561...
[2026-02-11 08:47:51.454] [grpc-default-worker-ELG-5-1] DEBUG
	io.netty.handler.codec.http2.Http2ConnectionHandler - [id: 0x463285ee, L:/10.244.2.158:26500 - R:/10.244.3.158:40984] Sent GOAWAY: lastStreamId '2147483647', errorCode '1', debugData 'Unexpected HTTP/1.x request: POST /gateway_protocol.Gateway/CreateProcessInstance '. Forcing shutdown of the connection.
[2026-02-11 08:47:51.454] [grpc-default-worker-ELG-5-1] INFO 
	io.grpc.netty.NettyServerTransport.connections - Transport failed
io.netty.handler.codec.http2.Http2Exception: Unexpected HTTP/1.x request: POST /gateway_protocol.Gateway/CreateProcessInstance 
	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:107)
	at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:315)

curl -vk --http2 https://grpc.camunda##/ 2>&1
* Host grpc.camunda##:443 was resolved.
* IPv6: (none)
* IPv4: 1##
*   Trying ##:443...
* Connected to grpc.camunda## port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_CHACHA20_POLY1305_SHA256 / X25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: [NONE]
*  start date: Feb  6 09:06:56 2026 GMT
*  expire date: May  7 09:06:55 2026 GMT
*  issuer: C=US; O=Let's Encrypt; CN=YR2
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
*   Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://grpc.camunda##/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: grpc.camunda##.de]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: grpc.camunda##
> User-Agent: curl/8.5.0
> Accept: */*
>
< HTTP/2 500
< set-cookie: camunda01-demo-sticky-cookie=1fac0068dd24da6c; Path=/; HttpOnly; Secure; SameSite=Lax
< content-length: 21

Cris_Ron · February 11, 2026, 10:37pm

Ok, I solved this issue by using an ingressroute in traefik instead of the inbuilt ingress resource from the camunda values.yaml. With the ingressroute I can define the protocol which is used for backend communication which is not possible with a normal Kubernetes ingress object.

system · February 18, 2026, 10:38pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.