How to connect gRPC gateway with TLS autentication?

We have Camunda 8 Self managed-Ingress configuration of Camunda platform gRPC is as below:

annotations:
  ingress.kubernetes.io/rewrite-target: /
  meta.helm.sh/release-name: camunda
  meta.helm.sh/release-namespace: pp-dev
  nginx.ingress.kubernetes.io/backend-protocol: GRPC
  nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
  nginx.ingress.kubernetes.io/ssl-redirect: 'true'
managedFields:
  - manager: nginx-ingress-controller
    operation: Update
    apiVersion: networking.k8s.io/v1
    time: '2025-09-24T06:24:08Z'
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        f:loadBalancer:
          f:ingress: {}
    subresource: status
  - manager: helm
    operation: Update
    apiVersion: networking.k8s.io/v1
    time: '2025-09-26T09:27:36Z'
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:ingress.kubernetes.io/rewrite-target: {}
          f:meta.helm.sh/release-name: {}
          f:meta.helm.sh/release-namespace: {}
          f:nginx.ingress.kubernetes.io/backend-protocol: {}
          f:nginx.ingress.kubernetes.io/proxy-buffer-size: {}
          f:nginx.ingress.kubernetes.io/ssl-redirect: {}
        f:labels:
          .: {}
          f:app: {}
          f:app.kubernetes.io/component: {}
          f:app.kubernetes.io/instance: {}
          f:app.kubernetes.io/managed-by: {}
          f:app.kubernetes.io/name: {}
          f:app.kubernetes.io/part-of: {}
          f:app.kubernetes.io/version: {}
          f:helm.sh/chart: {}
      f:spec:
        f:ingressClassName: {}
        f:rules: {}
        f:tls: {}
spec:
ingressClassName: nginx
tls:
  - hosts:
      - projectdomain.dev.echonet
    secretName: camunda-tls
rules:
  - host: projectdomain.dev.echonet
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: camunda-zeebe-gateway
              port:
                number: 26500
status:

In spring boot project our application.properties has the below https gRPC attribute in place :

camunda.client.zeebe.grpc-address=https://projectdomain.dev.echonet/

No more properties is configured related to camunda.client(Please sugguest if any thing we are missing for tls authentication)


Also while starting springboot application in local we are passing the below CLI arugmnent for our keystore and trustStore:

-Djavax.net.ssl.keyStore=C:/Software/cert/https-keystore.jks
-Djavax.net.ssl.keyStorePassword=password
-Djavax.net.ssl.trustStore=C:/Software/cert/truststore.jks
-Djavax.net.ssl.trustStorePassword=password


Based on your configuration, I can see you’re trying to connect to Camunda 8 Self-Managed via gRPC with TLS. Looking at your setup, there are a few things to consider:

Current Configuration Analysis

Your current setup shows:

  • Ingress: Properly configured with TLS termination and GRPC backend protocol
  • Spring Boot: Basic gRPC address configuration
  • JVM SSL: Keystore and truststore configured via system properties

Missing Camunda Client Properties

For the Spring Zeebe SDK, you should add the following property to your application.properties:

camunda.client.zeebe.ca-certificate-path=/path/to/certificate.pem

This property allows the Zeebe client to use a specific CA certificate to validate the gateway’s certificate chain, rather than relying solely on the JVM’s default truststore.

Important Considerations

  1. Certificate Format: The ca-certificate-path property expects a PEM format certificate, not a JKS truststore. You may need to extract the CA certificate from your truststore in PEM format.

  2. TLS Termination: Since your ingress is handling TLS termination (as indicated by ssl-redirect: 'true'), the communication between the ingress and the Zeebe gateway service is likely plain HTTP/gRPC. However, your client still needs to trust the ingress certificate.

  3. gRPC Address: Your current address https://projectdomain.dev.echonet/ looks correct for HTTPS/TLS connection.

Recommended Configuration

Update your application.properties:

camunda.client.zeebe.grpc-address=https://projectdomain.dev.echonet/
camunda.client.zeebe.ca-certificate-path=/path/to/your-ca-certificate.pem

Alternative Approach

If you prefer to continue using JVM system properties for SSL configuration (which should also work), ensure that:

  • Your truststore contains the CA certificate that signed your ingress TLS certificate
  • The keystore is only needed if you’re doing mutual TLS authentication (client certificates)

Troubleshooting

If you’re still having connection issues, check:

  1. That the certificate in your truststore matches the one presented by your ingress
  2. The certificate chain is complete
  3. The certificate hasn’t expired

References:

Let me know if you need help with certificate extraction or if you encounter any specific errors!

What i understand certificate.pem file is required when we configure Ingress with Mutual TLS. But in our case we are not using MTLS. In this case what extension file we need to pass in camunda.client.zeebe.ca-certificate-path= ? property in our application.properties file.

And in alternative approach when we use JVM system properties for SSL configuration and start the application by passing truststore.jks. Do we still need to pass camunda.client.zeebe.ca-certificate-path property along with gRPC url?

Please let me know if any additional information is needed. Thanks !!

Great questions! Let me clarify both scenarios for you:

1. Certificate File for Non-Mutual TLS

You’re correct that mutual TLS (mTLS) is not the same as server certificate validation. Even without mTLS, you still need to validate the server’s certificate. Here’s what you need:

For camunda.client.zeebe.ca-certificate-path:

  • File format: PEM format (.pem, .crt, or .cer extension)
  • Content: The CA certificate that signed your ingress/server certificate
  • Purpose: Server certificate validation (not client authentication)

Since you’re using an ingress with TLS, you need the CA certificate that signed your ingress certificate. This could be:

  • Your internal CA certificate (if using self-signed or internal CA)
  • The public CA certificate (if using Let’s Encrypt, DigiCert, etc.)

To extract the CA certificate from your existing truststore:

keytool -exportcert -alias your-ca-alias -file ca-certificate.pem -keystore truststore.jks -rfc

2. JVM System Properties vs Camunda Properties

You do NOT need both approaches. Here’s how they work:

Option A: JVM System Properties (Your Current Approach)

-Djavax.net.ssl.trustStore=C:/Software/cert/truststore.jks
-Djavax.net.ssl.trustStorePassword=password

Application.properties:

camunda.client.zeebe.grpc-address=https://projectdomain.dev.echonet/
# NO need for ca-certificate-path when using JVM truststore

Option B: Camunda-Specific Property

Application.properties:

camunda.client.zeebe.grpc-address=https://projectdomain.dev.echonet/
camunda.client.zeebe.ca-certificate-path=/path/to/ca-certificate.pem

No JVM SSL properties needed

Recommendation for Your Setup

Since you already have a working truststore setup with JVM properties, stick with Option A. Your current configuration should work:

# application.properties
camunda.client.zeebe.grpc-address=https://projectdomain.dev.echonet/
# JVM arguments
-Djavax.net.ssl.trustStore=C:/Software/cert/truststore.jks
-Djavax.net.ssl.trustStorePassword=password

Note: The keystore properties are only needed if your server requires client certificates (mutual TLS), which doesn’t seem to be your case.

Troubleshooting

If you’re still having connection issues, verify:

  1. Your truststore contains the correct CA certificate
  2. The certificate chain is complete
  3. The hostname in your certificate matches your domain

References:

Does this clarify the setup for you?

The error i am getting while starting up the application as below:

io.grpc.StatusRuntimeException: UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
	at io.grpc.Status.asRuntimeException(Status.java:532)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489)
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453)
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2025-09-26T16:36:08.510+05:30  WARN 23072 --- [pool-2-thread-1] io.camunda.client.job.poller             : Failed to activate jobs for worker Email template connector and job type io.camunda:template:1

io.grpc.StatusRuntimeException: UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
	at io.grpc.Status.asRuntimeException(Status.java:532)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:729)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:710)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

The error you’re encountering is an ALPN (Application-Layer Protocol Negotiation) failure, which is a common issue when connecting gRPC clients to Zeebe through an ingress controller. This error indicates that the client and server cannot agree on a compatible protocol (HTTP/2 is required for gRPC).

Root Cause Analysis

The issue is likely related to your Ingress NGINX configuration. gRPC requires:

  1. HTTP/2 protocol support
  2. ALPN negotiation over TLS
  3. Proper TLS termination

Solution: Update Your Ingress Configuration

Looking at your current ingress configuration, I notice it might be missing some critical annotations for gRPC support. Here’s what you should update:

Updated Ingress Configuration

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: camunda-zeebe-gateway-grpc
  annotations:
    # Essential for gRPC support
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/grpc-backend: "true"
    
    # HTTP/2 support (critical for ALPN)
    nginx.ingress.kubernetes.io/http2-push-off: "true"
    
    # SSL configuration
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    
    # Buffer settings (you already have this)
    nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
    
    # Optional: Increase timeouts for long-running gRPC connections
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - projectdomain.dev.echonet
      secretName: camunda-tls
  rules:
    - host: projectdomain.dev.echonet
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: camunda-zeebe-gateway
                port:
                  number: 26500

Key Changes:

  1. Added nginx.ingress.kubernetes.io/grpc-backend: "true" - Explicitly enables gRPC support
  2. Added HTTP/2 configuration - Ensures proper protocol negotiation
  3. Enhanced SSL settings - Forces SSL redirect for proper TLS handling

Verify Ingress Controller Version

Ensure your NGINX Ingress Controller supports gRPC and HTTP/2:

kubectl get pods -n ingress-nginx -o yaml | grep image:

You need ingress-nginx version 0.40.0 or later for proper gRPC support.

Alternative: Test Direct Connection

To isolate the issue, try connecting directly to the Zeebe gateway (bypassing ingress) using port-forwarding:

kubectl port-forward svc/camunda-zeebe-gateway 26500:26500

Then update your application.properties temporarily:

camunda.client.zeebe.grpc-address=http://localhost:26500

If this works, the issue is definitely with the ingress configuration.

Additional Troubleshooting

  1. Check ingress controller logs:

    kubectl logs -n ingress-nginx deployment/ingress-nginx-controller
    
  2. Verify TLS certificate:

    openssl s_client -connect projectdomain.dev.echonet:443 -alpn h2
    

    Look for “ALPN protocol: h2” in the output.

  3. Test with grpcurl:

    grpcurl -insecure projectdomain.dev.echonet:443 list
    

References:

Try updating your ingress configuration first, as this is the most likely cause of the ALPN negotiation failure. Let me know if you need help with any of these steps!

Even after updating above Ingress Configuration we are getting the error as below on application start up :

Caused by: io.camunda.zeebe.client.api.command.ClientStatusException: Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:122) ~[zeebe-client-java-8.7.10.jar:8.7.10]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.7.10.jar:8.7.10]
at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.lambda$start$6(ZeebeDeploymentAnnotationProcessor.java:86) ~[spring-boot-starter-camunda-sdk-8.7.10.jar:8.7.10]
at java.base/java.util.ArrayList.forEach(ArrayList.java:1596) ~[na:na]
at io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor.start(ZeebeDeploymentAnnotationProcessor.java:63) ~[spring-boot-starter-camunda-sdk-8.7.10.jar:8.7.10]
at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.lambda$startAll$0(ZeebeAnnotationProcessorRegistry.java:63) ~[spring-boot-starter-camunda-sdk-8.7.10.jar:8.7.10]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at io.camunda.zeebe.spring.client.annotation.processor.ZeebeAnnotationProcessorRegistry.startAll(ZeebeAnnotationProcessorRegistry.java:63) ~[spring-boot-starter-camunda-sdk-8.7.10.jar:8.7.10]
at io.camunda.zeebe.spring.client.event.ZeebeClientEventListener.handleStart(ZeebeClientEventListener.java:32) ~[spring-boot-starter-camunda-sdk-8.7.10.jar:8.7.10]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
… 14 common frames omitted
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[na:na]
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[na:na]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.7.10.jar:8.7.10]
… 33 common frames omitted
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: Failed ALPN negotiation: Unable to find compatible protocol
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at io.grpc.Status.asRuntimeException(Status.java:532) ~[grpc-api-1.68.3.jar:1.68.3]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.68.3.jar:1.68.3]
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.68.3.jar:1.68.3]
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.68.3.jar:1.68.3]
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.68.3.jar:1.68.3]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:564) ~[grpc-core-1.68.3.jar:1.68.3]

The persistent ALPN negotiation error suggests we need to take a systematic troubleshooting approach. Let’s work through this step by step:

Step 1: Test with Plaintext Connection (Bypass TLS)

First, let’s isolate whether this is a TLS/ALPN issue or a general connectivity problem. Try connecting directly to the Zeebe gateway without TLS:

Port Forward to Zeebe Gateway

kubectl port-forward svc/camunda-zeebe-gateway 26500:26500

Update Application Properties (Temporarily)

camunda.client.zeebe.grpc-address=http://localhost:26500
camunda.client.zeebe.plaintext-connection=true

Remove the JVM SSL arguments temporarily and test if the connection works.

Step 2: Check Java Version and ALPN Support

What Java version are you using? Run:

java -version

Java 11+ is recommended for proper HTTP/2 and ALPN support. If you’re using Java 8, you might need additional ALPN libraries.

Step 3: Verify Ingress Controller Capabilities

Check your NGINX Ingress Controller version:

kubectl get pods -n ingress-nginx -o jsonpath='{.items[0].spec.containers[0].image}'

Test if your ingress supports HTTP/2:

curl -I --http2 https://projectdomain.dev.echonet/

Look for HTTP/2 in the response.

Step 4: Test with grpcurl

Install and test with grpcurl to verify gRPC connectivity:

# Test without TLS (port-forward)
grpcurl -plaintext localhost:26500 list

# Test with TLS through ingress
grpcurl -insecure projectdomain.dev.echonet:443 list

Step 5: Alternative Ingress Configuration

If the above doesn’t work, try this alternative ingress configuration that explicitly handles HTTP/2:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: camunda-zeebe-gateway-grpc
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/grpc-backend: "true"
    
    # Force HTTP/2
    nginx.ingress.kubernetes.io/server-snippet: |
      grpc_read_timeout 300;
      grpc_send_timeout 300;
      client_body_timeout 60;
      client_header_timeout 60;
      client_max_body_size 1m;
      
    # Disable buffering for gRPC
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - projectdomain.dev.echonet
      secretName: camunda-tls
  rules:
    - host: projectdomain.dev.echonet
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: camunda-zeebe-gateway
                port:
                  number: 26500

Step 6: Check Zeebe Gateway Configuration

Verify that your Zeebe Gateway is configured to handle TLS properly. Check the gateway configuration:

kubectl get configmap camunda-zeebe-gateway-config -o yaml

Look for TLS-related settings in the gateway configuration.

Step 7: Alternative Client Configuration

If you must use TLS, try adding these properties to force specific TLS behavior:

camunda.client.zeebe.grpc-address=https://projectdomain.dev.echonet/
camunda.client.zeebe.keep-alive=30000
camunda.client.zeebe.request-timeout=20000

Step 8: Check Network Connectivity

Test basic connectivity:

telnet projectdomain.dev.echonet 443

Diagnostic Questions

Please run the plaintext test (Step 1) first and let me know:

  1. Does the plaintext connection work?
  2. What Java version are you using?
  3. What’s the output of the grpcurl tests?
  4. What NGINX Ingress Controller version are you running?

Based on these results, we can determine if this is:

  • A TLS/ALPN configuration issue
  • An ingress controller limitation
  • A Java/client-side problem
  • A network connectivity issue

References:

Let’s start with the plaintext test to isolate the issue!

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.