How can I enable TLS ins Zeebe-Gateway via Helm?

Camunda 8 Topics
1

I’m nearly through setting up Camunda in Google Kubernetes Engine, but the last puzzle piece I just can’t get setup is to enable Zeebe access via GKE ingress. It works fine with port forwarding.

I created the ingress and I can see that requests actually reach the Zeebe-Gateway pods, but they fail with the error HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 16030100e8010000e403037227fd6c1eca59ff037b419a2a.

I followed the pitfalls notice here and enabled HTTP/2 for the backend gateway port so the GKE load balancer forwards HTTP/2 traffic to the backend:

zeebeGateway:
  service:
    annotations:
      cloud.google.com/app-protocols: '{"gateway":"HTTP2"}'  # Specifies HTTP2 for gRPC
      beta.cloud.google.com/backend-config: '{"default": "camunda-zeebe-backendconfig"}'

It was still not working and the error message above normally would suggest the GRPC service expected non-TLS traffic while actually receiving TLS traffic. Looking through GCP documentation, it is mentioned that the GCP load balancer always forwards HTTP/2 traffic to the backend via TLS, no way around it and no HTTPS offloading happening. It will gladly accept any certificate, also self signed ones, so that should be no issue. I was able to add certs to the pods easily. But I can’t figure out how I can enable TLS for the Zeebe-Gateway … I tried something like this as I found some mentions that this should work:

zeebeGateway:
  service:
    annotations:
      cloud.google.com/app-protocols: '{"gateway":"HTTP2"}'
      beta.cloud.google.com/backend-config: '{"default": "camunda-zeebe-backendconfig"}'
  extraVolumes:
    - name: zeebe-tls-cert
      secret:
        secretName: zeebe-tls-cert
  extraVolumeMounts:
    - name: zeebe-tls-cert
      mountPath: "/path/to/certs"
      readOnly: true
  security:
    tls:
      enabled: true
      privateKeyPath: /path/to/certs/tls.key
      certificateChainPath: /path/to/certs/tls.crt

But it doesn’t and the Camunda helm charts actually do not really support the claim this should work as far as I can tell.

How would I go about enabling TLS with the helm deployment?

Thanks!

2

Did you check this url, Camunda Self-Managed for Absolute Beginners, Part 2—Ingress and TLS SSL | Camunda ?

you can create separate tls secret with all the cert/key and mention in the zeebe Gateway section.

1 Like
3

I did go over this - as I’m using gke-ingress, I somehow thought the ingress section in all camunda services is not relevant as this applies to Camunda’s OOB ingress.

But now that you mention it, if I’d provide className: gce instead of className: nginx this might apply to GKE’s ingress too?

I’ll try it out.

4

Hmmm this creates a new nginx-based ingress in GKE in addition to my existing gke-ingress.

I’m unsure how I could combine that with my pre-existing GKE Ingress configuraiton.

5

We are experiencing similar issues with zeebe-gateway using gce-ingress. Here is a separate ingress.yaml for zeebe-gateway.

We are unable to connect using zbctl even with port-forwarding at the service on port 26500

ingress.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: camunda-zeebe-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "camunda-zeebe-dev-ingress"
    kubernetes.io/ingress.class: "gce"
    networking.gke.io/managed-certificates: "dev-camunda"
spec:
  rules:
    - host: zeebe-dev.camunda.abcdefg.dev
      http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: camunda-zeebe-gateway
              port:
                number: 26500

camunda-values.yaml with zeebe-gateway component

zeebe-gateway:
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      cloud.google.com/app-protocols: '{"gateway":"HTTP2"}'
      cloud.google.com/backend-config: '{"default": "camunda-hc-zeebe-gateway"}'    
    spec:
      # type: LoadBalancer
      ports:
      - name: gateway
        port: 26500
        targetPort: 26500
        protocol: TCP
        tls:
          enabled: true
          secretName: "tls-secret-zeebe"
      selector:
        app: camunda-platform
        app.kubernetes.io/component: camunda-zeebe-gateway
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: zeebe-gateway
        app.kubernetes.io/part-of: camunda-platform

So we cannot even deploy BPMNs from desktop modeler. Any ideas, what we may be overlooking?

Thanks,
Atul

6

Finally got zeebe-gateway working albeit with port-forwarding. Still struggling getting gce-ingresss to allow connection of zeebe gateway over SSL, even after importing the SSL certificates into Keychain trusted stores.

Working through it. Both zbctl and DesktopModeler failing to connect over SSL.

7

How are you attempting to deploy from Modeler?
Modeler will default to port 26500, but an ingress doesn’t listen for that port… you have to reconfigure zbctl or Desktop Modeler to connect to https or http (depending on how you have your ingress configured)

8

Thank you. How does one reconfigure the Desktop Modeler to use port 443 (ingress default SSL port). Using zbctl, that does not work with gce-ingress

zbctl status --address "zeebe-dev-camunda.abcdefg.dev" --host "zeebe-dev-camunda.abcedefg.dev" --port 443 --clientId zeebe --clientSecret ****** --scope "camunda-identity" --authzUrl "https://dev-camunda.abcdefg.dev/auth/realms/camunda-platform/protocol/openid-connect/token" --audience "zeebe-api" --certPath "dev-camunda.abcdefg.dev.pem"

I’d think the above would work. However, that does not work. With port-forwarding, and using http://localhost:26500, works with the exact same options in zbctl with the obvious changes.

I think the issue is what Jack described earlier - gce-ingress does not allow non-SSL traffic for zeebe

9

Ok, but what is the return from running the command that way?

Note: I don’t have access to GCE environment, and am just another user like you, so would need you to provide complete information to make suggestions.

Since Ingress is doing the TLS termination, you’d have to tell your Ingress to use TLS to talk to the Gateway rather than http.

10

Got it. For zbctl using nginx, I get this error:

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for ingress.local, not zeebe-nginx-dev-camunda.abcdefg.dev"

We are working to get a valid cert for the new domain we created for nginx.

For zbctl using gc-ingress`

Unavailable desc = connection error: desc = "transport: authentication handshake failed: EOF"

The DesktopModeler logs show this:

2024-11-18T19:19:39.228Z  ERROR app:zeebe-api connection check failed {
  parameters: {
    endpoint: {
      type: 'selfHosted',
      authType: 'oauth',
      url: 'https://zeebe-nginx-dev-camunda.abcdefg.dev:443',
      oauthURL: 'https://dev-camunda.abcdefg.dev/auth/realms/camunda-platform/protocol/openid-connect/token',
      audience: 'zeebe-api',
      scope: 'camunda-identity',
      clientId: '******',
      clientSecret: '******'
    }
  }
} Error: 14 UNAVAILABLE: No connection established. Last error: unable to verify the first certificate (2024-11-18T19:19:39.181Z)
    at callErrorFromStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
    at Object.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/client.js:192:76)
    at /Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/call-interface.js:78:35
    at Object.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/zeebe-node/dist/lib/GrpcClient.js:97:36)
    at InterceptingListenerImpl.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/call-interface.js:73:23)
    at Object.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
    at Object.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
    at /Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/resolving-call.js:99:78
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/client.js:160:32)
    at ServiceClientImpl.<anonymous> (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at /Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/zeebe-node/dist/lib/GrpcClient.js:272:47
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 14,
  details: 'No connection established. Last error: unable to verify the first certificate (2024-11-18T19:19:39.181Z)',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Does this help?

11

Yes, I think it does.

The error message is saying that the first hop (End-Device → Ingress) cannot be created validly, because the certificate for your Ingress isn’t valid. Until you have a valid certificate, you’re not going to be able to go much further.

12

Hello @GotnOGuts
Finally got a valid certificate. With Desktop Modeler, I get this error:

2024-11-21T01:55:07.142Z  ERROR app:zeebe-api connection check failed {
  parameters: {
    endpoint: {
      type: 'selfHosted',
      authType: 'oauth',
      url: 'http://zeebe-nginx-dev-camunda.abcdefg.dev',
      oauthURL: 'https://dev-camunda.abcdefg.dev/auth/realms/camunda-platform/protocol/openid-connect/token',
      audience: 'zeebe-api',
      scope: 'camunda-identity',
      clientId: '******',
      clientSecret: '******'
    }
  }
} Error: 13 INTERNAL: Received RST_STREAM with code 0
    at callErrorFromStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/call.js:31:19)
    at Object.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/client.js:192:76)
    at /Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/call-interface.js:78:35
    at Object.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/zeebe-node/dist/lib/GrpcClient.js:97:36)
    at InterceptingListenerImpl.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/call-interface.js:73:23)
    at Object.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141)
    at Object.onReceiveStatus (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181)
    at /Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/resolving-call.js:99:78
    at process.processTicksAndRejections (node:internal/process/task_queues:77:11)
for call at
    at ServiceClientImpl.makeUnaryRequest (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/client.js:160:32)
    at ServiceClientImpl.<anonymous> (/Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/@grpc/grpc-js/build/src/make-client.js:105:19)
    at /Applications/Camunda Modeler.app/Contents/Resources/app.asar/node_modules/zeebe-node/dist/lib/GrpcClient.js:272:47
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  code: 13,
  details: 'Received RST_STREAM with code 0',
  metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Here is the zbctl command and the error

zbctl status --address "zeebe-nginx-dev-camunda.abcdefg.dev" --host "zeebe-nginx-dev-camunda.abcdefg.dev" --port 443 --clientId zeebe --clientSecret ******** --scope "camunda-identity" --authzUrl "https://dev-camunda.abcdefg.dev/auth/realms/camunda-platform/protocol/openid-connect/token" --audience "zeebe-api" --certPath "good-dev-camunda.pem"

Error is:

Error: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 0 (); malformed header: missing HTTP content-type

Any thoughts? This is with a nginx-ingress on GCP using tls,