I’m nearly through setting up Camunda in Google Kubernetes Engine, but the last puzzle piece I just can’t get setup is to enable Zeebe access via GKE ingress. It works fine with port forwarding.
I created the ingress and I can see that requests actually reach the Zeebe-Gateway pods, but they fail with the error HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 16030100e8010000e403037227fd6c1eca59ff037b419a2a.
I followed the pitfalls notice here and enabled HTTP/2 for the backend gateway port so the GKE load balancer forwards HTTP/2 traffic to the backend:
It was still not working and the error message above normally would suggest the GRPC service expected non-TLS traffic while actually receiving TLS traffic. Looking through GCP documentation, it is mentioned that the GCP load balancer always forwards HTTP/2 traffic to the backend via TLS, no way around it and no HTTPS offloading happening. It will gladly accept any certificate, also self signed ones, so that should be no issue. I was able to add certs to the pods easily. But I can’t figure out how I can enable TLS for the Zeebe-Gateway … I tried something like this as I found some mentions that this should work:
I did go over this - as I’m using gke-ingress, I somehow thought the ingress section in all camunda services is not relevant as this applies to Camunda’s OOB ingress.
But now that you mention it, if I’d provide className: gce instead of className: nginx this might apply to GKE’s ingress too?
Finally got zeebe-gateway working albeit with port-forwarding. Still struggling getting gce-ingresss to allow connection of zeebe gateway over SSL, even after importing the SSL certificates into Keychain trusted stores.
Working through it. Both zbctl and DesktopModeler failing to connect over SSL.
How are you attempting to deploy from Modeler?
Modeler will default to port 26500, but an ingress doesn’t listen for that port… you have to reconfigure zbctl or Desktop Modeler to connect to https or http (depending on how you have your ingress configured)
I’d think the above would work. However, that does not work. With port-forwarding, and using http://localhost:26500, works with the exact same options in zbctl with the obvious changes.
I think the issue is what Jack described earlier - gce-ingress does not allow non-SSL traffic for zeebe
Ok, but what is the return from running the command that way?
Note: I don’t have access to GCE environment, and am just another user like you, so would need you to provide complete information to make suggestions.
Since Ingress is doing the TLS termination, you’d have to tell your Ingress to use TLS to talk to the Gateway rather than http.
The error message is saying that the first hop (End-Device → Ingress) cannot be created validly, because the certificate for your Ingress isn’t valid. Until you have a valid certificate, you’re not going to be able to go much further.
Unfortunately, I don’t have more info on this.
It looks like the Ingress didn’t like the connection. There’s likely more information in your nginx-ingress logs.
Hello @jack
We finally got gce-ingress working, albeit using a separate ingress. But the point is, we got TLS working for zeebe-gateway, by manually adding these two into the config-map of camunda-zeebe-gateway for application.yaml
However, that broke camunda-connectors service. Seems like any zeebe client working through zeebe-gateway requires TLS, and since camunda-connectors is a client, it probably needs TLS as well. I could not find any documentation in Camunda docs that discusses setting up TLS for connectors.