Camunda 8 on Google Cloud / GKE: Health Check issues

[jack]

I’m trying to get Camunda 8 running on Google Kubernetes Engine which turns out to be quite hard to do. There are a lot of not-so-obvious puzzle pieces, few from Camunda, few from GKE …

But I’m making some progress, although I’m now stuck on a really annoying issue: all pods are running, I can confirm they are working when I make calls to the services inside the cluster - but the GKE ingress just does not work. The reason is that the GKE Load Balancer sees the backend as unhealthy.

The reason is that the health check that GKE creates for the load balancer to see if a backend is healthy goes to the root path (/) on port 80, for example for Operate. But nothing on port 80 can be called without authentication, so the health check never gets back a 200.

The liveliness probes live on 9600 in Camunda, for example again also in Operate. I created an ingress rule for both 80 and 9600 to test this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: camunda-ingress
spec:
  rules:
  - http:
      paths:
      - path: /operate
        pathType: Prefix
        backend:
          service:
            name: camunda-operate
            port:
              number: 80
      - path: /operate/health
        pathType: Prefix
        backend:
          service:
            name: camunda-operate
            port:
              number: 9600

The backend for 9600 is healthy. But for 80 it’s not. So the obvious thing is to change the health check in GCP for 80 also to 9600 … but well, GCP does not allow this. The healtch check gets reverted immeadiately back to SERVING_PORT after I change the port manually.

Well, and as long as I’m not getting the health check to execute successfully, no traffic will be routed from the LB to the cluster.

Has anybody running Camunda 8 on GKE? How did you solve this?

[Alex_Voloshyn]

Hi @jack
I did not try it myself but I would take a look at BackendConfig to make it working with your ingress.

Regards,
Alex

[jack]

Hi @Alex_Voloshyn

Thank you very much, that worked!

Just as a reference for others, here’s my BackendConfig (I used it only for Operate for now):

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: camunda-hc-backendconfig
spec:
  healthCheck:
    checkIntervalSec: 30
    timeoutSec: 10
    unhealthyThreshold: 2
    healthyThreshold: 1
    type: HTTP
    requestPath: /actuator/health/readiness
    port: 9600

And here’s the required annotation for the service which I added to the values.yaml for helm:

  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      **beta.cloud.google.com/backend-config: '{"default": "camunda-hc-backendconfig"}' # Attach the backend config to the service**