All components in Camunda 8.6 not working in GKE using single gke-ingress

Camunda 8 Topics
1

Hello Camunda,
We’ve been struggling to get our Camunda 8.6 environment running in GKE using gke-ingress.

The only two components that seem to be working using gke-ingress for us are:

  • /optimize
  • /auth

For tasklist and operate, we are experiencing this error when navigating to /tasklist and /operate

Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.

Any pointers will help? Here are the two values files we used:

ingress-values.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: camunda-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "camunda-dev-ingress"
    kubernetes.io/ingress.class: "gce"
    networking.gke.io/managed-certificates: "dev-camunda"
spec:
  rules:
    - host: dev-camunda.abcdefg.dev
      http:
        paths:
        - path: /operate
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-operate
              port:
                number: 80
        - path: /tasklist
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-tasklist
              port:
                number: 80
        - path: /optimize
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-optimize
              port:
                number: 80
        - path: /auth
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-keycloak
              port:
                number: 80

Camunda-values.yaml

global:
  identity:
    auth:
      publicIssuerUrl: "https://dev-camunda.abcdefg.dev/auth/realms/camunda-platform"
      operate:
        redirectUrl: "https://dev-camunda.abcdefg.dev/operate"
      tasklist:
        redirectUrl: "https://dev-camunda.abcdefg.dev/tasklist"
      optimize:
        redirectUrl: "https://dev-camunda.abcdefg.dev/optimize"
  # ingress:
  #   enabled: true
  #   className: gce
  #   host: "dev-camunda.abcdefg.dev"

operate:
  contextPath: "/operate"
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      beta.cloud.google.com/backend-config: '{"default": "camunda-operate-backendconfig"}' # Attach the backend config to the service

tasklist:
  contextPath: "/tasklist"
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      beta.cloud.google.com/backend-config: '{"default": "camunda-tasklist-backendconfig"}' # Attach the backend config to the service

optimize:
  contextPath: "/optimize"
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      beta.cloud.google.com/backend-config: '{"default": "camunda-optimize-backendconfig"}' # Attach the backend config to the service

identityKeycloak:
  contextPath: "/auth"
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created

zeebeGateway:
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      cloud.google.com/app-protocols: '{"my-port":"HTTP2"}'
  ingress:
    enabled: true
    className: gce
    host: "zeebe-dev-camunda.abcdefg.dev"

identity:
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      beta.cloud.google.com/backend-config: '{"default": "camunda-identity-backendconfig"}' # Attach the backend config to the service

Thanks and best regards,
Atul

2

Enable health check logging on the ingress for outbound traffic goes to Camunda gateway.

3

Thank you Chander. We configured our firewall for health check. After reviewing documentation related to operate and tasklist Monitoring https://docs.camunda.io/docs/self-managed/operate-deployment/operate-configuration/ we tried all these three health check endpoints, but endpoints checks are still failing. The health check logs do not indicate any warnings or errors.

The health check paths are /operate and /tasklist respectively. Both these endpoints are still failing the health-check test.

We tried using these 3 paths for the health check probes, but they all failed the health check

  • /operate/actuator/prometheus
  • /operate/actuator/health/readiness
  • /operate/actuator/health/liveness
  • /operate

We may be missing something too obvious. Any other pointers for us?

Regards,
Atul

4

Try enabling debug log at the pod level to get more details. Did you check pod logs logging into pods. they will give some hints.

Are you using ElK or Opensearch? Are you using existing or new ELK stack? Make sure necessary policy exists to communicate to ELK.

Please do check the logs from bottom up approach, starting with ELK

5

Thank you Chander. We got tasklist and operate working. We now have these components successfully running:

  • /operate
  • /tasklist
  • optimize
  • /auth
  • zeebe

But for some reason, Health-Check for identity is still failing even though we confirmed we are able to successful check for readiness at the service level IP:82/identity/actuator/health. Service maps port 82 to 8082.

We are using a contextPath /identity. GCP’s Healthcheck FW rule includes port 82. Here are the values.yaml for identity:

Camunda

global:
  identity:
    auth:
      publicIssuerUrl: "https://dev-camunda.abcdefg.dev/auth/realms/camunda-platform"
      operate:
        redirectUrl: "https://dev-camunda.abcdefg.dev/operate"
      tasklist:
        redirectUrl: "https://dev-camunda.abcdefg.dev/tasklist"
      optimize:
        redirectUrl: "https://dev-camunda.abcdefg.dev/optimize"
      identity:
        redirectUrl: "https://dev-camunda.abcdefg.dev/identity"
  

identity:
  contextPath: "/identity"
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      cloud.google.com/backend-config: '{"default": "camunda-hc-identity"}' # Attach the backend config to the service

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: camunda-ingress
  annotations:
    kubernetes.io/ingress.global-static-ip-name: "camunda-dev-ingress"
    kubernetes.io/ingress.class: "gce"
    networking.gke.io/managed-certificates: "dev-camunda"
spec:
  rules:
    - host: dev-camunda.abcdefg.dev
      http:
        paths:
        - path: /operate
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-operate
              port:
                number: 80
        - path: /tasklist
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-tasklist
              port:
                number: 80
        - path: /optimize
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-optimize
              port:
                number: 80
        - path: /auth
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-keycloak
              port:
                number: 80
        - path: /identity
          pathType: Prefix
          backend:
            # This assumes http-svc exists and routes to healthy endpoints
            service:
              name: camunda-identity
              port:
                number: 80

Any insight on why identity is not working even through the probe works from within the cluster from another pod to the service :82/actuator/health

Regards,
Atul

6

Does /identity contextPath not work?

Finally figured out the Healthcheck for Identity. But whenever I navigate to https://domain.com/identity, it get’s redirected to http://localhost:8080/auth

What am I doing wrong? I have this

identity:
  contextPath: "/identity"
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      cloud.google.com/backend-config: '{"default": "camunda-hc-identity"}'

and an ingress

https://domain.com/identity to http://k8s-identity-svc:80/

Thoughts?

7

Finally got it to work but only with port-forwarding. Also, within GCP, the clusterIP range is outside of the allowed “External IPs”, so for now changed to SSL-Mode = none in Keycloak.

Also, added fullURL: "https://identity.domain/identity"

Still trying to figure out how to leverage gce-ingress over SSL for zeebe-gateway.

Regards,
Atul

8

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.