Camunda 8.8 - Orchestration Components not found when Zeebe Cluster size >1

Camunda 8 Topics
, ,
1

Hi,

I updated the contextPath of orchestration from “/” to “/orchestration” because too many unsolvable problems occured and after some testing I got everything running with 1 Zeebe pod. When I increase the Zeebe Cluster size to 2 or 3, the apps like operate/tasklist/identity show only an 404 error.

It seems that the internal gateway or the Traefik ingress controller are sending the web requests to different pods after the Entra ID authentication proess is done and with this the session isn’t recognized anymore.

The installation in my Kubernetes cluster went really fast and well but I cannot reach any orchestration webapp at all, only 404 Not found error occurs, which seems to sent from the Camunda apps/gateway^^ Everything else seems working fine and is reachable like /mgmtidentity and /optimize via the inbuild ingress.

Here is my current config:

global:
  security:
    authentication:
      method: oidc
  
  ingress:
    enabled: true
    className: ${ingress_controller}
    annotations:
      cert-manager.io/cluster-issuer: ${clusterissuer_name}
      traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
      traefik.ingress.kubernetes.io/router.tls: "true"
      acme.cert-manager.io/http01-edit-in-place: "false"
      acme.cert-manager.io/http01-ingress-class-name: ${ingress_controller}
      cert-manager.io/common-name: "${namespace_name}.${service_uri}${stage_uri}${domain}"
      cert-manager.io/duration: "720h"
      cert-manager.io/renew-before: "24h"
      cert-manager.io/subject-organizations: ${subject_organizations}
      cert-manager.io/subject-organizationalunits: ${subject_organizationalunits}
      cert-manager.io/subject-localities: ${subject_localities}
      cert-manager.io/subject-provinces: ${subject_provinces}
      cert-manager.io/subject-countries: ${subject_countries}
      cert-manager.io/private-key-size: "4096"
    host: "${namespace_name}.${service_uri}${stage_uri}${domain}"
    pathType: Prefix
    path: "/"
    tls:
      enabled: true
      secretName: "${namespace_name}-${stage}-tls"
  
  elasticsearch:
    enabled: true
    external: true
    prefix: "${namespace_name}-${stage}"
    tls:
      enabled: true
      existingSecret: elastic-jks
    auth:
      username: "${namespace_name}-${stage}"
      secret:
        existingSecret: "${namespace_name}-${stage}"
        existingSecretKey: elasticsearch-instance-secret
    url:
      protocol: ${elastic_service_protocol}
      host: ${elastic_service_url}
      port: ${elastic_service_port}        
  
  identity:
    auth:
      enabled: true
      issuer: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      issuerBackendUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      tokenUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
      authUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/authorize"
      jwksUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/discovery/v2.0/keys"
      type: "MICROSOFT"
        
      identity:
        clientId: ${microsoft_client_id_identity}
        audience: ${microsoft_client_id_identity}
        secret:
          existingSecret: "${namespace_name}-${stage}"
          existingSecretKey: "identity-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/mgmtidentity"
        initialClaimName: "groups"
        initialClaimValue: ${initial_claim_group}
        
      optimize:
        clientId: ${microsoft_client_id_optimize}
        audience: ${microsoft_client_id_optimize}
        secret:
          existingSecret: "${namespace_name}-${stage}"
          existingSecretKey: "optimize-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/optimize"

      console:
        clientId: ${microsoft_client_id_console}
        audience: ${microsoft_client_id_console}
        secret:
          existingSecret: "${namespace_name}-${stage}"
          existingSecretKey: "console-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/console"
        wellKnown: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0/.well-known/openid-configuration"

orchestration:
  enabled: true
  contextPath: "/orchestration"
  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration"

  index:
    prefix: "${namespace_name}-${stage}"

  profiles:
    operate: true
    tasklist: true

  clusterSize: "3"
  partitionCount: "3"
  replicationFactor: "3"

  configuration: |-
    camunda:
      data:
        secondary-storage:
          elasticsearch:
            url: "${elastic_service_protocol}://${elastic_service_url}:${elastic_service_port}"
            username: "${namespace_name}-${stage}"
            password: "${elastic_service_password}"
            security:
              enabled: true
              self-signed: true    
              verify-hostname: false  
            index-prefix: "${namespace_name}-${stage}"
      security:
        authentication:
          method: "oidc"
          oidc:
            client-id: ${microsoft_client_id_orchestration}
            issuer-uri: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
            client-secret: "${orchestration_secret}"

            audiences: [ "${microsoft_client_id_orchestration}", "${microsoft_client_id_webmodeler}" ]
            redirect-uri: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration/sso-callback"  
            client-id-claim: azp 
            username-claim: preferred_username
            groups-claim: groups
            scope: [ "${microsoft_client_id_orchestration}/.default", "openid", "profile", "offline_access" ]
            prefer-username-claim: true
        initialization:
          defaultRoles:
            admin:
              users:
                - ${initial_claim_user}
              groups:
                - ${initial_claim_group}
            connectors:
              clients:
                - "${microsoft_client_id_orchestration}"

    zeebe:
      broker:
        gateway:
          enable: true
        exporters:
          elasticsearch:
            classname: io.camunda.zeebe.exporter.ElasticsearchExporter
            args:
              url: "https://${elastic_service_url}:${elastic_service_port}"
              index:
                prefix: "${namespace_name}-${stage}-zeebe"
                createTemplate: true
              authentication:
                username: "${namespace_name}-${stage}"
                password: "${elastic_service_password}"

  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn
    - name: MANAGEMENT_SERVER_BASEPATH
      value: "/orchestration"
    - name: SERVER_SERVLET_CONTEXTPATH
      value: "/orchestration"
    - name: CAMUNDA_REST_ENABLED
      value: "true"
    - name: JAVA_TOOL_OPTIONS
      value: >-
        -Djavax.net.ssl.trustStore=/opt/certs/externaldb.jks
        -Djavax.net.ssl.trustStorePassword=${truststore_password}
        -Djavax.net.ssl.trustStoreType=JKS
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"

  extraVolumes:
    - name: elastic-ca
      secret:
        secretName: elastic-jks   
        defaultMode: 0444

  extraVolumeMounts:
    - name: elastic-ca
      mountPath: /opt/certs
      readOnly: true

optimize:
  enabled: true
  contextPath: "/optimize"
  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/optimize" 

  env:
    - name: LOGGING_LEVEL_ROOT
      value: debug # info
    - name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SETTINGS_INDEX_PREFIX
      value: ${namespace_name}-${stage}-optimize

identity:
  enabled: true
  contextPath: "/mgmtidentity"

  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/mgmtidentity"  
  
  nodeSelector:
      k8s-nodepool: ${nodepool_name}
  
  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info debug
    - name: PATH
      value: "/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    - name: JAVA_HOME
      value: "/opt/java/openjdk"
    - name: SERVER_TOMCAT_MAXHTTPRESPONSEHEADERSIZE
      value: "32KB"
    - name: SPRING_PROFILES_ACTIVE
      value: "oidc"
    - name: CAMUNDA_IDENTITY_AUDIENCE
      value: ${microsoft_client_id_identity}
   
  externalDatabase:
    enabled: true
    host: "${postgresql_service_url}"
    port: ${postgresql_service_port}
    username: "${identity_database_username}"
    database: "${identity_database}?sslmode=require"
    secret:
      existingSecret: "${namespace_name}-${stage}"
      existingSecretKey: identity-db-secret

connectors:
  enabled: true
  contextPath: "/connectors"

  readinessProbe:
    enabled: false
    probePath: "/actuator/health/readiness"

  configuration: |-
    camunda:
      client:
        mode: self-managed
        grpc-address: "http://${namespace_name}-${stage}-zeebe-gateway:26500"
        rest-address: "http://${namespace_name}-${stage}-zeebe-gateway:8080"
        auth:
          method: oidc
          client-id: ${microsoft_client_id_connectors}
          client-secret: "${connector_secret}"
          token-url: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
          audience: ${microsoft_client_id_orchestration}
          scope: "${microsoft_client_id_orchestration}/.default"

  security:
    authentication:
      method: "oidc"
      oidc:
        clientId: ${microsoft_client_id_connectors}
        audience: ${microsoft_client_id_orchestration}
        secret:
          existingSecret: "${namespace_name}-${stage}"
          existingSecretKey: "orchestration-secret"
        tokenScope: "${microsoft_client_id_orchestration}/.default"  

  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info debug
    - name: CAMUNDA_CLIENT_AUTH_METHOD
      value: "oidc"
    - name: OPERATE_CLIENT_AUTHURL
      value: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"

I don’t know if this is a bug, missing config items or a Traefik issue, which seems unlikely.

I hope someone is able to help me.

Kind Regards,

Julian

2

Hi Julian,

I can see the issue with your configuration. When you change the orchestration.contextPath from / to /orchestration in Camunda 8.8, there are several important configuration changes you need to make to ensure everything works properly.

The Problem

In Camunda 8.8+, the orchestration.contextPath controls all orchestration web applications. When you set:

orchestration:
  contextPath: "/orchestration"

The applications are now served at:

  • Operate: https://your-domain/orchestration/operate
  • Tasklist: https://your-domain/orchestration/tasklist
  • REST API: https://your-domain/orchestration/v2

Required Configuration Fixes

Looking at your configuration, I can identify several issues that need to be addressed:

1. OIDC Redirect URL Issue

Your current OIDC redirect URL configuration is incorrect:

# Current (incorrect)
redirect-uri: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration/sso-callback"

However, you’re using the inline configuration approach under orchestration.configuration, but you should be using the Helm chart OIDC configuration instead.

Fix: Replace your inline OIDC configuration with the proper Helm chart structure:

orchestration:
  enabled: true
  contextPath: "/orchestration"
  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration"
  
  # Remove the inline configuration section and use this instead:
  security:
    authentication:
      method: oidc
      oidc:
        clientId: ${microsoft_client_id_orchestration}
        audience: ${microsoft_client_id_orchestration}
        secret:
          existingSecret: "${namespace_name}-${stage}"
          existingSecretKey: "orchestration-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration"
        issuerUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
        tokenUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
        authUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/authorize"
        jwksUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/discovery/v2.0/keys"
        usernameClaim: "preferred_username"
        groupsClaim: "groups"
        scope: ["${microsoft_client_id_orchestration}/.default", "openid", "profile", "offline_access"]

2. Microsoft Azure AD Redirect URI Registration

In your Microsoft Azure AD app registration for orchestration, you need to register this exact redirect URI:

https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration/sso-callback

3. Connectors Configuration Update

Your connectors configuration also needs to be updated to use the new REST endpoint:

connectors:
  configuration: |-
    camunda:
      client:
        mode: self-managed
        grpc-address: "http://${namespace_name}-${stage}-zeebe-gateway:26500"
        rest-address: "http://${namespace_name}-${stage}-zeebe-gateway:8080/orchestration"  # Add /orchestration
        # ... rest of config

4. Environment Variables Update

Update your orchestration environment variables:

orchestration:
  env:
    - name: MANAGEMENT_SERVER_BASEPATH
      value: "/orchestration"
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    # Add these for proper context path handling:
    - name: SERVER_SERVLET_CONTEXTPATH
      value: "/orchestration"

Testing the Fix

After applying these changes:

  1. Access URLs:

    • Operate: https://your-domain/orchestration/operate
    • Tasklist: https://your-domain/orchestration/tasklist
    • REST API: https://your-domain/orchestration/v2

  2. Check logs for any authentication or routing errors

  3. Verify OIDC flow works by accessing the applications and ensuring the redirect to Microsoft login works properly

Alternative: Revert to Default

If you continue having issues, you might consider reverting to the default contextPath / temporarily while troubleshooting, as this is the most tested configuration.

The key issue is that Camunda 8.8 consolidated all orchestration components under a single contextPath, and both your Helm configuration and Azure AD redirect URIs need to be updated accordingly.

Let me know if you need help with any of these configuration changes or if you encounter any specific errors after applying the fixes!

References:

3

The issue was in the end that the ingress needs an annotation for a sticky cookies middleware. After that the traffic is routed correctly.

4

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.