Camunda 8.8 | Identity pod is running in permission error

Hi,

I’m currently installing Camunda 8.8 in a Kubernetes Cluster with Entra ID Authentication and ElasticSearch for data and Cloudnative-pg Postgres database for configuration database.

I used my working Camunda 8.7 config yaml files for the Helm installation and migrated them to 8.8 values.

Zeebe and Optimize pods are running but Identity is constnatly rebooting due to the following error:

2025-11-14 09:47:39.274 INFO 1 — [ main] i.c.i.Application : Starting Application using Java 17.0.16 with PID 1 (/app/identity.jar started by camunda in /app)
2025-11-14 09:47:39.293 INFO 1 — [ main] i.c.i.Application : The following 1 profile is active: “oidc”
2025-11-14 09:47:46.719 WARN 1 — [ main] trationDelegate$BeanPostProcessorChecker : Bean ‘globalMethodSecurityConfig’ of type [io.camunda.identity.security.config.GlobalMethodSecurityConfig$$SpringCGLIB$$0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying). Is this bean getting eagerly injected/applied to a currently created BeanPostProcessor [healthEndpointGroupsBeanPostProcessor]? Check the corresponding BeanPostProcessor declaration and its dependencies/advisors. If this bean does not have to be post-processed, declare it with ROLE_INFRASTRUCTURE.
2025-11-14 09:47:49.096 WARN 1 — [ main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2025-11-14 09:48:01.834 WARN 1 — [ main] i.c.i.c.AppConfig : Expected a valid license but encountered an invalid one instead. Please check the license key and try again.
2025-11-14 09:48:09.229 INFO 1 — [ main] i.c.i.Application : Started Application in 33.227 seconds (process running for 37.245)
2025-11-14 09:48:10.194 ERROR 1 — [ main] o.s.b.SpringApplication : Application run failed

java.lang.IllegalArgumentException: permission.notFound
at io.camunda.identity.impl.oidc.OidcRoleServiceImpl.lambda$assignPermission$3(OidcRoleServiceImpl.java:84) ~[!/:?]at java.base/java.util.Optional.orElseThrow(Unknown Source) ~[?:?]at io.camunda.identity.impl.oidc.OidcRoleServiceImpl.lambda$assignPermission$4(OidcRoleServiceImpl.java:84) ~[!/:?]at java.base/java.util.Optional.ifPresentOrElse(Unknown Source) ~[?:?]at io.camunda.identity.impl.oidc.OidcRoleServiceImpl.assignPermission(OidcRoleServiceImpl.java:78) ~[!/:?]at io.camunda.identity.impl.oidc.initializer.service.RoleInitializerService.lambda$initialize$0(RoleInitializerService.java:38) ~[!/:?]at java.base/java.lang.Iterable.forEach(Unknown Source) ~[?:?]at io.camunda.identity.impl.oidc.initializer.service.RoleInitializerService.lambda$initialize$1(RoleInitializerService.java:35) ~[!/:?]at java.base/java.util.ArrayList.forEach(Unknown Source) ~[?:?]

I couldn’t find any solution to this error, I built the config with the official documentation, here is my code:

global:  
identity:
    auth:
      enabled: true
      # Docu for Entra ID: https://docs.camunda.io/docs/self-managed/identity/configuration/connect-to-an-oidc-provider/?authPlatform=microsoftEntraId&optionsType=helm
      issuer: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      # this is used for container to container communication.
      issuerBackendUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      tokenUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
      authUrl: https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/authorize
      jwksUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/discovery/v2.0/keys"
      type: "MICROSOFT"
      
      publicIssuerUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
        
      identity:
        clientId: ${microsoft_client_id_identity}
        audience: ${microsoft_client_id_identity}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "identity-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"
        initialClaimName: preferred_username
        initialClaimValue: "mail@mail.com"
identityKeycloak:
  enabled: false

identity:
  enabled: true

  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"
  
  contextPath: "/identity"
  
  nodeSelector:
      k8s-nodepool: ${nodepool_name}
  
  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info debug
    - name: PATH
      value: "/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    - name: JAVA_HOME
      value: "/opt/java/openjdk"
    - name: SERVER_TOMCAT_MAXHTTPRESPONSEHEADERSIZE
      value: "32KB"

  command: ["java","-jar","identity.jar"]

  extraVolumes:
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podinfo
   
  externalDatabase:
    enabled: true
    host: "${postgresql_service_url}"
    port: ${postgresql_service_port}
    username: "${identity_database_username}"
    database: "${identity_database}?sslmode=require"
    existingSecret: ${namespace_name}
    existingSecretPasswordKey: identity-db-secret

identityPostgresql:
  enabled: false

  nodeSelector:
      k8s-nodepool: ${nodepool_name}

I have no clue where this error is coming from, database in postgres SQL is created and filled with values, Entra ID app registration has user permissions.
Kind Regards!

Hi @Cris_Ron,

I can see you’re encountering a permission.notFound error during Identity startup in Camunda 8.8 with Microsoft Entra ID. This appears to be a role initialization issue where the system is trying to assign permissions that don’t exist in the database.

Problem Classification: Problem - Identity pod startup failure

Potential Solutions

Based on similar issues reported with Camunda 8.8 Identity, here are several things to check:

1. Enable Resource Permissions

Add this to your Identity configuration:

identity:
  env:
    - name: RESOURCE_PERMISSIONS_ENABLED
      value: "true"

2. Configure Initial Admin Role Assignment

Camunda 8.8 requires explicit admin role assignment during initialization. Add this to your configuration:

orchestration:
  security:
    initialization:
      defaultRoles:
        admin:
          users:
            - "mail@mail.com"  # Your initial user email
    authorizations:
      enabled: true

3. Verify Database Schema

Since you mentioned the PostgreSQL database is created and filled with values, please verify:

  • The permissions table exists and is properly populated
  • The database schema matches Camunda 8.8 requirements
  • Check if there are any migration issues from 8.7 to 8.8

4. Check OIDC Claim Configuration

Ensure your initial claim configuration matches what Entra ID provides:

global:
  identity:
    auth:
      identity:
        initialClaimName: preferred_username  # or email, depending on your Entra ID setup
        initialClaimValue: "mail@mail.com"

5. Known Limitation

:warning: Important: According to the official documentation, there’s a known limitation: “Use of Microsoft Entra ID as an OpenID Connect (OIDC) identity provider in Self-Managed is not currently supported with IDP.”

This might be related to your issue, especially with the dual Identity architecture introduced in 8.8.

Debugging Steps

  1. Check database contents: Verify the permissions and roles tables in your PostgreSQL database
  2. Enable debug logging: Add LOGGING_LEVEL_ROOT: debug to see more detailed startup logs
  3. Verify Entra ID claims: Check what claims your Entra ID app registration is sending

Next Steps

If the above solutions don’t resolve the issue, this appears to be a specific implementation problem that may require deeper investigation. I’d recommend:

  1. Trying the configuration changes above
  2. If the issue persists, filing a bug report at Camunda GitHub Issues with:
    • Complete startup logs
    • Database schema/content verification
    • Your full Helm configuration (sanitized)

Could you try the resource permissions and admin role configuration changes first and let us know if that resolves the startup issue?

References:

Hi @zee-bot ,
the first bullet point is for a configuration with local identity management not for Microsoft Entra ID integration.
The initial admin role assignment is configured.

The database schema and the required table exists and are being filled with valid and recent values.

The OIDC claim configuration is in place and configured as described.

The known limitation is only valid for an older version of Camunda, the currently used verison 8.8 is supporting Entra ID.

Here is my current full configuration:

global:
  multitenancy:
    ## @param global.multitenancy.enabled if true, then enable multitenancy in all applicable components.
    enabled: false

  security:
    authentication:
      method: "oidc"
  
  # image:
  #   ## @param global.image.registry Can be used to set container image registry.
  #   registry: ${image_registry_url}${image_registry_port_core_uri}
  ingress:
    ## @param global.ingress.enabled if true, an ingress resource is deployed. Only useful if an ingress controller is available, like Ingress-NGINX.
    enabled: true
    ## @param global.ingress.className Ingress.className defines the class or configuration of ingress which should be used by the controller
    className: ${ingress_controller}
    ## @param global.ingress.annotations [object] defines the ingress related annotations, consumed mostly by the ingress controller
    annotations:
      cert-manager.io/cluster-issuer: ${clusterissuer_name}
      traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
      traefik.ingress.kubernetes.io/router.tls: "true"
      acme.cert-manager.io/http01-edit-in-place: "false"
      acme.cert-manager.io/http01-ingress-class-name: ${ingress_controller}
      cert-manager.io/common-name: "${namespace_name}.${service_uri}${stage_uri}${domain}"
      cert-manager.io/duration: "720h"
      cert-manager.io/renew-before: "24h"
      cert-manager.io/subject-organizations: ${subject_organizations}
      cert-manager.io/subject-organizationalunits: ${subject_organizationalunits}
      cert-manager.io/subject-localities: ${subject_localities}
      cert-manager.io/subject-provinces: ${subject_provinces}
      cert-manager.io/subject-countries: ${subject_countries}
      cert-manager.io/private-key-size: "4096"
    # Ingress.host can be used to define the host of the ingress rule. https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-rules
    ## @param global.ingress.host If not specified the rules applies to all inbound http traffic, if specified the rule applies to that host.
    host: "${namespace_name}.${service_uri}${stage_uri}${domain}"
    tls:
      enabled: true
      secretName: "${release_name}-tls"
  
  elasticsearch:
    enabled: true
    external: true
    tls:
      enabled: true
      existingSecret: elastic-jks           # Secret with CA-Cert for Elasticsearch
    auth:
      username: ${namespace_name}           # Your Elasticsearch instance username
      existingSecret: ${namespace_name}
      existingSecretKey: elasticsearch-instance-secret
    url:
      protocol: ${elastic_service_protocol} # Or 'https' if your Elasticsearch uses TLS/SSL
      host: ${elastic_service_url}          # FQDN of the Elasticsearch service
      port: ${elastic_service_port}         # The port your Elasticsearch service is using (9200 is default for HTTP)
    prefix: ${namespace_name}-zeebe
  
  identity:
    auth:
      enabled: true
      # Docu for Entra ID: https://docs.camunda.io/docs/self-managed/identity/configuration/connect-to-an-oidc-provider/?authPlatform=microsoftEntraId&optionsType=helm
      issuer: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      # this is used for container to container communication.
      issuerBackendUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      tokenUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
      authUrl: https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/authorize
      jwksUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/discovery/v2.0/keys"
      type: "MICROSOFT"
      
      publicIssuerUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
        
      identity:
        clientId: ${microsoft_client_id_identity}
        audience: ${microsoft_client_id_identity}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "identity-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"
        initialClaimName: preferred_username
        initialClaimValue: "jkadm@rznlsw.onmicrosoft.com"
        
      optimize:
        clientId: ${microsoft_client_id_optimize}
        audience: ${microsoft_client_id_optimize}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "optimize-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/optimize"
      
      console:
        clientId: ${microsoft_client_id_console}
        audience: ${microsoft_client_id_console}
        existingSecret:
          name: ${namespace_name}
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/console"
        wellKnown: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0/.well-known/openid-configuration"

      orchestration:
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration"

      connectors:
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "connectors-secret"

console:
  enabled: false

  image:
    registry: ${image_registry_url}${image_registry_port_component_uri}
  
  contextPath: "/console"
  
  nodeSelector:
      k8s-nodepool: ${nodepool_name}

orchestration:
  security:
    authentication:
      method: "oidc"
      oidc:
        clientId: ${microsoft_client_id_orchestration}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        audience: ${microsoft_client_id_orchestration}
        redirectUrl: https://${namespace_name}.${service_uri}${stage_uri}${domain}{{ .Values.orchestration.contextPath }}
        # redirectUrl: "<OC_URL>"
  
        clientIdClaim: azp
        usernameClaim: preferred_username
        groupsClaim: groups
        scope:
          - openid
          - profile
          - offline_access
          - "${microsoft_client_id_orchestration}/.default"
        preferUsernameClaim: true
    initialization:
      defaultRoles:
        admin:
          users:
            - "jkadm@rznlsw.onmicrosoft.com"
        connectors:
          clients:
            - "${microsoft_client_id_orchestration}"
    authorizations:
      enabled: false
 
  migration:
    data:
      enabled: false

  index:
    prefix: "${namespace_name}"

  enabled: true
  profiles:
    operate: true
    tasklist: true

  clusterSize: "3"
  partitionCount: "3"
  replicationFactor: "3"
  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn
    - name: JAVA_TOOL_OPTIONS
      value: >-
        -Djavax.net.ssl.trustStore=/opt/certs/externaldb.jks
        -Djavax.net.ssl.trustStorePassword=${truststore_password}
        -Djavax.net.ssl.trustStoreType=JKS
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    # - name: CAMUNDA_OPERATE_ELASTICSEARCH_INDEXPREFIX
    #   value: ${namespace_name}-operate
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    # - name: CAMUNDA_TASKLIST_ELASTICSEARCH_INDEXPREFIX
    #   value: ${namespace_name}-tasklist
  
  extraVolumes:
    - name: elastic-ca
      secret:
        secretName: elastic-jks         # enthält ca.crt
        defaultMode: 0444

  extraVolumeMounts:
    - name: elastic-ca
      mountPath: /opt/certs
      readOnly: true

  nodeSelector:
      k8s-nodepool: ${nodepool_name}
  
  broker:
    exporters:
      elasticsearch:
        className: io.camunda.zeebe.exporter.ElasticsearchExporter
        args:
          url: "https://${elastic_service_url}:${elastic_service_port}"
          index:
            prefix: "${namespace_name}-zeebe"
            createTemplate: true
  ingress:
    grpc:
      enabled: true
      annotations:
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        cert-manager.io/cluster-issuer: ${clusterissuer_name}
        traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
        traefik.ingress.kubernetes.io/router.tls: "true"
        acme.cert-manager.io/http01-edit-in-place: "false"
        acme.cert-manager.io/http01-ingress-class-name: ${ingress_controller}
        cert-manager.io/common-name: "${namespace_name}-orchestration.${service_uri}${stage_uri}${domain}"
        cert-manager.io/duration: "720h"
        cert-manager.io/renew-before: "24h"
        cert-manager.io/subject-organizations: ${subject_organizations}
        cert-manager.io/subject-organizationalunits: ${subject_organizationalunits}
        cert-manager.io/subject-localities: ${subject_localities}
        cert-manager.io/subject-provinces: ${subject_provinces}
        cert-manager.io/subject-countries: ${subject_countries}
        cert-manager.io/private-key-size: "4096"
      host: "${namespace_name}-orchestration.${service_uri}${stage_uri}${domain}"
      tls:
        enabled: true
        secretName: "${release_name}-orchestration-tls"

zeebe:
  broker:
    gateway:
      enable: true
      network:
        host: 0.0.0.0
        port: 26500
      security:
        enabled: false
        authentication:
          mode: none
    network:
      host: 0.0.0.0
      commandApi:
        port: 26501
      internalApi:
        port: 26502
      monitoringApi:
        port: "9600"
      security:
        enabled: false
        authentication:
          mode: none

optimize:
  enabled: true

  migration:
    env:
    - name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SETTINGS_INDEX_PREFIX
      value: ${namespace_name}-optimize
  
  contextPath: "/optimize"

  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info
    - name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SETTINGS_INDEX_PREFIX
      value: ${namespace_name}-optimize
    - name: OPTIMIZE_ELASTICSEARCH_HOST
      value: "https://${elastic_service_url}:${elastic_service_port}"
    - name: OPTIMIZE_ELASTICSEARCH_HTTP_PORT
      value: "9200"
    
  nodeSelector:
      k8s-nodepool: ${nodepool_name}

identityKeycloak:
  enabled: false

identity:
  enabled: true

  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"
  
  contextPath: "/managementidentity"
  
  nodeSelector:
      k8s-nodepool: ${nodepool_name}
  
  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info debug
    - name: PATH
      value: "/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    - name: JAVA_HOME
      value: "/opt/java/openjdk"
    - name: SERVER_TOMCAT_MAXHTTPRESPONSEHEADERSIZE
      value: "32KB"
    - name: SPRING_PROFILES_ACTIVE
      value: "oidc"

  command: ["java","-jar","identity.jar"]

  extraVolumes:
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podinfo
   
  externalDatabase:
    enabled: true
    host: "${postgresql_service_url}"
    port: ${postgresql_service_port}
    username: "${identity_database_username}"
    database: "${identity_database}?sslmode=require"
    secret:
      existingSecret: ${namespace_name}
      existingSecretKey: identity-db-secret

identityPostgresql:
  enabled: false

  nodeSelector:
      k8s-nodepool: ${nodepool_name}

connectors:
  security:
    authentication:
      method: "oidc"
      oidc:
        clientId: ${microsoft_client_id_orchestration}
        audience: ${microsoft_client_id_orchestration}
        secret:
          # existingSecret: ${namespace_name}
          # existingSecretKey: "connectors-secret"
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        tokenScope: "${microsoft_client_id_orchestration}/.default"

  enabled: true

  contextPath: "/connectors"

  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info debug
    - name: OPERATE_CLIENT_AUTHURL
      value: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
    
  nodeSelector:
      k8s-nodepool: ${nodepool_name}

elasticsearch:
  enabled: false
  master:
    masterOnly: false
    replicaCount: 3
  data:
    replicaCount: 0
  coordinating:
    replicaCount: 0
  ingest:
    enabled: false
    replicaCount: 0
  
  nodeSelector:
      k8s-nodepool: ${nodepool_name}

I tested several other config items but identity is throwing the same error.

Also the Zeebe gateway isn’t running too, similar if I try the embedded or standalone config for it.

Thanks for the detailed configuration! I can see several issues that are likely causing your problems:

Critical Configuration Issues

1. Context Path Mismatch

Your configuration has a mismatch between the Identity context path and redirect URLs:

# Current configuration - INCORRECT
identity:
  contextPath: "/managementidentity"  # ❌ Wrong path
  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"

global:
  identity:
    auth:
      identity:
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"  # ❌ Doesn't match contextPath

Fix: The redirect URLs must match the context paths:

identity:
  contextPath: "/identity"  # ✅ Standard path for Management Identity
  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"

global:
  identity:
    auth:
      identity:
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"  # ✅ Now matches

2. Orchestration Context Path Issue

Your orchestration configuration is missing the context path:

# Add this to your orchestration section:
orchestration:
  contextPath: "/"  # ✅ Root path for Orchestration Cluster
  # ... rest of your config

3. Database Configuration Issue

Your database configuration has an incorrect structure:

# Current - INCORRECT
externalDatabase:
  secret:
    existingSecret: ${namespace_name}
    existingSecretKey: identity-db-secret

# Should be - CORRECT
externalDatabase:
  enabled: true
  host: "${postgresql_service_url}"
  port: ${postgresql_service_port}
  username: "${identity_database_username}"
  database: "${identity_database}?sslmode=require"
  existingSecret: ${namespace_name}          # ✅ Direct property
  existingSecretPasswordKey: identity-db-secret  # ✅ Correct key name

4. Authorization Configuration

You have authorization disabled, but with admin roles configured:

orchestration:
  security:
    authorizations:
      enabled: false  # ❌ Should be true if you want role-based access

Fix:

orchestration:
  security:
    authorizations:
      enabled: true  # ✅ Enable to use the admin roles you've configured

Complete Corrected Configuration Sections

Here are the key sections with corrections:

global:
  identity:
    auth:
      enabled: true
      issuer: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      issuerBackendUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      tokenUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
      authUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/authorize"
      jwksUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/discovery/v2.0/keys"
      type: "MICROSOFT"
      publicIssuerUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      
      identity:
        clientId: ${microsoft_client_id_identity}
        audience: ${microsoft_client_id_identity}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "identity-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"  # ✅ Matches contextPath
        initialClaimName: preferred_username
        initialClaimValue: "jkadm@rznlsw.onmicrosoft.com"
      
      orchestration:
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/"  # ✅ Root path

orchestration:
  contextPath: "/"  # ✅ Add this
  security:
    authentication:
      method: "oidc"
      oidc:
        clientId: ${microsoft_client_id_orchestration}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        audience: ${microsoft_client_id_orchestration}
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/"  # ✅ Root path
        clientIdClaim: azp
        usernameClaim: preferred_username
        groupsClaim: groups
        scope:
          - openid
          - profile
          - offline_access
          - "${microsoft_client_id_orchestration}/.default"
        preferUsernameClaim: true
    initialization:
      defaultRoles:
        admin:
          users:
            - "jkadm@rznlsw.onmicrosoft.com"
        connectors:
          clients:
            - "${microsoft_client_id_orchestration}"
    authorizations:
      enabled: true  # ✅ Enable this

identity:
  enabled: true
  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"
  contextPath: "/identity"  # ✅ Standard path, not "/managementidentity"
  
  externalDatabase:
    enabled: true
    host: "${postgresql_service_url}"
    port: ${postgresql_service_port}
    username: "${identity_database_username}"
    database: "${identity_database}?sslmode=require"
    existingSecret: ${namespace_name}  # ✅ Fixed structure
    existingSecretPasswordKey: identity-db-secret  # ✅ Correct key name

Why This Fixes Your Issues

  1. Permission.notFound Error: This is likely caused by the context path mismatch preventing proper initialization
  2. Zeebe Gateway Issues: The missing orchestration context path and disabled authorizations prevent proper startup
  3. Database Connection: The incorrect database secret structure may cause connection issues

Try these changes and let me know if the Identity pod starts successfully!

References:

I corrected my config file but the database config issue is outdated and the new Camunda version is using the existingSecretKey value entry.

Here is my current config but the same errors regarding identity and gateway remain:

global:
  multitenancy:
    ## @param global.multitenancy.enabled if true, then enable multitenancy in all applicable components.
    enabled: false

  security:
    authentication:
      method: "oidc"
  
  # image:
  #   ## @param global.image.registry Can be used to set container image registry.
  #   registry: ${image_registry_url}${image_registry_port_core_uri}
  ingress:
    ## @param global.ingress.enabled if true, an ingress resource is deployed. Only useful if an ingress controller is available, like Ingress-NGINX.
    enabled: true
    ## @param global.ingress.className Ingress.className defines the class or configuration of ingress which should be used by the controller
    className: ${ingress_controller}
    ## @param global.ingress.annotations [object] defines the ingress related annotations, consumed mostly by the ingress controller
    annotations:
      cert-manager.io/cluster-issuer: ${clusterissuer_name}
      traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
      traefik.ingress.kubernetes.io/router.tls: "true"
      acme.cert-manager.io/http01-edit-in-place: "false"
      acme.cert-manager.io/http01-ingress-class-name: ${ingress_controller}
      cert-manager.io/common-name: "${namespace_name}.${service_uri}${stage_uri}${domain}"
      cert-manager.io/duration: "720h"
      cert-manager.io/renew-before: "24h"
      cert-manager.io/subject-organizations: ${subject_organizations}
      cert-manager.io/subject-organizationalunits: ${subject_organizationalunits}
      cert-manager.io/subject-localities: ${subject_localities}
      cert-manager.io/subject-provinces: ${subject_provinces}
      cert-manager.io/subject-countries: ${subject_countries}
      cert-manager.io/private-key-size: "4096"
    # Ingress.host can be used to define the host of the ingress rule. https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-rules
    ## @param global.ingress.host If not specified the rules applies to all inbound http traffic, if specified the rule applies to that host.
    host: "${namespace_name}.${service_uri}${stage_uri}${domain}"
    tls:
      enabled: true
      secretName: "${release_name}-tls"
  
  elasticsearch:
    enabled: true
    external: true
    tls:
      enabled: true
      existingSecret: elastic-jks           # Secret with CA-Cert for Elasticsearch
    auth:
      username: ${namespace_name}           # Your Elasticsearch instance username
      existingSecret: ${namespace_name}
      existingSecretKey: elasticsearch-instance-secret
    url:
      protocol: ${elastic_service_protocol} # Or 'https' if your Elasticsearch uses TLS/SSL
      host: ${elastic_service_url}          # FQDN of the Elasticsearch service
      port: ${elastic_service_port}         # The port your Elasticsearch service is using (9200 is default for HTTP)
    prefix: ${namespace_name}-zeebe
  
  identity:
    auth:
      enabled: true
      # Docu for Entra ID: https://docs.camunda.io/docs/self-managed/identity/configuration/connect-to-an-oidc-provider/?authPlatform=microsoftEntraId&optionsType=helm
      issuer: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      # this is used for container to container communication.
      issuerBackendUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
      tokenUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
      authUrl: https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/authorize
      jwksUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/discovery/v2.0/keys"
      type: "MICROSOFT"
      
      publicIssuerUrl: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0"
        
      identity:
        clientId: ${microsoft_client_id_identity}
        audience: ${microsoft_client_id_identity}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "identity-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"
        initialClaimName: preferred_username
        initialClaimValue: "jkadm@rznlsw.onmicrosoft.com"
        
      optimize:
        clientId: ${microsoft_client_id_optimize}
        audience: ${microsoft_client_id_optimize}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "optimize-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/optimize"
      
      console:
        clientId: ${microsoft_client_id_console}
        audience: ${microsoft_client_id_console}
        existingSecret:
          name: ${namespace_name}
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/console"
        wellKnown: "https://login.microsoftonline.com/${microsoft_tenant_id}/v2.0/.well-known/openid-configuration"

      orchestration:
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration"

      connectors:
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "connectors-secret"

console:
  enabled: false

  image:
    registry: ${image_registry_url}${image_registry_port_component_uri}
  
  contextPath: "/console"
  
  nodeSelector:
      k8s-nodepool: ${nodepool_name}

orchestration:
  contextPath: "/"
  security:
    authentication:
      method: "oidc"
      oidc:
        clientId: ${microsoft_client_id_orchestration}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        audience: ${microsoft_client_id_orchestration}
        redirectUrl: https://${namespace_name}.${service_uri}${stage_uri}${domain}{{ .Values.orchestration.contextPath }}
        # redirectUrl: "<OC_URL>"
  
        clientIdClaim: azp
        usernameClaim: preferred_username
        groupsClaim: groups
        scope:
          - openid
          - profile
          - offline_access
          - "${microsoft_client_id_orchestration}/.default"
        preferUsernameClaim: true
    initialization:
      defaultRoles:
        admin:
          users:
            - "jkadm@rznlsw.onmicrosoft.com"
        connectors:
          clients:
            - "${microsoft_client_id_orchestration}"
    authorizations:
      enabled: true
 
  migration:
    data:
      enabled: false

  index:
    prefix: "${namespace_name}"

  enabled: true
  profiles:
    operate: true
    tasklist: true

  clusterSize: "3"
  partitionCount: "3"
  replicationFactor: "3"
  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn
    - name: JAVA_TOOL_OPTIONS
      value: >-
        -Djavax.net.ssl.trustStore=/opt/certs/externaldb.jks
        -Djavax.net.ssl.trustStorePassword=${truststore_password}
        -Djavax.net.ssl.trustStoreType=JKS
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    # - name: CAMUNDA_OPERATE_ELASTICSEARCH_INDEXPREFIX
    #   value: ${namespace_name}-operate
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    # - name: CAMUNDA_TASKLIST_ELASTICSEARCH_INDEXPREFIX
    #   value: ${namespace_name}-tasklist
  
  extraVolumes:
    - name: elastic-ca
      secret:
        secretName: elastic-jks         # enthält ca.crt
        defaultMode: 0444

  extraVolumeMounts:
    - name: elastic-ca
      mountPath: /opt/certs
      readOnly: true

  nodeSelector:
      k8s-nodepool: ${nodepool_name}
  
  broker:
    exporters:
      elasticsearch:
        className: io.camunda.zeebe.exporter.ElasticsearchExporter
        args:
          url: "https://${elastic_service_url}:${elastic_service_port}"
          index:
            prefix: "${namespace_name}-zeebe"
            createTemplate: true
  # ingress:
  #   grpc:
  #     enabled: true
  #     annotations:
  #       nginx.ingress.kubernetes.io/ssl-redirect: "true"
  #       cert-manager.io/cluster-issuer: ${clusterissuer_name}
  #       traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
  #       traefik.ingress.kubernetes.io/router.tls: "true"
  #       acme.cert-manager.io/http01-edit-in-place: "false"
  #       acme.cert-manager.io/http01-ingress-class-name: ${ingress_controller}
  #       cert-manager.io/common-name: "${namespace_name}-orchestration.${service_uri}${stage_uri}${domain}"
  #       cert-manager.io/duration: "720h"
  #       cert-manager.io/renew-before: "24h"
  #       cert-manager.io/subject-organizations: ${subject_organizations}
  #       cert-manager.io/subject-organizationalunits: ${subject_organizationalunits}
  #       cert-manager.io/subject-localities: ${subject_localities}
  #       cert-manager.io/subject-provinces: ${subject_provinces}
  #       cert-manager.io/subject-countries: ${subject_countries}
  #       cert-manager.io/private-key-size: "4096"
  #     host: "${namespace_name}-orchestration.${service_uri}${stage_uri}${domain}"
  #     tls:
  #       enabled: true
  #       secretName: "${release_name}-orchestration-tls"

zeebe:
  broker:
    gateway:
      enable: true
      network:
        host: 0.0.0.0
        port: 26500
      security:
        enabled: false
        authentication:
          mode: none
    network:
      host: 0.0.0.0
      commandApi:
        port: 26501
      internalApi:
        port: 26502
      monitoringApi:
        port: "9600"
      security:
        enabled: false
        authentication:
          mode: none

optimize:
  enabled: true

  migration:
    env:
    - name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SETTINGS_INDEX_PREFIX
      value: ${namespace_name}-optimize
  
  contextPath: "/optimize"

  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info
    - name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SETTINGS_INDEX_PREFIX
      value: ${namespace_name}-optimize
    - name: OPTIMIZE_ELASTICSEARCH_HOST
      value: "https://${elastic_service_url}:${elastic_service_port}"
    - name: OPTIMIZE_ELASTICSEARCH_HTTP_PORT
      value: "9200"
    
  nodeSelector:
      k8s-nodepool: ${nodepool_name}

identityKeycloak:
  enabled: false

identity:
  enabled: true

  fullURL: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/identity"
  
  contextPath: "/identity"
  
  nodeSelector:
      k8s-nodepool: ${nodepool_name}
  
  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info debug
    - name: PATH
      value: "/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    - name: JAVA_HOME
      value: "/opt/java/openjdk"
    - name: SERVER_TOMCAT_MAXHTTPRESPONSEHEADERSIZE
      value: "32KB"
    - name: SPRING_PROFILES_ACTIVE
      value: "oidc"

  command: ["java","-jar","identity.jar"]

  extraVolumes:
  - downwardAPI:
      defaultMode: 420
      items:
      - fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations
        path: annotations
    name: podinfo
   
  externalDatabase:
    enabled: true
    host: "${postgresql_service_url}"
    port: ${postgresql_service_port}
    username: "${identity_database_username}"
    database: "${identity_database}?sslmode=require"
    secret:
      existingSecret: ${namespace_name}
      existingSecretKey: identity-db-secret

identityPostgresql:
  enabled: false

  nodeSelector:
      k8s-nodepool: ${nodepool_name}

connectors:
  security:
    authentication:
      method: "oidc"
      oidc:
        clientId: ${microsoft_client_id_orchestration}
        audience: ${microsoft_client_id_orchestration}
        secret:
          # existingSecret: ${namespace_name}
          # existingSecretKey: "connectors-secret"
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        tokenScope: "${microsoft_client_id_orchestration}/.default"

  enabled: true

  contextPath: "/connectors"

  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn # info debug
    - name: OPERATE_CLIENT_AUTHURL
      value: "https://login.microsoftonline.com/${microsoft_tenant_id}/oauth2/v2.0/token"
    
  nodeSelector:
      k8s-nodepool: ${nodepool_name}

elasticsearch:
  enabled: false
  master:
    masterOnly: false
    replicaCount: 3
  data:
    replicaCount: 0
  coordinating:
    replicaCount: 0
  ingest:
    enabled: false
    replicaCount: 0
  
  nodeSelector:
      k8s-nodepool: ${nodepool_name}

Thanks for the updated configuration. I can see you’ve made the context path corrections, but there are still a couple of issues that could be causing the permission.notFound error. Looking at your configuration more closely, I notice some remaining problems:

Remaining Configuration Issues

1. Orchestration Redirect URL Template Issue

Your orchestration redirect URL has a Helm template that won’t resolve properly:

# Current - PROBLEMATIC
orchestration:
  security:
    authentication:
      oidc:
        redirectUrl: https://${namespace_name}.${service_uri}${stage_uri}${domain}{{ .Values.orchestration.contextPath }}

Fix: Since you’re using variable substitution, this should be:

orchestration:
  security:
    authentication:
      oidc:
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/"  # Direct path since contextPath is "/"

2. Global vs Local Orchestration Redirect URL Mismatch

You have conflicting redirect URLs:

# In global.identity.auth.orchestration
global:
  identity:
    auth:
      orchestration:
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/orchestration"  # ❌ Wrong path

# In orchestration section  
orchestration:
  security:
    authentication:
      oidc:
        redirectUrl: https://${namespace_name}.${service_uri}${stage_uri}${domain}{{ .Values.orchestration.contextPath }}  # ❌ Template issue

Fix: Both should point to the root path:

global:
  identity:
    auth:
      orchestration:
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/"  # ✅ Root path

orchestration:
  security:
    authentication:
      oidc:
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/"  # ✅ Root path

Potential Database Permission Issue

Based on similar reported issues, the permission.notFound error during role initialization might be caused by a database schema mismatch. This can happen when:

  1. Audience Mismatch: The permissions table contains old audience values that don’t match your current OIDC client IDs
  2. Missing Default Permissions: The database lacks the required default permissions for role initialization

Database Verification Steps

Can you check your PostgreSQL Identity database and verify:

-- Check if permissions table exists and has data
SELECT * FROM permissions LIMIT 10;

-- Check if the audience values match your client IDs
SELECT DISTINCT audience FROM permissions;

-- Check roles table
SELECT * FROM roles LIMIT 10;

-- Check role_permissions relationship
SELECT * FROM role_permissions LIMIT 10;

The audience values should match your Microsoft client IDs, not generic values like operate-api, tasklist-api, etc.

Alternative Troubleshooting Approach

Since this appears to be a specific startup issue with role initialization, try this minimal configuration to isolate the problem:

  1. Temporarily disable authorization to see if Identity starts:
orchestration:
  security:
    authorizations:
      enabled: false  # Temporarily disable
    initialization:
      # Comment out the defaultRoles section temporarily
  1. Enable debug logging for more detailed error information:
identity:
  env:
    - name: LOGGING_LEVEL_ROOT
      value: debug  # Change from warn to debug
    - name: LOGGING_LEVEL_IO_CAMUNDA_IDENTITY
      value: debug

Potential Known Issue

This specific permission.notFound error during OidcRoleServiceImpl.assignPermission might be an undocumented bug in Camunda 8.8 with Microsoft Entra ID integration.

Next Steps:

  1. Try the configuration fixes above
  2. Check your database as suggested
  3. If the issue persists, this may require filing a bug report with Camunda support

Can you try these changes and let me know:

  1. Does Identity start with authorization disabled?
  2. What do you see in the database permissions table?
  3. Any additional error details with debug logging enabled?

@Cris_Ron - this is a known issue: [EPIC] Bring Managment Identity up to Standard for 8.8 · Issue #4588 · camunda/camunda-platform-helm · GitHub

The workaround (which is in the comments of those GitHub Issues somewhere) is to add a CAMUNDA_IDENTITY_AUDIENCE environment variable to your Identity config that specifies the proper audience for your EntraID instance. There’s a bug behind the scenes where the Helm charts always setting that value to “camunda-identity-resource-server”. In your case, something like this should work:

identity:
  env:
    - name: CAMUNDA_IDENTITY_AUDIENCE
      value: ${microsoft_client_id_identity}

Hi @nathan.loding , thx for this tip, this solved the identity problem finally :slight_smile:

Now only the gateway issue persists..

@zee-bot Identity is fixed with the missing env parameter CAMUNDA_IDENTITY_AUDIENCE

Atm only the Zeebe gateway isn’t coming up and the connectors pod is throwing this error:

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed with error(-111): Connection refused: camunda01-zeebe-gateway/10.101.160.254:26500
Caused by: java.net.ConnectException: finishConnect(..) failed with error(-111): Connection refused
at io.netty.channel.unix.Errors.newConnectException0(Errors.java:166)

@Cris_Ron - what errors are you seeing in the Gateway pod? There are levels of dependencies, and Zeebe is required to be running for several other services to start, so the Connector Runtime not launching would be expected if Zeebe isn’t running.

@nathan.loding I even haven’t a dedicated pod, I tried to configure one with the documentation but nothing appears so I’m currently trying with the embbeded one but this isn’t working at all, too.

I will test with the standalone one tomorrow again, my last posted config here was with embbeded settings.

@Cris_Ron - looking through your values.yaml and I have a few clarifying questions, though I don’t think these are related:

  • why are you setting the network for the broker and gateway? Was this part of troubleshooting?
  • why are you setting the identity.command value?

When you say “I even haven’t a dedicated pod … nothing appears” - do you mean you’ve deployed your values.yaml, the gateway is never deployed?

@nathan.loding - Yes, this was part of troubleshooting and testing if something changes or if some values are mandatory for the configuration and activation of the gateway.

Regarding the command value, I copied it from a former deployment and didn’t check what it is really doing but asked a colleague to check if it is still needed.

Yes, I tried to deploy the gateway as standalone pod but the pod is not created at all.
I still get the error in the connectors pod:

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed with error(-111): Connection refused: camunda01-zeebe-gateway/10.104.181.138:26500

My current config for that looks like:

orchestration:
  contextPath: "/"
  security:
    authentication:
      method: "oidc"
      oidc:
        clientId: ${microsoft_client_id_orchestration}
        secret:
          existingSecret: ${namespace_name}
          existingSecretKey: "orchestration-secret"
        audience: ${microsoft_client_id_orchestration}
        redirectUrl: "https://${namespace_name}.${service_uri}${stage_uri}${domain}/"
        # redirectUrl: "<OC_URL>"
  
        clientIdClaim: azp
        usernameClaim: preferred_username
        groupsClaim: groups
        scope:
          - openid
          - profile
          - offline_access
          - "${microsoft_client_id_orchestration}/.default"
        preferUsernameClaim: true
    initialization:
      defaultRoles:
        admin:
          users:
            - "jkadm@rznlsw.onmicrosoft.com"
        connectors:
          clients:
            - "${microsoft_client_id_orchestration}"
    authorizations:
      enabled: true
 
  migration:
    data:
      enabled: false

  index:
    prefix: "${namespace_name}"

  enabled: true
  profiles:
    operate: true
    tasklist: true

  clusterSize: "3"
  partitionCount: "3"
  replicationFactor: "3"
  env:
    - name: LOGGING_LEVEL_ROOT
      value: warn
    - name: JAVA_TOOL_OPTIONS
      value: >-
        -Djavax.net.ssl.trustStore=/opt/certs/externaldb.jks
        -Djavax.net.ssl.trustStorePassword=${truststore_password}
        -Djavax.net.ssl.trustStoreType=JKS
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    # - name: CAMUNDA_OPERATE_ELASTICSEARCH_INDEXPREFIX
    #   value: ${namespace_name}-operate
    - name: CAMUNDA_TASKLIST_ELASTICSEARCH_URL
      value: "https://${elastic_service_url}:${elastic_service_port}"
    # - name: CAMUNDA_TASKLIST_ELASTICSEARCH_INDEXPREFIX
    #   value: ${namespace_name}-tasklist
  
  extraVolumes:
    - name: elastic-ca
      secret:
        secretName: elastic-jks         # enthält ca.crt
        defaultMode: 0444

  extraVolumeMounts:
    - name: elastic-ca
      mountPath: /opt/certs
      readOnly: true

  nodeSelector:
      k8s-nodepool: ${nodepool_name}
  
  broker:
    exporters:
      elasticsearch:
        className: io.camunda.zeebe.exporter.ElasticsearchExporter
        args:
          url: "https://${elastic_service_url}:${elastic_service_port}"
          index:
            prefix: "${namespace_name}-zeebe"
            createTemplate: true
  ingress:
    grpc:
      enabled: true
      annotations:
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        cert-manager.io/cluster-issuer: ${clusterissuer_name}
        traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
        traefik.ingress.kubernetes.io/router.tls: "true"
        acme.cert-manager.io/http01-edit-in-place: "false"
        acme.cert-manager.io/http01-ingress-class-name: ${ingress_controller}
        cert-manager.io/common-name: "${namespace_name}-orchestration.${service_uri}${stage_uri}${domain}"
        cert-manager.io/duration: "720h"
        cert-manager.io/renew-before: "24h"
        cert-manager.io/subject-organizations: ${subject_organizations}
        cert-manager.io/subject-organizationalunits: ${subject_organizationalunits}
        cert-manager.io/subject-localities: ${subject_localities}
        cert-manager.io/subject-provinces: ${subject_provinces}
        cert-manager.io/subject-countries: ${subject_countries}
        cert-manager.io/private-key-size: "4096"
      host: "${namespace_name}-orchestration.${service_uri}${stage_uri}${domain}"
      tls:
        enabled: true
        secretName: "${release_name}-orchestration-tls"

zeebe:
  broker:
    gateway:
      enable: false
  gateway:
    network:
      port: 26500
    security:
      enabled: false
      authentication:
        mode: none

Hi @Cris_Ron - I looked over your config again, with the help from one of our engineers, and there’s a few potential issues we found.

Primarily - and I didn’t notice this when I looked before - the zeebe section at the bottom isn’t actually valid in the Helm config: those should be part of the application.yaml. I believe the correct config here is:

zeebe:
  configuration: |-
    zeebe:
      broker:
        gateway:
          enable: false
      gateway:
        network:
          port: 26500
        security:
          enabled: false
          authentication:
            mode: none

This is also true of the broker.exporters configuration, I think - that should be in the application.yaml or configured with environment variables. However, the documentation page for the Elasticsearch configuration does not make this clear, so I’ve opened a ticket with the documentation team to improve that.

The engineer also said that they wouldn’t expect to see a specific gateway pod; rather, with 8.8 and the orchestration cluster, you should see at least one Zeebe pod (RELEASE-zeebe-0 is the naming convention).

Can you share a couple screenshots of what pods you are seeing? And if you see any Zeebe pods, what do the logs and events show?

The engineer also wasn’t sure if the Traefik TLS config for gRPC would work, this isn’t something that we’ve tested. The communication inside the cluster isn’t going over that TLS ingress, so perhaps it may not be an issue, but that configuration was a question mark for them.