Tasklist Fails to Connect to Zeebe – Using OpenSearch & Aurora PostgreSQL

Camunda 8 Topics
1

Hi Camunda team and community,

I’m deploying Camunda Platform Helm chart via ArgoCD and am trying to integrate with AWS OpenSearch and Aurora PostgreSQL (both external). My goal is to disable the built-in Elasticsearch and PostgreSQL and instead use the managed services.

Here is a simplified snippet of my values.yaml wrapped in an ArgoCD Application:

global:
  elasticsearch:
    enabled: false
  opensearch:
    enabled: true
    distribution: opensearch
    version: 2.5
    auth:
      username: "admin"
      password: "<redacted>"
    url:
      protocol: https
      host: <redacted-opensearch-host>
      port: 443
    aws:
      enabled: true
      region: us-east-2
      serviceName: es

  identity:
    auth:
      publicIssuerUrl: "https://camunda.dev.mycompany.io/auth/realms/camunda-platform"
      operate:
        redirectUrl: "https://camunda.dev.mycompany.io/operate"
      tasklist:
        redirectUrl: "https://camunda.dev.mycompany.io/tasklist"
      optimize:
        redirectUrl: "https://camunda.dev.mycompany.io/optimize"
      webModeler:
        redirectUrl: "https://camunda.dev.mycompany.io/modeler"
      console:
        redirectUrl: "https://camunda.dev.mycompany.io/console"

elasticsearch:
  enabled: false

postgresql:
  enabled: false

identityKeycloak:
  postgresql:
    enabled: false
  externalDatabase:
    host: "<redacted-aurora-host>"
    port: 5432
    user: "postgres"
    database: "bitnami_keycloak"
    password: "<redacted>"

webModeler:
  postgresql:
    enabled: false
  restapi:
    enabled: true
    externalDatabase:
      host: "<redacted-aurora-host>"
      port: 5432
      user: "postgres"
      database: "bitnami_keycloak"
      password: "<redacted>"

Most services seem to be up and working—Zeebe appears healthy, and I’m able to access the other UIs.

However, Tasklist fails to connect to Zeebe, and I’m getting repeated errors like this:

io.camunda.tasklist.zeebe.PartitionHolder - Error occurred when requesting partition ids from Zeebe client: null
Caused by: io.grpc.StatusRuntimeException: CANCELLED
...
Caused by: java.io.IOException: Failed while requesting access token with status code 401 and message Unauthorized.
...
Partition ids can't be fetched from Zeebe. Try next round (45).

Thanks in advance for any tips or experience with similar setups!

2

I’m not an expert in this, but this error looks like it’s an issue between identity and tasklist, rather than with Aurora. 401 means that the Token being used to connect isn’t allowed to get that information.

I’m not in the deployment side of things enough to help (I’m more modeling side), but if there’s traffic on a thread, more people tend to look at it.

1 Like
3

@artem Can you try restarting the tasklist pods alone and see if the issue persists?

4

yes I tried that, didn’t help.

5

@artem to scope down the error. Try to run keyclaok using the built-in internal database, and see if the error persists.

6

@Hamza_Masood I didn’t have issues with default postgres and elasticsearch running in k8s.

7

The issues have been resolved. It turns out that not only the identityKeycloak section needed to connect to PostgreSQL, but the identity section as well.

Also, I struggled using IAM alone to access OpenSearch, went for password based authentication and had no issues.

In any case, here are the Helm(chart v11.3.0) values that worked for me:

global:
  secrets:
    autoGenerated: false
  elasticsearch:
    enabled: false
    external: false
  opensearch:
    enabled: true
    distribution: opensearch
    version: 2.17
    auth:
      username: "admin"
      existingSecret: "camunda-camunda-platform-opensearch"
      existingSecretPasswordKey: "password"
    url:
      protocol: https
      host: <opensearch-endpoint>
      port: 443
    aws:
      enabled: false
  identity:
    auth:
      publicIssuerUrl: "https://camunda.mycompany.com/auth/realms/camunda-platform"
      operate:
        redirectUrl: "https://camunda.mycompany.com/operate"
      tasklist:
        redirectUrl: "https://camunda.mycompany.com/tasklist"
      optimize:
        redirectUrl: "https://camunda.mycompany.com/optimize"
      webModeler:
        redirectUrl: "https://camunda.mycompany.com/modeler"
      console:
        redirectUrl: "https://camunda.mycompany.com/console"

elasticsearch:
  enabled: false

console:
  image:
    repository: camunda/console
    tag: 8.6.55
  contextPath: "/console"
  enabled: true
  configuration: |
    camunda:
      console:
        oAuth:
          audience: "console-api"
          clientId: "console"
          issuer: "https://camunda.mycompany.com/auth/realms/camunda-platform"
          jwksUri: "http://camunda-keycloak:80/auth/realms/camunda-platform/protocol/openid-connect/certs"
          type: "KEYCLOAK"
          wellKnown: "https://well-known-uri"
        managed:
          method: plain
          releases:
          - name: camunda
            namespace: camunda
            version: 11.3.0
            components:
            - name: Console
              id: console
              version: 8.6.55
              url: https://camunda.mycompany.com/console
              readiness: http://camunda-console.camunda:9100/health/readiness
              metrics: http://camunda-console.camunda:9100/prometheus
            - name: Keycloak
              id: keycloak
              version: 25.0.6
              url: https://camunda.mycompany.com/auth
            - name: Identity
              id: identity
              version: 8.6.8
              url: https://camunda.mycompany.com/identity
              readiness: http://camunda-identity.camunda:82/actuator/health
              metrics: http://camunda-identity.camunda:82/actuator/prometheus
            - name: Operate
              id: operate
              version: 8.6.9
              url: https://camunda.mycompany.com/operate
              readiness: http://camunda-operate.camunda:9600/operate/actuator/health/readiness
              metrics: http://camunda-operate.camunda:9600/operate/actuator/prometheus
            - name: Optimize
              id: optimize
              version: 8.6.5
              url: https://camunda.mycompany.com/optimize
              readiness: http://camunda-optimize.camunda:80/optimize/api/readyz
              metrics: http://camunda-optimize.camunda:8092/actuator/prometheus
            - name: Tasklist
              id: tasklist
              version: 8.6.9
              url: https://camunda.mycompany.com/tasklist
              readiness: http://camunda-tasklist.camunda:9600/tasklist/actuator/health/readiness
              metrics: http://camunda-tasklist.camunda:9600/tasklist/actuator/prometheus
            - name: WebModeler WebApp
              id: webModelerWebApp
              version: 8.6.7
              url: https://camunda.mycompany.com/modeler
              readiness: http://camunda-web-modeler-webapp.camunda:8071/health/readiness
              metrics: http://camunda-web-modeler-webapp.camunda:8071/metrics
            - name: Zeebe Gateway
              id: zeebeGateway
              version: 8.6.9
              urls:
                grpc: http://camunda-zeebe-gateway.camunda:26500
                http: https://camunda.mycompany.com/zeebe
              readiness: http://camunda-zeebe-gateway.camunda:9600/zeebe/actuator/health/readiness
              metrics: http://camunda-zeebe-gateway.camunda:9600/zeebe/actuator/prometheus
            - name: Zeebe
              id: zeebe
              version: 8.6.9
              readiness: http://camunda-zeebe.camunda:9600/actuator/health/readiness
              metrics: http://camunda-zeebe.camunda:9600/actuator/prometheus

connectors:
  image:
    repository: camunda/connectors-bundle
    tag: 8.6.8
  contextPath: "/connectors"
  logging:
    level:
      io.camunda.connector: DEBUG

operate:
  image:
    repository: camunda/operate
    tag: 8.6.9
  contextPath: "/operate"

optimize:
  image:
    repository: camunda/optimize
    tag: 8.6.5
  contextPath: "/optimize"

tasklist:
  image:
    repository: camunda/tasklist
    tag: 8.6.9
  contextPath: "/tasklist"

webModeler:
  image:
    tag: 8.6.7
  enabled: true
  restapi:
    mail:
      fromAddress: noreply@camunda.mycompany.com
      fromName: Camunda 8 WebModeler
    externalDatabase:
      url: "jdbc:postgresql://<db-endpoint>:5432/modeler"
      user: "postgres"
      existingSecret: "camunda-web-modeler-restapi"
      existingSecretKey: "password"

zeebe:
  image:
    repository: camunda/zeebe
    tag: 8.6.9

zeebeGateway:
  image:
    repository: camunda/zeebe
    tag: 8.6.9
  contextPath: "/zeebe"

identity:
  contextPath: "/identity"
  image:
    repository: camunda/identity
    tag: 8.6.8
  fullURL: "https://camunda.mycompany.com/identity"
  externalDatabase:
    enabled: true
    host: "<db-endpoint>"
    port: 5432
    username: "postgres"
    existingSecret: "camunda-identity-postgresql"
    existingSecretKey: "password"
    database: "keycloak"

identityKeycloak:
  auth:
    adminUser: admin
    existingSecret: "camunda-keycloak"
    existingSecretPasswordKey: "admin-password"
  image:
    repository: bitnami/keycloak
    tag: 25.0.6
  postgresql:
    enabled: false
    image:
      repository: bitnami/postgresql
      tag: 15.10.0-debian-12-r2
  externalDatabase:
    host: "<db-endpoint>"
    port: 5432
    user: "postgres"
    existingSecret: "camunda-externaldb"
    existingSecretKey: "password"
    database: "keycloak"
  extraEnvVars:
    - name: KEYCLOAK_PROXY_ADDRESS_FORWARDING
      value: "true"

Here’s what’s now backed by AWS-managed services:
• Keycloak , Camunda Identity , and Web Modeler are now using AWS Aurora PostgreSQL
• Optimize , Operate , and Tasklist are now running on AWS OpenSearch

Pro tip: Every time you upgrade using Helm, it tends to regenerate Secrets by default, which can break your setup, especially for components like Keycloak that rely on stable credentials. The best approach is to ensure your Secrets remain consistent across upgrades.

A solid strategy is to use ArgoCD together with the External Secrets Operator. This allows you to manage Secrets externally (e.g., in AWS Secrets Manager or Vault) and sync them into Kubernetes without Helm touching them.

Additionally, when setting up your ArgoCD Application, you can configure it to ignore changes to Secrets managed outside of Helm. This prevents Helm upgrades from overwriting them.

If you let Helm regenerate Secrets every time, you’d end up needing to delete and recreate the Keycloak database to align with the new credentials, which is definitely not ideal.

1 Like
8

@artem
I have created an issue for a guide to be created in our docs for connecting to external psql: [ENHANCEMENT] Create a guide for external PostgreSQL · Issue #3323 · camunda/camunda-platform-helm · GitHub

Please give some more feedback on how we can improve our docs.

2 Likes
9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.