Hi @Maurice_Tchangue ,
The key thing first: your assumption doesn’t match the 8.9 architecture
In Camunda 8.9, Operate and Tasklist are no longer standalone services that authenticate to Zeebe with an OIDC token. They are part of the single consolidated Orchestration Cluster application (StandaloneCamunda) together with the Zeebe broker/gateway and Admin. They run in the same JVM/pod and share the same configuration — there is no machine-to-machine token exchange between Operate/Tasklist and Zeebe that could be “rejected by Zeebe.”
That older model (separate Operate/Tasklist doing M2M to a zeebe/zeebe-api audience) was 8.7 and earlier. So the theory “Console isn’t in the audiences → its token is rejected by Zeebe → Operate/Tasklist unhealthy” does not hold in 8.9:
- Console is a Management component and has its own audience (e.g.
console-api). It does not need to be in the Orchestration Cluster’s audiences list, and adding it there would not change Operate/Tasklist health.
- Health/readiness actuator endpoints are not behind OIDC. The
9600/actuator/health/* endpoints the platform (and Console) use to determine health are unauthenticated by default, so an audience mismatch cannot make them report “unhealthy.”
Your audiences block controls validation of tokens presented to the Orchestration Cluster REST API / UI — nothing more.
What actually determines Operate/Tasklist health in 8.9
Because they’re part of one app, “Operate/Tasklist unhealthy” is reported through the Orchestration Cluster’s Spring Boot Actuator on port 9600. The health indicators that matter are:
"searchEngineCheck": { "status": "UP" },
"indicesCheck": { "status": "UP" },
"brokerReady": { "status": "UP" }
readinessProbe → /actuator/health/readiness — tied to secondary storage (Elasticsearch/OpenSearch) and index availability.livenessProbe → /actuator/health/liveness — in 8.9 this was deliberately decoupled from Elasticsearch (the default livenessProbe.probePath changed from /actuator/health/readiness to /actuator/health/liveness) so ES blips don’t restart the pod.
So in an overwhelming majority of cases, Operate/Tasklist show “unhealthy/unready” because the Orchestration Cluster cannot reach or initialize its Elasticsearch/OpenSearch indices, not because of OIDC.
Two most likely real root causes (in order)
1. Secondary storage (Elasticsearch/OpenSearch) is unreachable from the orchestration namespace
Operate/Tasklist readiness fails (searchEngineCheck/indicesCheck = DOWN → HTTP 503) when the search engine URL, credentials, TLS, or cross-namespace network policy is wrong. Since you split into two namespaces, this is the prime suspect.
Diagnose — hit the health endpoint directly inside the orchestration pod:
kubectl -n orchestration exec -it <orchestration-pod> -- \
curl -s localhost:9600/actuator/health | jq
Look for which sub-indicator is DOWN. If it’s searchEngineCheck/indicesCheck, it’s storage, not auth. Then check the logs:
kubectl -n orchestration logs <orchestration-pod> | grep -iE "elasticsearch|opensearch|readiness|index|connection"
2. Console’s monitoring is pointed at wrong/unreachable readiness URLs
If health is being read by Console’s Cluster Status Dashboard in the management namespace, it reaches the orchestration namespace via explicit readiness URLs. In 8.9 the correct URLs (note the /operate and /tasklist context paths on port 9600) are:
- name: Operate
id: operate
readiness: http://camunda-zeebe.orchestration:9600/operate/actuator/health/readiness
metrics: http://camunda-zeebe.orchestration:9600/operate/actuator/prometheus
- name: Tasklist
id: tasklist
readiness: http://camunda-zeebe.orchestration:9600/tasklist/actuator/health/readiness
metrics: http://camunda-zeebe.orchestration:9600/tasklist/actuator/prometheus
- name: Orchestration Cluster
id: orchestration
readiness: http://camunda-zeebe.orchestration:9600/actuator/health/readiness
If those URLs are wrong, or a NetworkPolicy blocks management → orchestration traffic on 9600, Console will mark Operate/Tasklist unhealthy while everything in the management namespace (Identity, Modeler, Console) plus Optimize shows healthy — which is exactly your symptom pattern. Verify cross-namespace reachability:
kubectl -n management exec -it <console-pod> -- \
curl -s http://camunda-zeebe.orchestration:9600/operate/actuator/health/readiness
About your OIDC audiences block
Your snippet is a reasonable Orchestration Cluster OIDC config, but a few notes:
security:
authentication:
oidc:
username-claim: "preferred_username"
client-id-claim: "client_id"
client-id: "orchestration"
client-secret: ${VALUES_ORCHESTRATION_CLIENT_SECRET:}
audiences:
- "orchestration"
- "orchestration-api"
- "web-modeler-api" # keep this so Web Modeler can deploy to the cluster
Recommended action plan
- Curl
localhost:9600/actuator/health inside the orchestration pod and identify the failing sub-indicator. This immediately tells you storage-vs-something-else.
- If
searchEngineCheck/indicesCheck is DOWN → fix Elasticsearch/OpenSearch connectivity (URL, credentials, TLS, cross-namespace NetworkPolicy). This is the most probable cause.
- If the pod’s own health is UP but Console shows red → fix Console’s readiness URLs / cross-namespace networking (use the
/operate and /tasklist context-path URLs above).
- Only if logs actually show “Invalid token / Audience mismatch” → adjust
audiences (and possibly client-id-claim) to match the real aud/claims in your Keycloak tokens. Do not add Console to the orchestration audiences.
Bottom line: In 8.9 Operate/Tasklist are internal to the single Orchestration Cluster app, so their “unhealthy” status is driven by the cluster’s readiness (almost always Elasticsearch/OpenSearch reachability) or by Console’s health-URL/network configuration across your two namespaces — not by an OIDC audience list that excludes Console.
Hints: Use the Ask AI feature in 