Troubleshooting OAuth Issue with zbctl Status Call in Camunda Self-Managed Setup

Hi there,

I’ve been following the excellent example outlined in this blog post and have progressed to the “What’s Next?” section to carry out “Add Identity and Optimize, configure the ingress, and test the authentication with zbctl”.

Unfortunately, I encountered an issue during the last step when calling zbctl status:

$ zbctl status --certPath cert-zeebe.pem --address zeebe.camunda.local:443 --authzUrl https://camunda.local/auth/realms/camunda-platform --clientId [Client ID] --clientSecret [Client Secret]

returns =>

Error: rpc error: code = Canceled desc = failed to apply token: failed to obtain access token: oauth2: "RESTEASY003650: No resource method found for POST, return 405 with Allow header"

Here’s a shorter description of what’s been done if you would like to reconstruct the error …

Setup KIND cluster

kind create cluster --name camunda-cluster --config kind.config

Install NGINX ingress

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml

Create certificates

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -sha256 -days 365 --nodes -addext 'subjectAltName=DNS:camunda.local'

openssl req -x509 -newkey rsa:4096 -keyout key-zeebe.pem -out cert-zeebe.pem -sha256 -days 365 --nodes -addext 'subjectAltName=DNS:zeebe.camunda.local'

Create secrets

kubectl create secret tls tls-secret --cert=cert.pem --key=key.pem

kubectl create secret tls tls-secret-zeebe --cert=cert-zeebe.pem --key=key-zeebe.pem

Check pod status

k9s

Install Camunda

helm install camunda-platform camunda/camunda-platform -f helm-identity-values.yaml

Check Camunda Identity

https://camunda.local/identity

Log in using the demo/demo credentials.

Lookup API Client

Use zeebe client id and its corresponding client secret.

Check Camunda Zeebe status

zbctl status --certPath cert-zeebe.pem --address zeebe.camunda.local:443 --authzUrl https://camunda.local/auth/realms/camunda-platform --clientId [Client ID] --clientSecret [Client Secret]

helm-identity-values.yaml:

global:
  ingress:
    enabled: true
    className: nginx
    host: "camunda.local"
    tls:
      enabled: true
      secretName: "tls-secret"
  identity:
    auth:
      publicIssuerUrl: "https://camunda.local/auth/realms/camunda-platform"
      operate:
        redirectUrl: "https://camunda.local/operate"
      tasklist:
        redirectUrl: "https://camunda.local/tasklist"        
identity:
  contextPath: "/identity"
  fullURL: "https://camunda.local/identity"

# Disable Optimize
optimize:
  enabled: false

operate:
  contextPath: "/operate"

tasklist:
  contextPath: "/tasklist"

# Reduce resource usage for Zeebe and Zeebe-Gateway
zeebe:
  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1
  pvcSize: 10Gi
  resources: {}
  initResources: {}

zeebe-gateway:
  replicas: 1
  ingress:
    enabled: true
    className: nginx
    host: "zeebe.camunda.local"
    tls:
      enabled: true
      secretName: "tls-secret-zeebe"

# Enable Outbound Connectors only
connectors:
  enabled: true
  inbound:
    mode: "disabled"

# Configure Elasticsearch to make it running for local development
elasticsearch:
  resources: {}
  initResources: {}
  replicas: 1
  minimumMasterNodes: 1
  # Allow no backup for single node setups
  clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"

  # Request smaller persistent volumes.
  volumeClaimTemplate:
    accessModes: [ "ReadWriteOnce" ]
    storageClassName: "standard"
    resources:
      requests:
        storage: 15Gi

kind.config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
  - containerPort: 443
    hostPort: 443
  - containerPort: 26500
    hostPort: 26500

Hi,
I think your authzUrl is incomplete, try:
–authzUrl https://camunda.local/auth/realms/camunda-platform/protocol/openid-connect/token

Hope it should work for you.

On my side, I’m stuck with
Error: rpc error: code = Canceled desc = failed to apply token: failed to obtain access token: oauth2: “invalid_scope” "Invalid scopes: "

And in Keycloak logs I have
2024-03-28T07:38:06.412572518Z 2024-03-28 07:38:06,411 WARN [org.keycloak.events] (executor-thread-87) type=CLIENT_LOGIN_ERROR, realmId=camunda-platform, clientId=zeebe-api, userId=e6d2---------804, ipAddress=–.–.–.–, error=invalid_request, grant_type=client_credentials, client_auth_method=client-secret, username=service-account-zeebe-api

Hi,
Changing to that token endpoint just gave me the same issue you’re facing :slight_smile: have you found a solution to it? @Gerald

btw. I see nothing in my kc log though

Hi,
Unfortunatly not. I cannot succeed to use zbctl

And using the 8.5.0-RC2 I succeed to connect to zeebe gateway using Java client (by creating a new application using the wabapp Identity (via port forward, because it is failing via ingress (we have an haproxy in front of)

But now I upgraded to 8.5.0 and it is worst. Still same issue but even the port forward does not work. I soon as I send a command to it, the connection is closed without any logs anywhere…

Have you come across any GitHub issues or other forum discussions pertaining to this issue?

not really

Hello everyone, I had the same issue as you with zbctl in 8.4.0.
Prefixing the command with ZEEBE_TOKEN_SCOPE=“camunda-identity” makes it work;

ZEEBE_TOKEN_SCOPE="camunda-identity" zbctl status --certPath cert-zeebe.pem --address zeebe.camunda.local:443 --authzUrl https://camunda.local/auth/realms/camunda-platform --clientId [Client ID] --clientSecret [Client Secret]

Finally got it working :slight_smile:

$ export ZEEBE_CLIENT_ID=zeebe
$ export ZEEBE_CLIENT_SECRET=MY-SECRET
$ export ZEEBE_TOKEN_AUDIENCE=zeebe-api
$ export ZEEBE_TOKEN_SCOPE=camunda-identity
$ export ZEEBE_AUTHORIZATION_SERVER_URL=https://camunda.local/auth/realms/camunda-platform/protocol/openid-connect/token

$ zbctl status --certPath cert-zeebe.pem --address zeebe.camunda.local:443

Cluster size: 1
Partitions count: 1
Replication factor: 1
Gateway version: 8.4.4
Brokers:
  Broker 0 - camunda-platform-zeebe-0.camunda-platform-zeebe.default.svc:26501
    Version: 8.4.4
    Partition 1 : Leader, Healthy
1 Like