Identity unable to connect to Keycloak (HTTP 403)

vincentgiraud · November 29, 2023, 11:28am

Hello,

Here are the details:

1. What did you expect to happen? What did happen?

Expected: Identity to connect to Keycloak in order to create Camunda realm, firts “demo” user and apps (Operate and Zeebe) setup.
Happened: Identity unable to connect to Keycloak. Forbidden (HTTP 403)

I’ve verified the value of the admin password from the Keycloak secret and both the env vars for admin password in Identity (KEYCLOAK_SETUP_PASSWORD) and Keycloak (KEYCLOAK_ADMIN_PASSWORD) pods are the same.
I’ve set the “Require SSL” = None in the Keycloak master realm settings and restarted the Identity pod.

2. YAML configuration for Helm.

Here is an extract of my values.yaml file.

global:
  identity:
    auth:
      enabled: true        
      publicIssuerUrl: https://my-keycloak-service-fullname.domain.tld/auth/realms/camunda-platform
      operate:
        redirectUrl: https://my-operate-service-fullname.domain.tld
…
identity:
  enabled: true
  # Note: doesn't seem effective
  logging:
    level:
      ROOT: DEBUG
# resources downgrade due to test namespace quotas restrictions
  resources: 
    requests:
      cpu: "200m"
      memory: "250Mi"
    limits:          
      cpu: "200m"
      memory: "250Mi" 
# specific user due to security restrictions
  podSecurityContext:
    fsGroup: 1xxxxxxxxxx
  containerSecurityContext:
    runAsUser: 1xxxxxxxxxx
  # https://forum.camunda.io/t/identity-unable-to-connect-to-keycloak/42419
  # Note: log level settings bring little additional information on the issue
  env:
    - name: IDENTITY_LOG_LEVEL
      value: "ALL"
    - name: IDENTITY_URL
      value: "https://my-identity-service-fullname.domain.tld" 
  keycloak:
    image:
      registry: "my-private-registry.domain.tld"
      pullPolicy: Always
      pullSecrets: 
        - my-secret
    url: "http://my-keycloak-service-fullname:80/auth"
    # https://forum.camunda.io/t/identity-unable-to-connect-to-keycloak/42419
    extraEnvVars:
      - name: KEYCLOAK_PROXY_ADDRESS_FORWARDING
        value: "true"    
      - name: KEYCLOAK_FRONTEND_URL
        value: "https://my-keycloak-service-fullname.domain.tld"
      - name: BITNAMI_DEBUG
        value: "true"          
    resources:
      requests:
        cpu: 125m
        memory: 200Mi
      limits:
        cpu: 300m
        memory: 400Mi
    podSecurityContext:
      fsGroup: 1xxxxxxxxxx
    containerSecurityContext:
      runAsUser: 1xxxxxxxxxx
…

All other values are inherited from camunda-platform/values.yaml

IDENTITY_AUTH_PROVIDER_BACKEND_URL = "https://my-keycloak-service-fullname:80/auth/realms/camunda-platform"
IDENTITY_AUTH_PROVIDER_ISSUER_URL = "https://my-keycloak-service-fullname.domain.tld/auth/realms/camunda-platform"
KEYCLOAK_URL= "https://my-keycloak-service-fullname:80/auth"

What environment are you running C8 on?
Own servers and managed Open Shift cluster v4.12.

3. What version are you running?

V8.3.2 on a self-managed (no Optimize yet, no Tasklist), custom Open Shift routes (Keycloak, Identity, Operate, Zeebe gateway) instead of ingresses.
All pods are running but Identity (and Operate which makes sense I guess since it cannot authenticate to Zeebe without credentials)

4. Error logs or stack traces.

2023-11-29 10:18:03.101  INFO 1 --- [           main] i.c.i.Application                        : Started Application in xx.xxx seconds (process running for xx.xxx)
2023-11-29 10:18:05.300 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Failure #1. Unable to connect to Keycloak.
2023-11-29 10:18:35.300  WARN 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Retrying...
2023-11-29 10:18:35.308 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Failure #2. Unable to connect to Keycloak.
2023-11-29 10:19:05.309  WARN 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Retrying...
2023-11-29 10:19:05.316 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Failure #3. Unable to connect to Keycloak.
2023-11-29 10:19:35.316  WARN 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Retrying...
2023-11-29 10:19:35.326 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Failure #4. Unable to connect to Keycloak.
2023-11-29 10:20:05.327  WARN 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Retrying...
2023-11-29 10:20:05.337 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Failure #5. Unable to connect to Keycloak.
2023-11-29 10:20:35.337  WARN 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Retrying...
2023-11-29 10:20:35.345 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Failure #6. Unable to connect to Keycloak.
2023-11-29 10:20:35.345 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : jakarta.ws.rs.ForbiddenException: HTTP 403 Forbidden

Thanks for your help!

vincentgiraud · November 30, 2023, 11:13am

UPDATE 1: I made progress thanks to this topic on manually disabling SSL in Keycloak. We don’t use the whitelisted IP ranges by Keycloak so SSL is required.

Under “master” realm, setting “Require SSL” to “None” will allow creation of “camunda-platform” realm.
Once created, under “camunda-platform” realm, setting “Require SSL” to “None” will allow creation of roles and “demo” user and Identity to eventually start.

Remaining issues:

Manually disabling SSL is just a workaround but I’m yet to find a long term solution for production.
Logging out in Operate or Identity won’t work.
Logging in will redirect to http only: (e.g. my-operate-fullname-service.domain.tld/api/login) even though my Open Shift route specifies “insecureEdgeTerminationPolicy: Redirect”
When connection token expires Identity will no longer be accessible and logs will show the following error: io.camunda.identity.sdk.impl.rest.exception.RestException: request failed with status code '400' and body '{"error":"invalid_grant","error_description":"Session not active"}'