C8 Identity 'Connect to an existing Keycloak Instance' does not work

[buildwithjz]

Hi,

Been testing out the ‘Connect to an existing Keycloak Instance’ documentation (Connect to an existing Keycloak instance | Camunda Platform 8 Docs) provided as part of Camunda 8.1. I created a standalone C8 identity service in Kubernetes using camunda/identity:8.1.2 to connect to my organisation’s existing keycloak instance.

As we were using keycloak v19, the UI for configuring the camunda-identity client is slightly different, however I believe it is identical to what is described in the doc.

As described in the document, I have set the following environment variables, keeping in mind that the realm path for keycloak 19 is slightly different:

  - name: KEYCLOAK_URL
    value: https://[KEYCLOAK_HOST]
  - name: IDENTITY_AUTH_PROVIDER_ISSUER_URL
    value: https://[KEYCLOAK_HOST]/realms/camunda-platform
  - name: IDENTITY_CLIENT_ID
    value: camunda-identity
  - name: IDENTITY_CLIENT_SECRET
    value: .......
  - name: KEYCLOAK_REALM
    value: camunda-platform

Note that this step of the documentation was followed:

However, I was getting a 401 error.

$ k logs -n dev c8-identity-56c48c9d5-gqwc4 --previous
...
2022-10-19 01:00:49.450 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : Failure #6. Unable to connect to Keycloak.
2022-10-19 01:00:49.450 ERROR 1 --- [           main] i.c.i.i.k.c.KeycloakConfiguration        : javax.ws.rs.NotAuthorizedException: HTTP 401 Unauthorized

Despite the manual verification of the client working as expected:

$ curl -d "client_id=camunda-identity" -d "client_secret=......" -d "grant_type=client_credentials" "https://[KEYCLOAK_HOST]/realms/camunda-platform/protocol/openid-connect/token"
{"access_token":"...","expires_in":300,"refresh_expires_in":0,"token_type":"Bearer","not-before-policy":0,"scope":"email profile"}

Checking the logs for keycloak itself, it looks like C8 Identity is still attempting to log into using the master realm using the KEYCLOAK_SETUP_* environment variables (i.e. new keycloak instance and realm), and the password grant type, rather than set up keycloak through my existing instance/realm.

2022-10-19 00:47:41,688 WARN  [org.keycloak.events] (executor-thread-161) type=LOGIN_ERROR, realmId=master, clientId=admin-cli, userId=null, ipAddress=10.xx.xx.xx, error=invalid_client_credentials, grant_type=password

Is there something I’m doing incorrectly here? Would really like to get C8 identity connected to an existing keycloak instance/realm.

[jonathan.lukas]

Hello @buildwithjz ,

from my understanding, you will need to set KEYCLOAK_SETUP_REALM to camunda-platform in the identity config.

Identity will access your keycloak at two points:

1. In the setup realm, there has to be a service account with sufficient rights to manage the KEYCLOAK_REALM through the API.
2. In the keycloak realm, the actual IAM is done.

I hope this helps

Jonathan

[buildwithjz]

Hey @jonathan.lukas

Thanks for your input on this issue.

It certainly makes sense to set KEYCLOAK_SETUP_REALM as camunda-platform, however I do still get the LOGIN_ERROR exception on keycloak’s side during the setup realm phase:

2022-10-19 20:10:56,590 WARN  [org.keycloak.events] (executor-thread-182) type=LOGIN_ERROR, realmId=7718e61b-7f78-4657-8af2-49559970b8b5, clientId=admin-cli, userId=null, ipAddress=10.xx.xx.xx, error=user_not_found, auth_method=openid-connect, grant_type=password, client_auth_method=client-secret, username=admin, authSessionParentId=xx, authSessionTabId=xx

This is telling me that C8 Identity is still trying to log into keycloak using admin credentials (similar to if I were to deploy C8 Identity using Helm with auth enabled), however now that its pointing to the camunda-platform realm, the admin user doesn’t exist and is throwing the above error.

Furthering my investigation, in regards to your first point:

"In the setup realm, there has to be a service account with sufficient rights to manage the KEYCLOAK_REALM through the API."

I set the environment variable

- name: KEYCLOAK_SETUP_USER
  value: service-account-camunda-identity

As per the service account username in an attempt to resolve the user_not_found error:

I was then met with an HTTP 500 response from C8 Identity in trying the initial login with keycloak, hence I don’t believe I was meant to set that.

My assumption was that by setting the IDENTITY_CLIENT_SECRET environment variable, keycloak switches to the client secret login flow, where only a client ID and secret are required to log in, however it looks like its defaulting to the admin user/password/clientid flow.

Please let me know if my assumptions are incorrect, or whether this is a bug with Camunda?

[ben.sheppard]

Hi @buildwithjz!

These issues are certainly odd, can I also confirm that when you are setting the environment variables, are you also setting IDENTITY_AUTH_PROVIDER_BACKEND_URL as well? The reason for this is that the path change for Keycloak 19 would also need to be reflected there too (the same value you have used for IDENTITY_AUTH_PROVIDER_ISSUER_URL should be sufficient).

The secondary issue you have found as well of the system still using KEYCLOAK_SETUP_* variables is unexpected in this scenario so I will raise an issue on our team board to look into this more.

Look forward to hearing from you,

Ben.

[buildwithjz]

Hi @ben.sheppard ,

Be keen to hear what you guys find.

I actually did not set IDENTITY_AUTH_PROVIDER_BACKEND_URL at all as I’d assumed this variable was used to support “container-to-container” communication (Configuration variables | Camunda Platform 8 Docs) when setting up keycloak using admin credentials.

Another reason for its omission was because we are using our own keycloak server, and we cannot assume that its running in the same cluster as the identity service.

I will give this another go and set IDENTITY_AUTH_PROVIDER_BACKEND_URL just to be sure.

Thanks,

EDIT:

Setting IDENTITY_AUTH_PROVIDER_BACKEND_URL had no effect. Keycloak reported the following logs:

2022-11-04 03:50:50,860 WARN  [org.keycloak.events] (executor-thread-418) type=LOGIN_ERROR, realmId=***, clientId=admin-cli, userId=null, ipAddress=***, error=user_not_found, auth_method=openid-connect, grant_type=password, client_auth_method=client-secret, username=admin, authSessionParentId=***, authSessionTabId=***

Sounds like a bug to me