Configure Identity TLS to connect to TLS enabled Keycloak

Camunda 8 Topics Discussion & Questions
1

I have a cluster with a self-managed deployment of Camunda 8 (I am using the Helm chart). Every communication must be encrypted (external and internal) and its mandatory to use Istio service mesh.

Said so, my current setup consists of an istio ingress gateway which terminates TLS for external inbound traffic and my idea is to use a single pair of truststore and keystore based on a self-signed wildcard certificate that will be used for every Camunda component just for internal communications (correct me if this is not possible).

I have enabled TLS in Keycloak by creating the appropriate secrets and using this configuration while deploying the chart:

identity:
  keycloak:
    tls:
      enabled: true
      existingSecret: "keycloakstores"
      truststoreFilename: "cacerts"
      keystoreFilename: "keystore.jks"
      passwordsSecret: "sslstorepass"
    spi:
      existingSecret: "keycloakstores"
      truststoreFilename: "cacerts"
      passwordsSecret: "sslstorepass"
      hostnameVerificationPolicy: "ANY"
    production: true
    proxy: passthrough
    containerSecurityContext:
      enabled: false

With this configuration, I can see the logs of Keycloak displaying that TLS and SPI are correctly configured and I am able to access Keycloak from outside the cluster using the ingress (configuring a destinationrule with enabled TLS to the Camunda namespace).

But as soon as I deploy Keycloak with this configuration, other components stop working. I guess this is normal since I have not configured the trustedstore and keystore, so I went and configured for example Identity to use the same trustedstore and keystore:

identity:
  env:
    - name: SERVER_SSL_ENABLED
      value: "true"
    - name: SERVER_SSL_KEYSTORE
      value: /etc/keystore/keystore.jks
    - name: SERVER_SSL_KEYSTOREPASSWORD
      valueFrom:
        secretKeyRef:
          name: sslstorepass
          key: password
    - name: SERVER_SSL_KEYSTORETYPE
      value: JKS
    - name: SERVER_SSL_TRUSTSTORE
      value: /etc/truststore/cacerts
    - name: SERVER_SSL_TRUSTSTOREPASSWORD
      valueFrom:
        secretKeyRef:
          name: sslstorepass
          key: password

  # ExtraVolumes can be used to define extra volumes for the Operate pods, useful for tls and self-signed certificates
  extraVolumes:
    - name: keystore
      secret:
        secretName: sslkeystore
    - name: truststore
      secret:
        secretName: sslcacerts

  # ExtraVolumeMounts can be used to mount extra volumes for the Operate pods, useful for tls and self-signed certificates
  extraVolumeMounts:
    - name: keystore
      mountPath: /etc/keystore
      readOnly: false
    - name: truststore
      mountPath: /etc/truststore
      readOnly: false

I can see the cacerts (truststore) and keystore are created inside the pod in the respective folders, but when I check the logs for the Identity pods, this is what I see:

k logs camunda-identity-c8478db84-fl45c -f

image
image1902×645 44.5 KB

Identity is not able to connect to Keycloak as you can see even though Keycloak is reachable through ingress/service.

I have read multiple threads but cannot find an answer to this issue.
Is there something am I missing?

2

For anyone that has dealt with this issue, I was able to solve it by overwriting the default Java cacerts file located in /opt/java/openjdk/lib/security/cacerts in the pod with my truststore.

For some reason, the application is not picking the custom truststore path specified in the SERVER_SSL_TRUSTSTORE environment variable.