noPermission when browsing to tasklist, operate and optimize

nhduy · June 6, 2023, 7:06am

Hi,

I’m trying to secure keycloak and identity by following this guide Making Identity production ready | Camunda Platform 8 Docs

After setting up a self-signed cert (auto-generated by keycloak helm chart), Identity connects to keycloak successfully. But tasklist, operate and optimize returns error

How to reproduce:

Install Camunda 8.2.5 using the value below

global:
  identity:
    auth:
      publicIssuerUrl: "https://keycloak.example.com/auth/realms/camunda-platform"
      operate:
        redirectUrl: "https://operate.example.com"
      tasklist:
        redirectUrl: "https://tasklist.example.com"
      optimize:
        redirectUrl: "https://optimize.example.com"

identity:
  service:
    type: NodePort

  keycloak:
    tls:
      enabled: true
      autoGenerated: true
    service:
      type: NodePort
    extraEnvVars:
      - name: KEYCLOAK_PROXY_ADDRESS_FORWARDING
        value: "true"
      - name: KEYCLOAK_FRONTEND_URL
        value: "https://keycloak.example.com"
      - name: KEYCLOAK_HTTPS_USE_PEM
        value: "true"
      - name: KEYCLOAK_HTTPS_CERTIFICATE_FILE
        value: "/opt/bitnami/keycloak/certs/tls.crt"
      - name: KEYCLOAK_HTTPS_CERTIFICATE_KEY_FILE
        value: "/opt/bitnami/keycloak/certs/tls.key"

  env:
    - name: IDENTITY_URL
      value: "https://identity.example.com"
    - name: JAVA_TOOL_OPTIONS
      value: "-Djavax.net.ssl.trustStore=/app/certificates/keycloak.jks"
  extraVolumeMounts:
    - name: keycloak-cert
      mountPath: /app/certificates/
  extraVolumes:
    - name: keycloak-cert
      secret:
        secretName: keycloak-jks
        optional: false


zeebe-gateway:
  service:
    type: NodePort

operate:
  service:
    type: NodePort
  env:
    - name: JAVA_TOOL_OPTIONS
      value: "-Djavax.net.ssl.trustStore=/usr/local/operate/certificates/keycloak.jks"
  extraVolumeMounts:
    - name: keystore
      mountPath: /usr/local/operate/certificates/keycloak.jks
      subPath: keycloak.jks
  extraVolumes:
    - name: keystore
      secret:
        secretName: keycloak-jks
        optional: false

tasklist:
  service:
    type: NodePort
  env:
    - name: JAVA_TOOL_OPTIONS
      value: "-Djavax.net.ssl.trustStore=/app/certificates/keycloak.jks"
  extraVolumeMounts:
    - name: keystore
      mountPath: "/app/certificates/keycloak.jks"
      subPath: keycloak.jks
  extraVolumes:
    - name: keystore
      secret:
        secretName: keycloak-jks
        optional: false

optimize:
  service:
    type: NodePort
  env:
    - name: JAVA_TOOL_OPTIONS
      value: "-Djavax.net.ssl.trustStore=/optimize/certificates/keycloak.jks"
  extraVolumeMounts:
    - name: keystore
      mountPath: /optimize/certificates/keycloak.jks
      subPath: keycloak.jks
  extraVolumes:
    - name: keystore
      secret:
        secretName: keycloak-jks
        optional: false

prometheusServiceMonitor:
  enabled: true
  labels:
    release: metrics
  scrapeInterval: 10s

After installation, keycloak auto-generate a secret containing certificate
Create a jks file and a secret out of it.

kubectl get secret camunda-keycloak-crt -n camunda  -o jsonpath='{.data.ca\.crt}'|base64 -d > keycloak-ca.crt
keytool -import -alias keycloak -keystore keycloak.jks -storetype jks -file keycloak-ca.crt -noprompt
Enter keystore password:
Re-enter new password:
Certificate was added to keystore
kubectl create secret generic keycloak-jks --from-file=keycloak.jks -n camunda

Logs:

2023-06-06 04:14:03.461 ERROR 1 --- [io-8080-exec-11] i.c.t.w.s.i.IdentityController           : Error in authentication callback: 

io.camunda.identity.sdk.impl.rest.exception.RestException: request failed with status code '403' and body '{"error":"invalid_request","error_description":"HTTPS required"}'
	at io.camunda.identity.sdk.impl.rest.RestClient.send(RestClient.java:114) ~[identity-sdk-8.2.5.jar:?]
	at io.camunda.identity.sdk.impl.rest.RestClient.request(RestClient.java:96) ~[identity-sdk-8.2.5.jar:?]
	at io.camunda.identity.sdk.impl.keycloak.KeycloakAuthentication.exchangeAuthCode(KeycloakAuthentication.java:72) ~[identity-sdk-8.2.5.jar:?]
	at jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at io.camunda.identity.sdk.annotation.AnnotationProcessor.lambda$apply$0(AnnotationProcessor.java:33) ~[identity-sdk-8.2.5.jar:?]
	at jdk.proxy2.$Proxy112.exchangeAuthCode(Unknown Source) ~[?:?]
	at io.camunda.tasklist.webapp.security.identity.IdentityService.lambda$getAuthenticationFor$0(IdentityService.java:104) ~[classes/:?]
	at net.jodah.failsafe.Functions.lambda$get$0(Functions.java:48) ~[failsafe-2.4.4.jar:2.4.4]
	at net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:66) ~[failsafe-2.4.4.jar:2.4.4]
	at net.jodah.failsafe.Execution.executeSync(Execution.java:128) ~[failsafe-2.4.4.jar:2.4.4]
	at net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:379) ~[failsafe-2.4.4.jar:2.4.4]
	at net.jodah.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:68) ~[failsafe-2.4.4.jar:2.4.4]
	at io.camunda.tasklist.webapp.security.identity.IdentityService.requestWithRetry(IdentityService.java:60) ~[classes/:?]
	at io.camunda.tasklist.webapp.security.identity.IdentityService.getAuthenticationFor(IdentityService.java:100) ~[classes/:?]
	at io.camunda.tasklist.webapp.security.identity.IdentityController.loggedInCallback(IdentityController.java:86) ~[classes/:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
	at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:207) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:152) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.8.jar:6.0]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.0.8.jar:6.0.8]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.8.jar:6.0]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.springframework.security.web.FilterChainProxy.lambda$doFilterInternal$3(FilterChainProxy.java:231) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$FilterObservation$SimpleFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:425) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$1(ObservationFilterChainDecorator.java:285) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator.lambda$wrapSecured$0(ObservationFilterChainDecorator.java:78) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:124) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:117) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:179) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.oauth2.server.resource.web.authentication.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:128) ~[spring-security-oauth2-resource-server-6.0.3.jar:6.0.3]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:107) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:93) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:82) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.context.SecurityContextHolderFilter.doFilter(SecurityContextHolderFilter.java:69) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:62) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:172) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.wrapFilter(ObservationFilterChainDecorator.java:185) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$AroundFilterObservation$SimpleAroundFilterObservation.lambda$wrap$0(ObservationFilterChainDecorator.java:268) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$ObservationFilter.doFilter(ObservationFilterChainDecorator.java:169) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.ObservationFilterChainDecorator$VirtualFilterChain.doFilter(ObservationFilterChainDecorator.java:133) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:233) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:191) ~[spring-security-web-6.0.3.jar:6.0.3]
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:352) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:268) ~[spring-web-6.0.8.jar:6.0.8]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:109) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.0.8.jar:6.0.8]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.0.8.jar:6.0.8]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:166) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:738) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:341) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:894) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1741) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-10.1.8.jar:10.1.8]
	at java.lang.Thread.run(Unknown Source) ~[?:?]

Environment:

Platform: EKS
Chart version: 8.2.5

sarslanturk · June 12, 2023, 8:56am

Did you find anything? I get also same error

Sametimes it work when i delete /noPermission path on url.

Thank you

@nhduy

nathan.loding · June 13, 2023, 4:07pm

Hi @nhduy and @sarslanturk - I had one of our platform engineers take a look, and there are some issues with the values.yml shared above. For instance, service.type: NodePort isn’t a supported option, and some Keycloak options such as autoGenerated: true aren’t supported either. Most importantly, though, is that the global.identity.auth.*.redirectUrl values are all https, but the tls settings for each component is missing.

All of the supported configuration values are documented here, along with some sample configurations that may help also:

You might also find this guide to be helpful:

Hope that helps!

sarslanturk · June 21, 2023, 10:20pm

Hello again,

Actually, I saw that we installed it as you described. In my case, I think the problem was caused by keycloak.

In the system we have installed, 2 different clusters and 2 different realms use the same external keycloak. After our experiments, when we increased the number of identity pods to 2, we observed that the error we received was gone. For about 6 or 7 days, when I tried to access urls via ingress, I observed that it responded almost 95% correctly.

Perhaps increasing the number of identities in proportion to the number of realms in the use of external keycloak will create a high available identity structure, so problems such as authentication have decreased.

These are just my observations. If I get more different observations about the situation in the future, I will update here.

Note: When we install camunda platform 8 via helmchart on the k8s cluster, the default number of identity pods is 1.