We suspect a bug in helm Camunda 8.6 enabling TLS for zeebe-gateway

atultewari · November 23, 2024, 8:27pm

Hello Camunda,
We suspect a bug in Camunda’s latest 8.6 helm charts. Trying to enable zeebe-gateway for either gce-ingress or nginx-ingress, the zeebe-gateway section fails to update broker.standalone.yaml.template or application.yaml on the 2 zeebe-gateway pods. Has anyone experienced this? We used this link to confirm our attempts were aligned to Camunda’s docs - https://docs.camunda.io/docs/self-managed/zeebe-deployment/security/secure-client-communication/

We confirmed both nginx & gce ingresses at the ALB are a pass-through and the backend does not seem to get enabled for TLS using the values.yaml changes as shown below.

zeebe-gateway:
  extraVolumes:
    - name: nginx-tls-secret
      secret:
        secretName: nginx-tls-secret
  extraVolumeMounts:
    - name: nginx-tls-secret
      mountPath: "/usr/local/zeebe/certs"
      readOnly: true
  security:
    enabled: true
    privateKeyPath: /usr/local/zeebe/certs/tls.key
    certificateChainPath: /usr/local/zeebe/certs/tls.crt
  server:
    ssl:
      enabled: true
      certificate-private-key: /usr/local/zeebe/certs/tls.key
      certificate: /usr/local/zeebe/certs/tls.crt
  service:
    annotations:
      cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
      cloud.google.com/app-protocols: '{"gateway":"HTTP2"}'
      cloud.google.com/backend-config: '{"default": "camunda-hc-zeebe-gateway"}' 
    labels:
      app: zeebe-gateway
    name: camunda-zeebe-gateway   
    spec:
      type: NodePort
      ports:
      - name: gateway
        port: 26500
        targetPort: 26500
        protocol: TCP
      selector:
        app: zeebe-gateway
        app.kubernetes.io/component: camunda-zeebe-gateway
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: zeebe-gateway
        app.kubernetes.io/part-of: camunda-platform

We executed helm charts providing environment variables.

helm install camunda camunda/camunda-platform -f camunda-values.yaml \
   --set 'primary.extraEnvVars[0].name=POSTGRESQL_REPLICATION_USE_PASSFILE' \
   --set 'primary.extraEnvVars[0].value=false' \
   --set 'primary.extraEnvVars[1].name=CAMUNDA_REST_QUERY_ENABLED' \
   --set 'primary.extraEnvVars[1].value=true' \
   --set 'primary.extraEnvVars[2].name=ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED' \
   --set 'primary.extraEnvVars[2].value=true' \
   --set 'primary.extraEnvVars[3].name=ZEEBE_BROKER_GATEWAY_SECURITY_CERTIFICATECHAINPATH' \
   --set 'primary.extraEnvVars[3].value=/usr/local/zeebe/certs/tls.crt' \
   --set 'primary.extraEnvVars[4].name=ZEEBE_BROKER_GATEWAY_SECURITY_PRIVATEKEYPATH' \
   --set 'primary.extraEnvVars[4].value=/usr/local/zeebe/certs/tls.key'

We’ve been struggling with this for 2 weeks now. Any insight would be greatly appreciated.

Best regards,
Atul

atultewari · November 24, 2024, 3:32pm

We manually modified zeebe-gateway’s ConfigMap to use SSL for rest and gRPC and forced it to rebuild the application.yaml. Restarted the zeebe-gateway pods and that worked. yay!

But now camunda-connectors Readiness probe fails and therefore, the pod is in failed state. Anyone have ideas? We created this backend config and that did not help either. We are unable to start an instance of any process definition because of this. It throws an error with no information.

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: camunda-hc-connectors
spec:
  healthCheck:
    checkIntervalSec: 15
    port: 8080
    type: HTTP
    requestPath: /actuator/health/readiness

In addition to this, we also modified configMap for camunda-connectors:

security:
  plaintext = false

Do we need to deploy certs to Connectors pod as well? From the docs, there is no mention of this. Since camunda-connectors is a zeebe client, simply changing plaintext value to false is sufficient. But that did not help. Tasklist throws errors

2024-11-24 20:18:44.754 [] [http-nio-0.0.0.0-8080-exec-21] [] WARN
io.camunda.tasklist.webapp.api.rest.v1.controllers.internal.ProcessInternalController - Network closed for unknown reason
io.camunda.tasklist.exceptions.TasklistRuntimeException: Network closed for unknown reason
at io.camunda.tasklist.webapp.service.ProcessService.startProcessInstance(ProcessService.java:128) ~[tasklist-webapp-8.6.5.jar:8.6.5]
at io.camunda.tasklist.webapp.api.rest.v1.controllers.internal.ProcessInternalController.startProcessInstance(ProcessInternalController.java:200) ~[tasklist-webapp-8.6.5.jar:8.6.5]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67) ~[spring-security-core-6.3.4.jar:6.3.4]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720) ~[spring-aop-6.1.14.jar:6.1.14]
at io.camunda.tasklist.webapp.api.rest.v1.controllers.internal.ProcessInternalController$$SpringCGLIB$$0.startProcessInstance(<generated>) ~[tasklist-webapp-8.6.5.jar:8.6.5]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:926) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:831) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:888) ~[spring-webmvc-6.1.14.jar:6.1.14]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614) ~[jakarta.servlet-api-6.0.0.jar:6.0.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.31.jar:10.1.31]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.14.jar:6.1.14]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) ~[spring-web-6.1.14.jar:6.1.14]
at io.camunda.zeebe.gateway.RestApiCompositeFilter.doFilter(RestApiCompositeFilter.java:36) ~[camunda-zeebe-8.6.5.jar:8.6.5]
... 1 more

Why would these be an issue since, these pods are within the same cluster?

Anyone have any recommendations?

Thanks,
Atul

Alex_Voloshyn · January 29, 2025, 7:14am

Hi @atultewari,
Were you able to find a solution?

Thanks,
Alex

Alex_Voloshyn · January 29, 2025, 11:48am

Can you share the ConfigMap(without any secure data of course)?

atultewari · February 4, 2025, 12:38pm

Hello Alex,
We abandoned enabling TLS for zeebe since all components (e.g. Tasklist & Operate) dependent on Zeebe require their application.yaml and ConfigMaps to be modified manually after the deployment. The Helm charts seem to not modify the application.yamls for these components correctly.

I really appreciate you reaching out.

nathan.loding · March 14, 2025, 6:09pm

Hey @atultewari - late reply, but I found this thread looking through the forum and wanted to give some additional information. I believe this is expected behavior because TLS isn’t enabled on the gateway but rather at the ingress in the Helm charts. If you want TLS to terminate at the gateway instead of the ingress, that requires custom configuration, as you found with your configmaps. Hope that helps!

atultewari · March 14, 2025, 7:04pm

Thank you. That makes sense

system · March 21, 2025, 7:05pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.