Hello Camunda,
We suspect a bug in Camunda’s latest 8.6 helm charts. Trying to enable zeebe-gateway for either gce-ingress
or nginx-ingress
, the zeebe-gateway
section fails to update broker.standalone.yaml.template
or application.yaml
on the 2 zeebe-gateway pods. Has anyone experienced this? We used this link to confirm our attempts were aligned to Camunda’s docs - https://docs.camunda.io/docs/self-managed/zeebe-deployment/security/secure-client-communication/
We confirmed both nginx & gce ingresses at the ALB are a pass-through and the backend does not seem to get enabled for TLS using the values.yaml changes as shown below.
zeebe-gateway:
extraVolumes:
- name: nginx-tls-secret
secret:
secretName: nginx-tls-secret
extraVolumeMounts:
- name: nginx-tls-secret
mountPath: "/usr/local/zeebe/certs"
readOnly: true
security:
enabled: true
privateKeyPath: /usr/local/zeebe/certs/tls.key
certificateChainPath: /usr/local/zeebe/certs/tls.crt
server:
ssl:
enabled: true
certificate-private-key: /usr/local/zeebe/certs/tls.key
certificate: /usr/local/zeebe/certs/tls.crt
service:
annotations:
cloud.google.com/neg: '{"ingress": true}' # Creates a NEG after an Ingress is created
cloud.google.com/app-protocols: '{"gateway":"HTTP2"}'
cloud.google.com/backend-config: '{"default": "camunda-hc-zeebe-gateway"}'
labels:
app: zeebe-gateway
name: camunda-zeebe-gateway
spec:
type: NodePort
ports:
- name: gateway
port: 26500
targetPort: 26500
protocol: TCP
selector:
app: zeebe-gateway
app.kubernetes.io/component: camunda-zeebe-gateway
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: zeebe-gateway
app.kubernetes.io/part-of: camunda-platform
We executed helm charts providing environment variables.
helm install camunda camunda/camunda-platform -f camunda-values.yaml \
--set 'primary.extraEnvVars[0].name=POSTGRESQL_REPLICATION_USE_PASSFILE' \
--set 'primary.extraEnvVars[0].value=false' \
--set 'primary.extraEnvVars[1].name=CAMUNDA_REST_QUERY_ENABLED' \
--set 'primary.extraEnvVars[1].value=true' \
--set 'primary.extraEnvVars[2].name=ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED' \
--set 'primary.extraEnvVars[2].value=true' \
--set 'primary.extraEnvVars[3].name=ZEEBE_BROKER_GATEWAY_SECURITY_CERTIFICATECHAINPATH' \
--set 'primary.extraEnvVars[3].value=/usr/local/zeebe/certs/tls.crt' \
--set 'primary.extraEnvVars[4].name=ZEEBE_BROKER_GATEWAY_SECURITY_PRIVATEKEYPATH' \
--set 'primary.extraEnvVars[4].value=/usr/local/zeebe/certs/tls.key'
We’ve been struggling with this for 2 weeks now. Any insight would be greatly appreciated.
Best regards,
Atul
We manually modified zeebe-gateway’s ConfigMap to use SSL for rest and gRPC and forced it to rebuild the application.yaml. Restarted the zeebe-gateway pods and that worked. yay!
But now camunda-connectors Readiness probe fails and therefore, the pod is in failed state. Anyone have ideas? We created this backend config and that did not help either. We are unable to start an instance of any process definition because of this. It throws an error with no information.
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
name: camunda-hc-connectors
spec:
healthCheck:
checkIntervalSec: 15
port: 8080
type: HTTP
requestPath: /actuator/health/readiness
In addition to this, we also modified configMap for camunda-connectors:
security:
plaintext = false
Do we need to deploy certs to Connectors pod as well? From the docs, there is no mention of this. Since camunda-connectors is a zeebe client, simply changing plaintext
value to false
is sufficient. But that did not help. Tasklist throws errors
2024-11-24 20:18:44.754 [] [http-nio-0.0.0.0-8080-exec-21] [] WARN
io.camunda.tasklist.webapp.api.rest.v1.controllers.internal.ProcessInternalController - Network closed for unknown reason
io.camunda.tasklist.exceptions.TasklistRuntimeException: Network closed for unknown reason
at io.camunda.tasklist.webapp.service.ProcessService.startProcessInstance(ProcessService.java:128) ~[tasklist-webapp-8.6.5.jar:8.6.5]
at io.camunda.tasklist.webapp.api.rest.v1.controllers.internal.ProcessInternalController.startProcessInstance(ProcessInternalController.java:200) ~[tasklist-webapp-8.6.5.jar:8.6.5]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:355) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:67) ~[spring-security-core-6.3.4.jar:6.3.4]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.14.jar:6.1.14]
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:720) ~[spring-aop-6.1.14.jar:6.1.14]
at io.camunda.tasklist.webapp.api.rest.v1.controllers.internal.ProcessInternalController$$SpringCGLIB$$0.startProcessInstance(<generated>) ~[tasklist-webapp-8.6.5.jar:8.6.5]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:926) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:831) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.1.14.jar:6.1.14]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:888) ~[spring-webmvc-6.1.14.jar:6.1.14]
at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:614) ~[jakarta.servlet-api-6.0.0.jar:6.0.0]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.31.jar:10.1.31]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.1.14.jar:6.1.14]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.31.jar:10.1.31]
at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) ~[spring-web-6.1.14.jar:6.1.14]
at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) ~[spring-web-6.1.14.jar:6.1.14]
at io.camunda.zeebe.gateway.RestApiCompositeFilter.doFilter(RestApiCompositeFilter.java:36) ~[camunda-zeebe-8.6.5.jar:8.6.5]
... 1 more
Why would these be an issue since, these pods are within the same cluster?
Anyone have any recommendations?
Thanks,
Atul
Hi @atultewari,
Were you able to find a solution?
Thanks,
Alex
Can you share the ConfigMap(without any secure data of course)?
Hello Alex,
We abandoned enabling TLS for zeebe since all components (e.g. Tasklist & Operate) dependent on Zeebe require their application.yaml and ConfigMaps to be modified manually after the deployment. The Helm charts seem to not modify the application.yamls for these components correctly.
I really appreciate you reaching out.