Zeebe 8.5.0 deploy errors

Hi! I am trying to deploy a zeebe cluster, but I have errors.

First with the init-exporter-hazelcast and with the zeebe cluster start:

Please, can someone help me to deploy correctly the zeebe cluster/?

init-exporters-hazelcast ERROR:

zeebe errors:



2024-04-24T15:34:55.186662524Z + exec /usr/local/zeebe/bin/broker 2024-04-24T15:34:55.203696778Z Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError 2024-04-24T15:35:00.827969001Z ______ ______ ______ ____ ______ ____ _____ ____ _ __ ______ _____ 2024-04-24T15:35:00.828090237Z |___ / | ____| | ____| | _ \ | ____| | _ \ | __ \ / __ \ | |/ / | ____| | __ \ 2024-04-24T15:35:00.828130326Z / / | |__ | |__ | |_) | | |__ | |_) | | |__) | | | | | | ' / | |__ | |__) |
/ / | __| | __| | _ < | __| | _ < | _ / | | | | | < | __| | _ /
/ /__ | |____ | |____ | |_) | | |____ | |_) | | | \ \ | |__| | | . \ | |____ | | \ \ 2024-04-24T15:35:00.828165591Z /_____| |______| |______| |____/ |______| |____/ |_| \_\ \____/ |_|\_\ |______| |_| \_\
2024-04-24T15:35:01.048885211Z 2024-04-24 15:35:01.045 [] [main] [] INFO 2024-04-24T15:35:01.048944249Z io.camunda.zeebe.broker.StandaloneBroker - Starting StandaloneBroker v8.5.0 using Java 21.0.2 with PID 7 (/usr/local/zeebe/lib/camunda-zeebe-8.5.0.jar started by camunda in /usr/local/zeebe)
2024-04-24 15:35:01.050 [] [main] [] INFO 2024-04-24T15:35:01.051532850Z io.camunda.zeebe.broker.StandaloneBroker - The following 1 profile is active: "broker"
2024-04-24 15:35:06.942 [] [main] [] WARN 2024-04-24T15:35:06.943391364Z io.camunda.zeebe.broker.system - Configuration parameter data.diskUsageCommandWatermark is deprecated. Use data.disk.freeSpace.processing instead. 2024-04-24T15:35:06.943699556Z 2024-04-24 15:35:06.943 [] [main] [] WARN 2024-04-24T15:35:06.943735126Z io.camunda.zeebe.broker.system - Configuration parameter data.diskUsageReplicationWatermark is deprecated. Use data.disk.freeSpace.replication instead.
2024-04-24 15:35:12.952 [] [main] [] INFO 2024-04-24T15:35:12.953656089Z org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port 8080
2024-04-24 15:35:13.339 [] [main] [] INFO 2024-04-24T15:35:13.339893370Z org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 13 endpoint(s) beneath base path '/actuator'
2024-04-24 15:35:13.722 [] [main] [] INFO 2024-04-24T15:35:13.723362220Z org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port 9600
2024-04-24 15:35:13.741 [] [main] [] INFO
io.camunda.zeebe.broker.StandaloneBroker - Started StandaloneBroker in 14.476 seconds (process running for 18.54)
2024-04-24 15:35:13.753 [] [main] [] WARN
io.camunda.zeebe.broker.system - No backup store is configured. Backups will not be taken
2024-04-24 15:35:13.961 [] [main] [] INFO
org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLogger - 2024-04-24T15:35:13.962282158Z
Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-04-24 15:35:13.964 [] [main] [] WARN 2024-04-24T15:35:14.026486467Z org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext - Exception thrown from ApplicationListener handling ContextClosedEvent 2024-04-24T15:35:14.026543958Z java.lang.NullPointerException: Cannot invoke "io.camunda.zeebe.broker.Broker.close()" because "this.broker" is null 2024-04-24T15:35:14.026556626Z at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:106) ~[camunda-zeebe-8.5.0.jar:8.5.0]
at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:34) ~[camunda-zeebe-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.026573971Z at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026583377Z at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026592197Z at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026601445Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026619634Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:457) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026637017Z at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1130) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026646556Z at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.doClose(ReactiveWebServerApplicationContext.java:149) ~[spring-boot-3.2.4.jar:3.2.4]
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1093) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026711657Z at org.springframework.boot.actuate.autoconfigure.web.server.ChildManagementContextInitializer$CloseManagementContextListener.propagateCloseIfNecessary(ChildManagementContextInitializer.java:242) ~[spring-boot-actuator-autoconfigure-3.2.4.jar:3.2.4]
at org.springframework.boot.actuate.autoconfigure.web.server.ChildManagementContextInitializer$CloseManagementContextListener.onApplicationFailedEvent(ChildManagementContextInitializer.java:237) ~[spring-boot-actuator-autoconfigure-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026741381Z at org.springframework.boot.actuate.autoconfigure.web.server.ChildManagementContextInitializer$CloseManagementContextListener.onApplicationEvent(ChildManagementContextInitializer.java:228) ~[spring-boot-actuator-autoconfigure-3.2.4.jar:3.2.4]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026763421Z at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026772091Z at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026790808Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026799063Z at org.springframework.boot.context.event.EventPublishingRunListener.failed(EventPublishingRunListener.java:119) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026807491Z at org.springframework.boot.SpringApplicationRunListeners.callFailedListener(SpringApplicationRunListeners.java:94) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026816471Z at org.springframework.boot.SpringApplicationRunListeners.lambda$failed$7(SpringApplicationRunListeners.java:85) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026826065Z at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) [spring-boot-3.2.4.jar:3.2.4]
at org.springframework.boot.SpringApplicationRunListeners.failed(SpringApplicationRunListeners.java:84) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026870291Z at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:810) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026880345Z at org.springframework.boot.SpringApplication.run(SpringApplication.java:344) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026888557Z at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:82) [camunda-zeebe-8.5.0.jar:8.5.0]
2024-04-24 15:35:14.026 [] [main] [] INFO 2024-04-24T15:35:14.028863879Z org.springframework.boot.web.embedded.netty.GracefulShutdown - Commencing graceful shutdown. Waiting for active requests to complete
2024-04-24 15:35:14.034 [] [netty-shutdown] [] INFO 2024-04-24T15:35:14.035783327Z org.springframework.boot.web.embedded.netty.GracefulShutdown - Graceful shutdown complete
2024-04-24 15:35:14.067 [] [main] [] ERROR 2024-04-24T15:35:14.119696317Z org.springframework.boot.SpringApplication - Application run failed 2024-04-24T15:35:14.119709236Z java.lang.IllegalStateException: Failed to load exporter with configuration: ExporterCfg{, jarPath='/exporters/zeebe-hazelcast-exporter.jar', className='io.zeebe.hazelcast.exporter.HazelcastExporter', args=null} 2024-04-24T15:35:14.119719933Z at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:151) ~[zeebe-broker-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119729165Z at io.camunda.zeebe.broker.Broker.<init>(Broker.java:71) ~[zeebe-broker-8.5.0.jar:8.5.0]
at io.camunda.zeebe.broker.Broker.<init>(Broker.java:49) ~[zeebe-broker-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119777826Z at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:99) ~[camunda-zeebe-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119787068Z at org.springframework.boot.SpringApplication.lambda$callRunner$5(SpringApplication.java:790) ~[spring-boot-3.2.4.jar:3.2.4]
at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83) ~[spring-core-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.119804649Z at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60) ~[spring-core-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.119812088Z at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88) ~[spring-core-6.1.5.jar:6.1.5]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.119827885Z at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:789) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.119835721Z at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:774) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.119843458Z at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?] 2024-04-24T15:35:14.119850882Z at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?] 2024-04-24T15:35:14.119867850Z at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?] 2024-04-24T15:35:14.119876253Z at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?] 2024-04-24T15:35:14.119909075Z at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774) [spring-boot-3.2.4.jar:3.2.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:341) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.119926351Z at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:82) [camunda-zeebe-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119935022Z Caused by: io.camunda.zeebe.util.jar.ExternalJarLoadException: Cannot load JAR at [/exporters/zeebe-hazelcast-exporter.jar]: is not readable 2024-04-24T15:35:14.119943973Z at io.camunda.zeebe.util.jar.ExternalJarRepository.verifyJarPath(ExternalJarRepository.java:78) ~[zeebe-util-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119973470Z at io.camunda.zeebe.util.jar.ExternalJarRepository.load(ExternalJarRepository.java:54) ~[zeebe-util-8.5.0.jar:8.5.0]
at io.camunda.zeebe.util.jar.ExternalJarRepository.load(ExternalJarRepository.java:47) ~[zeebe-util-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.120016704Z at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.load(ExporterRepository.java:76) ~[zeebe-broker-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.120026198Z at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:149) ~[zeebe-broker-8.5.0.jar:8.5.0]
... 21 more
2024-04-24 15:35:14.121 [] [main] [] WARN
org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext - Exception thrown from ApplicationListener handling ContextClosedEvent 2024-04-24T15:35:14.122785557Z java.lang.NullPointerException: Cannot invoke "io.camunda.zeebe.broker.Broker.close()" because "this.broker" is null 2024-04-24T15:35:14.122794079Z at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:106) ~[camunda-zeebe-8.5.0.jar:8.5.0]
at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:34) ~[camunda-zeebe-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.122812384Z at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122821842Z at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122830305Z at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122845513Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122851442Z at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1130) [spring-context-6.1.5.jar:6.1.5]
at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.doClose(ReactiveWebServerApplicationContext.java:149) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.122863619Z at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1093) [spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122869420Z at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:816) [spring-boot-3.2.4.jar:3.2.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:344) [spring-boot-3.2.4.jar:3.2.4]
at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:82) [camunda-zeebe-8.5.0.jar:8.5.0]
2024-04-24 15:35:14.123 [] [main] [] INFO
org.springframework.boot.web.embedded.netty.GracefulShutdown - Commencing graceful shutdown. Waiting for active requests to complete
2024-04-24 15:35:14.125 [] [netty-shutdown] [] INFO 2024-04-24T15:35:14.126995485Z org.springframework.boot.web.embedded.netty.GracefulShutdown - Graceful shutdown complete
2024-04-24 15:35:15.159 [] [atomix-cluster-0] [] INFO 2024-04-24T15:35:15.161588205Z io.atomix.cluster.AtomixCluster - Stopped

This is my HR:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: zeebe
  namespace: staging
spec:
  releaseName: zeebe
  interval: 5m
  chart:
    spec:
      chart: camunda-platform
      version: 10.0.2
      sourceRef:
        kind: HelmRepository
        name: camunda
        namespace: default
  values:
    global:
      zeebeClusterName: zeebe-staging
      identity:
        auth:
          enabled: false
    elasticsearch:
      enabled: true
      nodeSelector:
        node-role.kubernetes.io/zeebe: "true"
      resources:
        requests:
          cpu: 1
          memory: 2Gi
        limits:
          cpu: 2
          memory: 2Gi
      volumeClaimTemplate:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 32Gi
    connectors:
      enabled: false
    identity:
      enabled: false
    identityKeycloak:
      enabled: false
    identityPostgresql:
      enabled: false
    optimize: 
      enabled: false
    operate: 
      enabled: false
    tasklist: 
      enabled: false
    zeebe:
      enabled: true
      debug: true
      clusterSize: 3
      replicationFactor: 3
      partitionCount: 5
      image:
        repository: camunda/zeebe
        tag: 8.5.0
      nodeSelector:
        node-role.kubernetes.io/zeebe: "true"
      podSecurityContext:
        runAsNonRoot: false
      containerSecurityContext:
        readOnlyRootFilesystem: false
      initContainers:
        - name: init-exporters-hazelcast
          image: busybox:1.35
          command: ["/bin/sh", "-c"]
          args:
            [
              "wget --no-check-certificate https://repo1.maven.org/maven2/io/zeebe/hazelcast/zeebe-hazelcast-exporter/0.8.0-alpha1/zeebe-hazelcast-exporter-0.8.0-alpha1-jar-with-dependencies.jar -O /exporters/zeebe-hazelcast-exporter.jar; ls -al /exporters",
            ]
          volumeMounts:
            - name: exporters
              mountPath: /exporters/
      env:
        - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH
          value: /exporters/zeebe-hazelcast-exporter.jar
        - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME
          value: io.zeebe.hazelcast.exporter.HazelcastExporter
        - name: ZEEBE_HAZELCAST_PORT
          value: "5701"
        - name: ZEEBE_HAZELCAST_CLUSTER_NAME
          value: "zeebe-hazelcast-staging"
        - name: HZ_NETWORK_RESTAPI_ENABLED
          value: "true"
        - name: ZEEBE_HAZELCAST_SECURITY_ENABLED
          value: "false"
        - name: ZEEBE_HAZELCAST_REMOTE_ADDRESS
          value: "zeebe-hazelcast-staging"
        - name: ZEEBE_BROKER_DATA_SNAPSHOTPERIOD
          value: "15m"
        - name: ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK
          value: "0.85"
        - name: ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK
          value: "0.95"
        - name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_WRITE_BUFFER_SIZE
          value: "8MB"
        - name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_MAX_WRITE_BUFFER_SIZE_TO_MAINTAIN
          value: "16MB"
        - name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_MAX_OPEN_FILES
          value: "1024"
      resources:
        requests:
          cpu: 800m
          memory: 1200Mi
        limits:
          cpu: 960m
          memory: 1920Mi
    zeebe-gateway:
      image:
        repository: camunda/zeebe
        tag: 8.5.0
      nodeSelector:
        node-role.kubernetes.io/zeebe: "true"
      resources:
        requests:
          cpu: 400m
          memory: 450Mi
        limits:
          cpu: 400m
          memory: 450Mi

Hi @LuisoBKM - I don’t know if the broker not starting is related to the exporter or not, but I would start with troubleshooting the exporter since there’s a clear error in the log:

Cannot load JAR at [/exporters/zeebe-hazelcast-exporter.jar]: is not readable

You are loading version 0.8.0-alpha1 while the latest release is 1.4.0. I would start with troubleshooting that. The Hazelcast exporter is a community-maintained exporter and I do not know if it is compatible with 8.5.0, so that is something to consider as well.

Also, pay attention the note at the bottom of this page regarding writable volume mounts: Install Zeebe exporters | Camunda 8 Docs

Finally we don’t use the exporter and Zeebe works :slight_smile:

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.