Hi! I am trying to deploy a zeebe cluster, but I have errors.
First with the init-exporter-hazelcast and with the zeebe cluster start:
Please, can someone help me to deploy correctly the zeebe cluster/?
init-exporters-hazelcast ERROR:
zeebe errors:
2024-04-24T15:34:55.186662524Z + exec /usr/local/zeebe/bin/broker 2024-04-24T15:34:55.203696778Z Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError 2024-04-24T15:35:00.827969001Z ______ ______ ______ ____ ______ ____ _____ ____ _ __ ______ _____ 2024-04-24T15:35:00.828090237Z |___ / | ____| | ____| | _ \ | ____| | _ \ | __ \ / __ \ | |/ / | ____| | __ \ 2024-04-24T15:35:00.828130326Z / / | |__ | |__ | |_) | | |__ | |_) | | |__) | | | | | | ' / | |__ | |__) |
/ / | __| | __| | _ < | __| | _ < | _ / | | | | | < | __| | _ /
/ /__ | |____ | |____ | |_) | | |____ | |_) | | | \ \ | |__| | | . \ | |____ | | \ \ 2024-04-24T15:35:00.828165591Z /_____| |______| |______| |____/ |______| |____/ |_| \_\ \____/ |_|\_\ |______| |_| \_\
2024-04-24T15:35:01.048885211Z 2024-04-24 15:35:01.045 [] [main] [] INFO 2024-04-24T15:35:01.048944249Z io.camunda.zeebe.broker.StandaloneBroker - Starting StandaloneBroker v8.5.0 using Java 21.0.2 with PID 7 (/usr/local/zeebe/lib/camunda-zeebe-8.5.0.jar started by camunda in /usr/local/zeebe)
2024-04-24 15:35:01.050 [] [main] [] INFO 2024-04-24T15:35:01.051532850Z io.camunda.zeebe.broker.StandaloneBroker - The following 1 profile is active: "broker"
2024-04-24 15:35:06.942 [] [main] [] WARN 2024-04-24T15:35:06.943391364Z io.camunda.zeebe.broker.system - Configuration parameter data.diskUsageCommandWatermark is deprecated. Use data.disk.freeSpace.processing instead. 2024-04-24T15:35:06.943699556Z 2024-04-24 15:35:06.943 [] [main] [] WARN 2024-04-24T15:35:06.943735126Z io.camunda.zeebe.broker.system - Configuration parameter data.diskUsageReplicationWatermark is deprecated. Use data.disk.freeSpace.replication instead.
2024-04-24 15:35:12.952 [] [main] [] INFO 2024-04-24T15:35:12.953656089Z org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port 8080
2024-04-24 15:35:13.339 [] [main] [] INFO 2024-04-24T15:35:13.339893370Z org.springframework.boot.actuate.endpoint.web.EndpointLinksResolver - Exposing 13 endpoint(s) beneath base path '/actuator'
2024-04-24 15:35:13.722 [] [main] [] INFO 2024-04-24T15:35:13.723362220Z org.springframework.boot.web.embedded.netty.NettyWebServer - Netty started on port 9600
2024-04-24 15:35:13.741 [] [main] [] INFO
io.camunda.zeebe.broker.StandaloneBroker - Started StandaloneBroker in 14.476 seconds (process running for 18.54)
2024-04-24 15:35:13.753 [] [main] [] WARN
io.camunda.zeebe.broker.system - No backup store is configured. Backups will not be taken
2024-04-24 15:35:13.961 [] [main] [] INFO
org.springframework.boot.autoconfigure.logging.ConditionEvaluationReportLogger - 2024-04-24T15:35:13.962282158Z
Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-04-24 15:35:13.964 [] [main] [] WARN 2024-04-24T15:35:14.026486467Z org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext - Exception thrown from ApplicationListener handling ContextClosedEvent 2024-04-24T15:35:14.026543958Z java.lang.NullPointerException: Cannot invoke "io.camunda.zeebe.broker.Broker.close()" because "this.broker" is null 2024-04-24T15:35:14.026556626Z at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:106) ~[camunda-zeebe-8.5.0.jar:8.5.0]
at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:34) ~[camunda-zeebe-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.026573971Z at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026583377Z at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026592197Z at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026601445Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026619634Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:457) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026637017Z at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1130) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026646556Z at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.doClose(ReactiveWebServerApplicationContext.java:149) ~[spring-boot-3.2.4.jar:3.2.4]
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1093) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026711657Z at org.springframework.boot.actuate.autoconfigure.web.server.ChildManagementContextInitializer$CloseManagementContextListener.propagateCloseIfNecessary(ChildManagementContextInitializer.java:242) ~[spring-boot-actuator-autoconfigure-3.2.4.jar:3.2.4]
at org.springframework.boot.actuate.autoconfigure.web.server.ChildManagementContextInitializer$CloseManagementContextListener.onApplicationFailedEvent(ChildManagementContextInitializer.java:237) ~[spring-boot-actuator-autoconfigure-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026741381Z at org.springframework.boot.actuate.autoconfigure.web.server.ChildManagementContextInitializer$CloseManagementContextListener.onApplicationEvent(ChildManagementContextInitializer.java:228) ~[spring-boot-actuator-autoconfigure-3.2.4.jar:3.2.4]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026763421Z at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026772091Z at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026790808Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.026799063Z at org.springframework.boot.context.event.EventPublishingRunListener.failed(EventPublishingRunListener.java:119) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026807491Z at org.springframework.boot.SpringApplicationRunListeners.callFailedListener(SpringApplicationRunListeners.java:94) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026816471Z at org.springframework.boot.SpringApplicationRunListeners.lambda$failed$7(SpringApplicationRunListeners.java:85) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026826065Z at java.base/java.lang.Iterable.forEach(Unknown Source) [?:?]
at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) [spring-boot-3.2.4.jar:3.2.4]
at org.springframework.boot.SpringApplicationRunListeners.failed(SpringApplicationRunListeners.java:84) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026870291Z at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:810) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026880345Z at org.springframework.boot.SpringApplication.run(SpringApplication.java:344) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.026888557Z at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:82) [camunda-zeebe-8.5.0.jar:8.5.0]
2024-04-24 15:35:14.026 [] [main] [] INFO 2024-04-24T15:35:14.028863879Z org.springframework.boot.web.embedded.netty.GracefulShutdown - Commencing graceful shutdown. Waiting for active requests to complete
2024-04-24 15:35:14.034 [] [netty-shutdown] [] INFO 2024-04-24T15:35:14.035783327Z org.springframework.boot.web.embedded.netty.GracefulShutdown - Graceful shutdown complete
2024-04-24 15:35:14.067 [] [main] [] ERROR 2024-04-24T15:35:14.119696317Z org.springframework.boot.SpringApplication - Application run failed 2024-04-24T15:35:14.119709236Z java.lang.IllegalStateException: Failed to load exporter with configuration: ExporterCfg{, jarPath='/exporters/zeebe-hazelcast-exporter.jar', className='io.zeebe.hazelcast.exporter.HazelcastExporter', args=null} 2024-04-24T15:35:14.119719933Z at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:151) ~[zeebe-broker-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119729165Z at io.camunda.zeebe.broker.Broker.<init>(Broker.java:71) ~[zeebe-broker-8.5.0.jar:8.5.0]
at io.camunda.zeebe.broker.Broker.<init>(Broker.java:49) ~[zeebe-broker-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119777826Z at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:99) ~[camunda-zeebe-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119787068Z at org.springframework.boot.SpringApplication.lambda$callRunner$5(SpringApplication.java:790) ~[spring-boot-3.2.4.jar:3.2.4]
at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83) ~[spring-core-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.119804649Z at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60) ~[spring-core-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.119812088Z at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88) ~[spring-core-6.1.5.jar:6.1.5]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.119827885Z at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:789) ~[spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.119835721Z at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:774) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.119843458Z at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?] 2024-04-24T15:35:14.119850882Z at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?] 2024-04-24T15:35:14.119867850Z at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?] 2024-04-24T15:35:14.119876253Z at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?] 2024-04-24T15:35:14.119909075Z at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774) [spring-boot-3.2.4.jar:3.2.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:341) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.119926351Z at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:82) [camunda-zeebe-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119935022Z Caused by: io.camunda.zeebe.util.jar.ExternalJarLoadException: Cannot load JAR at [/exporters/zeebe-hazelcast-exporter.jar]: is not readable 2024-04-24T15:35:14.119943973Z at io.camunda.zeebe.util.jar.ExternalJarRepository.verifyJarPath(ExternalJarRepository.java:78) ~[zeebe-util-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.119973470Z at io.camunda.zeebe.util.jar.ExternalJarRepository.load(ExternalJarRepository.java:54) ~[zeebe-util-8.5.0.jar:8.5.0]
at io.camunda.zeebe.util.jar.ExternalJarRepository.load(ExternalJarRepository.java:47) ~[zeebe-util-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.120016704Z at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.load(ExporterRepository.java:76) ~[zeebe-broker-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.120026198Z at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:149) ~[zeebe-broker-8.5.0.jar:8.5.0]
... 21 more
2024-04-24 15:35:14.121 [] [main] [] WARN
org.springframework.boot.web.reactive.context.AnnotationConfigReactiveWebServerApplicationContext - Exception thrown from ApplicationListener handling ContextClosedEvent 2024-04-24T15:35:14.122785557Z java.lang.NullPointerException: Cannot invoke "io.camunda.zeebe.broker.Broker.close()" because "this.broker" is null 2024-04-24T15:35:14.122794079Z at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:106) ~[camunda-zeebe-8.5.0.jar:8.5.0]
at io.camunda.zeebe.broker.StandaloneBroker.onApplicationEvent(StandaloneBroker.java:34) ~[camunda-zeebe-8.5.0.jar:8.5.0] 2024-04-24T15:35:14.122812384Z at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122821842Z at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122830305Z at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.5.jar:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:451) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122845513Z at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:384) ~[spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122851442Z at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1130) [spring-context-6.1.5.jar:6.1.5]
at org.springframework.boot.web.reactive.context.ReactiveWebServerApplicationContext.doClose(ReactiveWebServerApplicationContext.java:149) [spring-boot-3.2.4.jar:3.2.4] 2024-04-24T15:35:14.122863619Z at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1093) [spring-context-6.1.5.jar:6.1.5] 2024-04-24T15:35:14.122869420Z at org.springframework.boot.SpringApplication.handleRunFailure(SpringApplication.java:816) [spring-boot-3.2.4.jar:3.2.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:344) [spring-boot-3.2.4.jar:3.2.4]
at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:82) [camunda-zeebe-8.5.0.jar:8.5.0]
2024-04-24 15:35:14.123 [] [main] [] INFO
org.springframework.boot.web.embedded.netty.GracefulShutdown - Commencing graceful shutdown. Waiting for active requests to complete
2024-04-24 15:35:14.125 [] [netty-shutdown] [] INFO 2024-04-24T15:35:14.126995485Z org.springframework.boot.web.embedded.netty.GracefulShutdown - Graceful shutdown complete
2024-04-24 15:35:15.159 [] [atomix-cluster-0] [] INFO 2024-04-24T15:35:15.161588205Z io.atomix.cluster.AtomixCluster - Stopped
This is my HR:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: zeebe
namespace: staging
spec:
releaseName: zeebe
interval: 5m
chart:
spec:
chart: camunda-platform
version: 10.0.2
sourceRef:
kind: HelmRepository
name: camunda
namespace: default
values:
global:
zeebeClusterName: zeebe-staging
identity:
auth:
enabled: false
elasticsearch:
enabled: true
nodeSelector:
node-role.kubernetes.io/zeebe: "true"
resources:
requests:
cpu: 1
memory: 2Gi
limits:
cpu: 2
memory: 2Gi
volumeClaimTemplate:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 32Gi
connectors:
enabled: false
identity:
enabled: false
identityKeycloak:
enabled: false
identityPostgresql:
enabled: false
optimize:
enabled: false
operate:
enabled: false
tasklist:
enabled: false
zeebe:
enabled: true
debug: true
clusterSize: 3
replicationFactor: 3
partitionCount: 5
image:
repository: camunda/zeebe
tag: 8.5.0
nodeSelector:
node-role.kubernetes.io/zeebe: "true"
podSecurityContext:
runAsNonRoot: false
containerSecurityContext:
readOnlyRootFilesystem: false
initContainers:
- name: init-exporters-hazelcast
image: busybox:1.35
command: ["/bin/sh", "-c"]
args:
[
"wget --no-check-certificate https://repo1.maven.org/maven2/io/zeebe/hazelcast/zeebe-hazelcast-exporter/0.8.0-alpha1/zeebe-hazelcast-exporter-0.8.0-alpha1-jar-with-dependencies.jar -O /exporters/zeebe-hazelcast-exporter.jar; ls -al /exporters",
]
volumeMounts:
- name: exporters
mountPath: /exporters/
env:
- name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH
value: /exporters/zeebe-hazelcast-exporter.jar
- name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME
value: io.zeebe.hazelcast.exporter.HazelcastExporter
- name: ZEEBE_HAZELCAST_PORT
value: "5701"
- name: ZEEBE_HAZELCAST_CLUSTER_NAME
value: "zeebe-hazelcast-staging"
- name: HZ_NETWORK_RESTAPI_ENABLED
value: "true"
- name: ZEEBE_HAZELCAST_SECURITY_ENABLED
value: "false"
- name: ZEEBE_HAZELCAST_REMOTE_ADDRESS
value: "zeebe-hazelcast-staging"
- name: ZEEBE_BROKER_DATA_SNAPSHOTPERIOD
value: "15m"
- name: ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK
value: "0.85"
- name: ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK
value: "0.95"
- name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_WRITE_BUFFER_SIZE
value: "8MB"
- name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_MAX_WRITE_BUFFER_SIZE_TO_MAINTAIN
value: "16MB"
- name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_MAX_OPEN_FILES
value: "1024"
resources:
requests:
cpu: 800m
memory: 1200Mi
limits:
cpu: 960m
memory: 1920Mi
zeebe-gateway:
image:
repository: camunda/zeebe
tag: 8.5.0
nodeSelector:
node-role.kubernetes.io/zeebe: "true"
resources:
requests:
cpu: 400m
memory: 450Mi
limits:
cpu: 400m
memory: 450Mi