Hi people! We recently update some ours camunda clusters (we only use zeebe + elasticsearch + zeebe-gw) with the Camunda Chart. We use hazelcast too, but outside of this chart.
We have right now in two k8s clusters working fine all the componens (one of them is in Google Cloud Platform [GKE] and the other one is a bare-metal k8s cluster).
But the problem is in our old infrastructure on premise (bare-metal cluster too) that all the components works fine but the elasticsearch never create the cluster fine and allways is restarting. We have defined the HR yaml exactly in the other new one bare-metal cluster, but we don’t no why this never good formed the elasticsearch cluster with the two nodes in this old infrastructure (we have here 3 enviroments).
- In the past with the outdated YAML worked fine, but right now this fails all the time.
I attached the logs if someone can help me, becasuse we are in a pooint that we don’t know what we can do 
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: zeebe
namespace: develop
spec:
releaseName: zeebe
interval: 5m
chart:
spec:
chart: camunda-platform
version: 10.0.2
sourceRef:
kind: HelmRepository
name: camunda
namespace: default
values:
global:
zeebeClusterName: zeebe-develop
identity:
auth:
enabled: false
elasticsearch:
enabled: true
nodeSelector:
node-role.kubernetes.io/zeebe: "true"
resources:
requests:
cpu: 1
memory: 2Gi
limits:
cpu: 2
memory: 2Gi
volumeClaimTemplate:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 32Gi
minimumMasterNodes: 1
clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"
connectors:
enabled: false
identity:
enabled: false
identityKeycloak:
enabled: false
identityPostgresql:
enabled: false
optimize:
enabled: false
operate:
enabled: false
tasklist:
enabled: false
zeebe:
enabled: true
debug: true
clusterSize: 3
replicationFactor: 3
partitionCount: 5
image:
repository: camunda/zeebe
tag: 8.5.0
nodeSelector:
node-role.kubernetes.io/zeebe: "true"
podSecurityContext:
runAsNonRoot: false
containerSecurityContext:
readOnlyRootFilesystem: false
runAsNonRoot: false
# initContainers:
# - name: init-exporters-hazelcast
# image: busybox:1.35
# command: ["/bin/sh", "-c"]
# args:
# [
# "wget --no-check-certificate https://repo1.maven.org/maven2/io/zeebe/hazelcast/zeebe-hazelcast-exporter/1.4.0/zeebe-hazelcast-exporter-1.4.0-jar-with-dependencies.jar -O /exporters/zeebe-hazelcast-exporter.jar; ls -al /exporters",
# ]
# volumeMounts:
# - name: exporters
# mountPath: /exporters/
env:
# - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH
# value: /exporters/zeebe-hazelcast-exporter.jar
# - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME
# value: io.zeebe.hazelcast.exporter.HazelcastExporter
- name: ZEEBE_HAZELCAST_PORT
value: "5701"
- name: ZEEBE_HAZELCAST_CLUSTER_NAME
value: "zeebe-hazelcast-develop"
- name: HZ_NETWORK_RESTAPI_ENABLED
value: "true"
- name: ZEEBE_HAZELCAST_SECURITY_ENABLED
value: "false"
- name: ZEEBE_HAZELCAST_REMOTE_ADDRESS
value: "zeebe-hazelcast-develop:5701"
- name: ZEEBE_BROKER_DATA_SNAPSHOTPERIOD
value: "15m"
- name: ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK
value: "0.85"
- name: ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK
value: "0.95"
- name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_WRITE_BUFFER_SIZE
value: "8MB"
- name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_MAX_WRITE_BUFFER_SIZE_TO_MAINTAIN
value: "16MB"
- name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_MAX_OPEN_FILES
value: "1024"
resources:
requests:
cpu: 800m
memory: 1200Mi
limits:
cpu: 960m
memory: 1920Mi
zeebe-gateway:
image:
repository: camunda/zeebe
tag: 8.5.0
nodeSelector:
node-role.kubernetes.io/zeebe: "true"
resources:
requests:
cpu: 400m
memory: 450Mi
limits:
cpu: 400m
memory: 450Mi
2024-05-16T06:47:57,558][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [unsigned-long]
[2024-05-16T06:47:57,558][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [x-pack-sql]
[2024-05-16T06:47:57,558][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [runtime-fields-common]
[2024-05-16T06:47:57,559][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [x-pack-async]
[2024-05-16T06:47:57,559][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [vector-tile]
[2024-05-16T06:47:57,560][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [lang-expression]
[2024-05-16T06:47:57,560][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [x-pack-eql]
[2024-05-16T06:47:59,916][INFO ][o.e.e.NodeEnvironment ] [zeebe-elasticsearch-master-0] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/mapper/ubuntu--vg-ubuntu--lv)]], net usable_space [13.8gb], net total_space [77.5gb], types [ext4]
[2024-05-16T06:47:59,916][INFO ][o.e.e.NodeEnvironment ] [zeebe-elasticsearch-master-0] heap size [1gb], compressed ordinary object pointers [true]
[2024-05-16T06:48:00,033][INFO ][o.e.n.Node ] [zeebe-elasticsearch-master-0] node name [zeebe-elasticsearch-master-0], node ID [v-0UTcrRRz2Md4BDpBX6FA], cluster name [elastic], roles [ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold]
[2024-05-16T06:48:11,055][INFO ][o.e.f.FeatureService ] [zeebe-elasticsearch-master-0] Registered local node features [features_supported, health.dsl.info, usage.data_tiers.precalculate_stats]
[2024-05-16T06:48:12,052][INFO ][o.e.t.a.APM ] [zeebe-elasticsearch-master-0] Sending apm metrics is disabled
[2024-05-16T06:48:12,052][INFO ][o.e.t.a.APM ] [zeebe-elasticsearch-master-0] Sending apm traces is disabled
[2024-05-16T06:48:12,100][INFO ][o.e.x.s.Security ] [zeebe-elasticsearch-master-0] Security is disabled
[2024-05-16T06:48:12,613][INFO ][o.e.x.w.Watcher ] [zeebe-elasticsearch-master-0] Watcher initialized components at 2024-05-16T06:48:12.612Z
[2024-05-16T06:48:12,706][INFO ][o.e.x.p.ProfilingPlugin ] [zeebe-elasticsearch-master-0] Profiling is enabled
[2024-05-16T06:48:12,735][INFO ][o.e.x.p.ProfilingPlugin ] [zeebe-elasticsearch-master-0] profiling index templates will not be installed or reinstalled
[2024-05-16T06:48:13,391][INFO ][o.e.x.a.APMPlugin ] [zeebe-elasticsearch-master-0] APM ingest plugin is disabled
[2024-05-16T06:48:14,464][INFO ][o.e.t.n.NettyAllocator ] [zeebe-elasticsearch-master-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]
[2024-05-16T06:48:14,508][INFO ][o.e.i.r.RecoverySettings ] [zeebe-elasticsearch-master-0] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2024-05-16T06:48:14,590][INFO ][o.e.d.DiscoveryModule ] [zeebe-elasticsearch-master-0] using discovery type [multi-node] and seed hosts providers [settings]
[2024-05-16T06:48:17,189][INFO ][o.e.n.Node ] [zeebe-elasticsearch-master-0] initialized
[2024-05-16T06:48:17,191][INFO ][o.e.n.Node ] [zeebe-elasticsearch-master-0] starting ...
[2024-05-16T06:48:17,238][INFO ][o.e.x.s.c.f.PersistentCache] [zeebe-elasticsearch-master-0] persistent cache index loaded
[2024-05-16T06:48:17,240][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [zeebe-elasticsearch-master-0] deprecation component started
[2024-05-16T06:48:17,597][INFO ][o.e.t.TransportService ] [zeebe-elasticsearch-master-0] publish_address {zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local/192.168.29.151:9300}, bound_addresses {0.0.0.0:9300}
[2024-05-16T06:48:17,876][INFO ][o.e.b.BootstrapChecks ] [zeebe-elasticsearch-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-05-16T06:48:17,881][INFO ][o.e.c.c.ClusterBootstrapService] [zeebe-elasticsearch-master-0] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1]
[2024-05-16T06:48:27,896][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:48:37,898][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:48:47,900][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:48:57,902][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:07,904][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:17,906][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:27,908][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:37,910][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:47,912][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:57,915][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:07,916][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:17,918][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:27,920][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:37,923][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:47,927][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:57,930][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
K8S Version: v1.19.16
OS: Ubuntu 18.04.4 LTS
Container Runtime: 20.10.21