Elasticsearch errors with Camunda chart

Hi people! We recently update some ours camunda clusters (we only use zeebe + elasticsearch + zeebe-gw) with the Camunda Chart. We use hazelcast too, but outside of this chart.

We have right now in two k8s clusters working fine all the componens (one of them is in Google Cloud Platform [GKE] and the other one is a bare-metal k8s cluster).

But the problem is in our old infrastructure on premise (bare-metal cluster too) that all the components works fine but the elasticsearch never create the cluster fine and allways is restarting. We have defined the HR yaml exactly in the other new one bare-metal cluster, but we don’t no why this never good formed the elasticsearch cluster with the two nodes in this old infrastructure (we have here 3 enviroments).

  • In the past with the outdated YAML worked fine, but right now this fails all the time.

I attached the logs if someone can help me, becasuse we are in a pooint that we don’t know what we can do :frowning:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: zeebe
  namespace: develop
spec:
  releaseName: zeebe
  interval: 5m
  chart:
    spec:
      chart: camunda-platform
      version: 10.0.2
      sourceRef:
        kind: HelmRepository
        name: camunda
        namespace: default
  values:
    global:
      zeebeClusterName: zeebe-develop
      identity:
        auth:
          enabled: false
    elasticsearch:
      enabled: true
      nodeSelector:
        node-role.kubernetes.io/zeebe: "true"
      resources:
        requests:
          cpu: 1
          memory: 2Gi
        limits:
          cpu: 2
          memory: 2Gi
      volumeClaimTemplate:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 32Gi
      minimumMasterNodes: 1
      clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"
    connectors:
      enabled: false
    identity:
      enabled: false
    identityKeycloak:
      enabled: false
    identityPostgresql:
      enabled: false
    optimize: 
      enabled: false
    operate: 
      enabled: false
    tasklist: 
      enabled: false
    zeebe:
      enabled: true
      debug: true
      clusterSize: 3
      replicationFactor: 3
      partitionCount: 5
      image:
        repository: camunda/zeebe
        tag: 8.5.0
      nodeSelector:
        node-role.kubernetes.io/zeebe: "true"
      podSecurityContext:
        runAsNonRoot: false
      containerSecurityContext:
        readOnlyRootFilesystem: false
        runAsNonRoot: false
      # initContainers:
      #   - name: init-exporters-hazelcast
      #     image: busybox:1.35
      #     command: ["/bin/sh", "-c"]
      #     args:
      #       [
      #         "wget --no-check-certificate https://repo1.maven.org/maven2/io/zeebe/hazelcast/zeebe-hazelcast-exporter/1.4.0/zeebe-hazelcast-exporter-1.4.0-jar-with-dependencies.jar -O /exporters/zeebe-hazelcast-exporter.jar; ls -al /exporters",
      #       ]
      #     volumeMounts:
      #       - name: exporters
      #         mountPath: /exporters/
      env:
        # - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH
        #   value: /exporters/zeebe-hazelcast-exporter.jar
        # - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME
        #   value: io.zeebe.hazelcast.exporter.HazelcastExporter
        - name: ZEEBE_HAZELCAST_PORT
          value: "5701"
        - name: ZEEBE_HAZELCAST_CLUSTER_NAME
          value: "zeebe-hazelcast-develop"
        - name: HZ_NETWORK_RESTAPI_ENABLED
          value: "true"
        - name: ZEEBE_HAZELCAST_SECURITY_ENABLED
          value: "false"
        - name: ZEEBE_HAZELCAST_REMOTE_ADDRESS
          value: "zeebe-hazelcast-develop:5701"
        - name: ZEEBE_BROKER_DATA_SNAPSHOTPERIOD
          value: "15m"
        - name: ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK
          value: "0.85"
        - name: ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK
          value: "0.95"
        - name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_WRITE_BUFFER_SIZE
          value: "8MB"
        - name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_MAX_WRITE_BUFFER_SIZE_TO_MAINTAIN
          value: "16MB"
        - name: ZEEBE_BROKER_DATA_ROCKSDB_COLUMNFAMILYOPTIONS_MAX_OPEN_FILES
          value: "1024"
      resources:
        requests:
          cpu: 800m
          memory: 1200Mi
        limits:
          cpu: 960m
          memory: 1920Mi
    zeebe-gateway:
      image:
        repository: camunda/zeebe
        tag: 8.5.0
      nodeSelector:
        node-role.kubernetes.io/zeebe: "true"
      resources:
        requests:
          cpu: 400m
          memory: 450Mi
        limits:
          cpu: 400m
          memory: 450Mi

2024-05-16T06:47:57,558][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [unsigned-long]
[2024-05-16T06:47:57,558][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [x-pack-sql]
[2024-05-16T06:47:57,558][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [runtime-fields-common]
[2024-05-16T06:47:57,559][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [x-pack-async]
[2024-05-16T06:47:57,559][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [vector-tile]
[2024-05-16T06:47:57,560][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [lang-expression]
[2024-05-16T06:47:57,560][INFO ][o.e.p.PluginsService ] [zeebe-elasticsearch-master-0] loaded module [x-pack-eql]
[2024-05-16T06:47:59,916][INFO ][o.e.e.NodeEnvironment ] [zeebe-elasticsearch-master-0] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/mapper/ubuntu--vg-ubuntu--lv)]], net usable_space [13.8gb], net total_space [77.5gb], types [ext4]
[2024-05-16T06:47:59,916][INFO ][o.e.e.NodeEnvironment ] [zeebe-elasticsearch-master-0] heap size [1gb], compressed ordinary object pointers [true]
[2024-05-16T06:48:00,033][INFO ][o.e.n.Node ] [zeebe-elasticsearch-master-0] node name [zeebe-elasticsearch-master-0], node ID [v-0UTcrRRz2Md4BDpBX6FA], cluster name [elastic], roles [ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold]
[2024-05-16T06:48:11,055][INFO ][o.e.f.FeatureService ] [zeebe-elasticsearch-master-0] Registered local node features [features_supported, health.dsl.info, usage.data_tiers.precalculate_stats]
[2024-05-16T06:48:12,052][INFO ][o.e.t.a.APM ] [zeebe-elasticsearch-master-0] Sending apm metrics is disabled
[2024-05-16T06:48:12,052][INFO ][o.e.t.a.APM ] [zeebe-elasticsearch-master-0] Sending apm traces is disabled
[2024-05-16T06:48:12,100][INFO ][o.e.x.s.Security ] [zeebe-elasticsearch-master-0] Security is disabled
[2024-05-16T06:48:12,613][INFO ][o.e.x.w.Watcher ] [zeebe-elasticsearch-master-0] Watcher initialized components at 2024-05-16T06:48:12.612Z
[2024-05-16T06:48:12,706][INFO ][o.e.x.p.ProfilingPlugin ] [zeebe-elasticsearch-master-0] Profiling is enabled
[2024-05-16T06:48:12,735][INFO ][o.e.x.p.ProfilingPlugin ] [zeebe-elasticsearch-master-0] profiling index templates will not be installed or reinstalled
[2024-05-16T06:48:13,391][INFO ][o.e.x.a.APMPlugin ] [zeebe-elasticsearch-master-0] APM ingest plugin is disabled
[2024-05-16T06:48:14,464][INFO ][o.e.t.n.NettyAllocator ] [zeebe-elasticsearch-master-0] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=1mb, factors={es.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=4mb, heap_size=1gb}]
[2024-05-16T06:48:14,508][INFO ][o.e.i.r.RecoverySettings ] [zeebe-elasticsearch-master-0] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2024-05-16T06:48:14,590][INFO ][o.e.d.DiscoveryModule ] [zeebe-elasticsearch-master-0] using discovery type [multi-node] and seed hosts providers [settings]
[2024-05-16T06:48:17,189][INFO ][o.e.n.Node ] [zeebe-elasticsearch-master-0] initialized
[2024-05-16T06:48:17,191][INFO ][o.e.n.Node ] [zeebe-elasticsearch-master-0] starting ...
[2024-05-16T06:48:17,238][INFO ][o.e.x.s.c.f.PersistentCache] [zeebe-elasticsearch-master-0] persistent cache index loaded
[2024-05-16T06:48:17,240][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [zeebe-elasticsearch-master-0] deprecation component started
[2024-05-16T06:48:17,597][INFO ][o.e.t.TransportService ] [zeebe-elasticsearch-master-0] publish_address {zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local/192.168.29.151:9300}, bound_addresses {0.0.0.0:9300}
[2024-05-16T06:48:17,876][INFO ][o.e.b.BootstrapChecks ] [zeebe-elasticsearch-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-05-16T06:48:17,881][INFO ][o.e.c.c.ClusterBootstrapService] [zeebe-elasticsearch-master-0] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1]
[2024-05-16T06:48:27,896][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:48:37,898][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:48:47,900][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:48:57,902][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:07,904][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:17,906][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:27,908][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:37,910][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:47,912][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:49:57,915][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:07,916][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:17,918][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:27,920][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:37,923][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:47,927][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-05-16T06:50:57,930][WARN ][o.e.c.c.ClusterFormationFailureHelper] [zeebe-elasticsearch-master-0] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [zeebe-elasticsearch-master-0, zeebe-elasticsearch-master-1] to bootstrap a cluster: have discovered [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}]; discovery will continue using [192.168.98.5:9300] from hosts providers and [{zeebe-elasticsearch-master-0}{v-0UTcrRRz2Md4BDpBX6FA}{itvwBoO4TGS1F1HmLGodlA}{zeebe-elasticsearch-master-0}{zeebe-elasticsearch-master-0.zeebe-elasticsearch-master-hl.develop.svc.cluster.local}{192.168.29.151:9300}{cdfhilmrstw}{8.12.2}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html

K8S Version: v1.19.16
OS: Ubuntu 18.04.4 LTS
Container Runtime: 20.10.21