Helm Camunda 8 with hazelcast exporter

Hi,
i’m trying to install camunda 8 with hazelcast exporter via helm chart. I’m following the guide from the documentation, but it seems to be ineffective. This is the values.yaml i’m using:


global:
  identity:
    auth:
      enabled: false

identity:
  enabled: false

optimize:
  enabled: false

connectors:
  enabled: false

operate:
  enabled: false

tasklist:
  enabled: false

zeebe:
  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1
  pvcSize: 10Gi

zeebe-gateway:
  replicas: 1

elasticsearch:
  replicas: 1

extraInitContainers:
  - name: init-exporters-hazelcast
    image: busybox:1.35
    command: ["/bin/sh", "-c"]
    args:
      [
        "wget --no-check-certificate https://repo1.maven.org/maven2/io/zeebe/hazelcast/zeebe-hazelcast-exporter/1.4.0/zeebe-hazelcast-exporter-1.4.0-jar-with-dependencies.jar -O /exporters/zeebe-hazelcast-exporter.jar; ls -al /exporters",
      ]
    volumeMounts:
      - name: exporters
        mountPath: /exporters/
env:
  - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH
    value: /exporters/zeebe-hazelcast-exporter.jar
  - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME
    value: io.zeebe.hazelcast.exporter.HazelcastExporter
  - name: ZEEBE_HAZELCAST_REMOTE_ADDRESS
    value: "{{ .Release.Name }}-hazelcast"

But describing zebee pod i can’t see the exporter env variables:

    Environment:
      LC_ALL:                                                  C.UTF-8
      K8S_NAME:                                                camunda-zeebe-0 (v1:metadata.name)
      K8S_SERVICE_NAME:                                        camunda-zeebe
      K8S_NAMESPACE:                                           default (v1:metadata.namespace)
      ZEEBE_BROKER_NETWORK_ADVERTISEDHOST:                     $(K8S_NAME).$(K8S_SERVICE_NAME).$(K8S_NAMESPACE).svc
      ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS:               $(K8S_SERVICE_NAME)-0.$(K8S_SERVICE_NAME).$(K8S_NAMESPACE).svc:26502,
      ZEEBE_BROKER_CLUSTER_CLUSTERNAME:                        camunda-zeebe
      ZEEBE_LOG_LEVEL:                                         info
      ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT:                    1
      ZEEBE_BROKER_CLUSTER_CLUSTERSIZE:                        1
      ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR:                  1
      ZEEBE_BROKER_THREADS_CPUTHREADCOUNT:                     3
      ZEEBE_BROKER_THREADS_IOTHREADCOUNT:                      3
      ZEEBE_BROKER_GATEWAY_ENABLE:                             false
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME:          io.camunda.zeebe.exporter.ElasticsearchExporter
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL:           http://camunda-elasticsearch:9200
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_PREFIX:  zeebe-record
      ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT:                    26501
      ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT:                   26502
      ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT:                 9600
      K8S_POD_NAME:                                            camunda-zeebe-0 (v1:metadata.name)
      JAVA_TOOL_OPTIONS:                                       -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/zeebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
      ZEEBE_BROKER_DATA_SNAPSHOTPERIOD:                        5m
      ZEEBE_BROKER_DATA_DISK_FREESPACE_REPLICATION:            2GB
      ZEEBE_BROKER_DATA_DISK_FREESPACE_PROCESSING:             3GB

What i’m doing wrong? Can you please help me?

Hi @gmosti
Could you please explain what you mean by “but it seems to be ineffective.”? Do you see some errors during startup? Have you checked the Note in the documentation regarding pods with a read-only root file system?

Regards,
Alex

Hi @gmosti - the extraInitContainers and env sections at the bottom of your values.yaml file need to be under the zeebe key. See here and here for reference!

I think the documentation could be improved. Do you have any suggestions on what we could add to the docs to clarify this? Do you think just having zeebe: at the top of the code block would be enough?

1 Like

Hi, thanks for your responses.
@Alex_Voloshyn exporters are not under /exporters and variables are not setted but the reaseon is the one explained by @nathan.loding

@nathan.loding yes i think it could help, for the rest seems clear to me. I’ve corrected the yaml but the zeebe pod doesn’t start:
camunda-zeebe-0 0/1 Init:CreateContainerConfigError 0 103s

This the last events describing the pod :

Events:
  Type     Reason                  Age               From                     Message
  ----     ------                  ----              ----                     -------
  Normal   Scheduled               57s               default-scheduler        Successfully assigned default/camunda-zeebe-0 to 10.0.10.225
  Normal   SuccessfulAttachVolume  35s               attachdetach-controller  AttachVolume.Attach succeeded for volume "csi-32dbacf8-beaf-453c-a5da-11a2d1301614"
  Normal   Pulled                  9s (x3 over 23s)  kubelet                  Container image "busybox:1.35" already present on machine
  Warning  Failed                  9s (x3 over 23s)  kubelet                  Error: container has runAsNonRoot and image will run as root (pod: "camunda-zeebe-0_default(b3aa2512-829f-424d-9d63-977a1c2a1e2f)", container: init-exporters-hazelcast)

Here the yaml i’m using now, i’ve inserted imagePullPolicy :

# Specify the image repository and tag for Camunda
image:
  repository: camunda/camunda-bpm-platform
  tag: latest
  
global:
  identity:
    auth:
      enabled: false

identity:
  enabled: false

optimize:
  enabled: false

connectors:
  enabled: false

operate:
  enabled: false

tasklist:
  enabled: false

zeebe:
  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1
  pvcSize: 10Gi
  extraInitContainers:
    - name: init-exporters-hazelcast
      image: busybox:1.35
      imagePullPolicy: Always
      command: ["/bin/sh", "-c"]
      args:
        [
          "wget --no-check-certificate https://repo1.maven.org/maven2/io/zeebe/hazelcast/zeebe-hazelcast-exporter/1.4.0/zeebe-hazelcast-exporter-1.4.0-jar-with-dependencies.jar -O /exporters/zeebe-hazelcast-exporter.jar; ls -al /exporters",
        ]
      volumeMounts:
        - name: exporters
          mountPath: /exporters/   
  env:
    - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH
      value: /exporters/zeebe-hazelcast-exporter.jar
    - name: ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME
      value: io.zeebe.hazelcast.exporter.HazelcastExporter
    - name: ZEEBE_HAZELCAST_REMOTE_ADDRESS
      value: "{{ .Release.Name }}-hazelcast"

Have you some suggestions?

Hi @gmosti - the events don’t show anything helpful here unfortunately. Can you also share the log?

Here the complete describe pod:

Name:             camunda-zeebe-0
Namespace:        default
Priority:         0
Service Account:  default
Node:             10.0.10.225/10.0.10.225
Start Time:       Fri, 23 Feb 2024 11:22:21 +0100
Labels:           app=camunda-platform
                  app.kubernetes.io/component=zeebe-broker
                  app.kubernetes.io/instance=camunda
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=camunda-platform
                  app.kubernetes.io/part-of=camunda-platform
                  app.kubernetes.io/version=latest
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=camunda-zeebe-6589cbcf5c
                  helm.sh/chart=camunda-platform-9.1.0
                  statefulset.kubernetes.io/pod-name=camunda-zeebe-0
Annotations:      <none>
Status:           Pending
IP:               10.0.10.71
IPs:
  IP:           10.0.10.71
Controlled By:  StatefulSet/camunda-zeebe
Init Containers:
  init-exporters-hazelcast:
    Container ID:
    Image:         busybox:1.35
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -c
    Args:
      wget --no-check-certificate https://repo1.maven.org/maven2/io/zeebe/hazelc
ast/zeebe-hazelcast-exporter/1.4.0/zeebe-hazelcast-exporter-1.4.0-jar-with-depen
dencies.jar -O /exporters/zeebe-hazelcast-exporter.jar; ls -al /exporters
    State:          Waiting
      Reason:       CreateContainerConfigError
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /exporters/ from exporters (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b9qfv (
ro)
Containers:
  zeebe:
    Container ID:
    Image:          camunda/zeebe:8.4.1
    Image ID:
    Ports:          9600/TCP, 26501/TCP, 26502/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     960m
      memory:  1920Mi
    Requests:
      cpu:      800m
      memory:   1200Mi
    Readiness:  http-get http://:9600/ready delay=30s timeout=1s period=30s #suc
cess=1 #failure=5
    Environment:
      LC_ALL:                                                  C.UTF-8
      K8S_NAME:                                                camunda-zeebe-0 (
v1:metadata.name)
      K8S_SERVICE_NAME:                                        camunda-zeebe
      K8S_NAMESPACE:                                           default (v1:metad
ata.namespace)
      ZEEBE_BROKER_NETWORK_ADVERTISEDHOST:                     $(K8S_NAME).$(K8S
_SERVICE_NAME).$(K8S_NAMESPACE).svc
      ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS:               $(K8S_SERVICE_NAM
E)-0.$(K8S_SERVICE_NAME).$(K8S_NAMESPACE).svc:26502,
      ZEEBE_BROKER_CLUSTER_CLUSTERNAME:                        camunda-zeebe
      ZEEBE_LOG_LEVEL:                                         info
      ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT:                    1
      ZEEBE_BROKER_CLUSTER_CLUSTERSIZE:                        1
      ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR:                  1
      ZEEBE_BROKER_THREADS_CPUTHREADCOUNT:                     3
      ZEEBE_BROKER_THREADS_IOTHREADCOUNT:                      3
      ZEEBE_BROKER_GATEWAY_ENABLE:                             false
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME:          io.camunda.zeebe.
exporter.ElasticsearchExporter
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL:           http://camunda-el
asticsearch:9200
      ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_INDEX_PREFIX:  zeebe-record
      ZEEBE_BROKER_NETWORK_COMMANDAPI_PORT:                    26501
      ZEEBE_BROKER_NETWORK_INTERNALAPI_PORT:                   26502
      ZEEBE_BROKER_NETWORK_MONITORINGAPI_PORT:                 9600
      K8S_POD_NAME:                                            camunda-zeebe-0 (
v1:metadata.name)
      JAVA_TOOL_OPTIONS:                                       -XX:+HeapDumpOnOu
tOfMemoryError -XX:HeapDumpPath=/usr/local/zeebe/data -XX:ErrorFile=/usr/local/z
eebe/data/zeebe_error%p.log -XX:+ExitOnOutOfMemoryError
      ZEEBE_BROKER_EXPORTERS_HAZELCAST_JARPATH:                /exporters/zeebe-
hazelcast-exporter.jar
      ZEEBE_BROKER_EXPORTERS_HAZELCAST_CLASSNAME:              io.zeebe.hazelcas
t.exporter.HazelcastExporter
      ZEEBE_HAZELCAST_REMOTE_ADDRESS:                          camunda-hazelcast
    Mounts:
      /exporters from exporters (rw)
      /tmp from tmp (rw)
      /usr/local/bin/startup.sh from config (rw,path="startup.sh")
      /usr/local/zeebe/data from data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b9qfv (
ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in
 the same namespace)
    ClaimName:  data-camunda-zeebe-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      camunda-zeebe
    Optional:  false
  exporters:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-b9qfv:
    Type:                    Projected (a volume that contains injected data fro
m multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists fo
r 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists
for 300s
Events:
  Type    Reason  Age                      From     Message
  ----    ------  ----                     ----     -------
  Normal  Pulled  50s (x19641 over 2d22h)  kubelet  Container image "busybox:1.3
5" already present on machine

I’ve no result in logs :


kubectl logs -f camunda-zeebe-0 -c init-exporters-hazelcast
Error from server (BadRequest): container "init-exporters-hazelcast" in pod "camunda-zeebe-0" is waiting to start: CreateContainerConfigError
kubectl logs -f camunda-zeebe-0
Defaulted container "zeebe" out of: zeebe, init-exporters-hazelcast (init)
Error from server (BadRequest): container "zeebe" in pod "camunda-zeebe-0" is waiting to start: PodInitializing

@gmosti - that’s strange, I was expecting something in the logs! If you remove the exporter (remove the extraInitContainers and env sections you added), does Zeebe start normally?

yes the pod is starting correctly

@gmosti What if you replace extraInitContainers with initContainers?

Hi @gmosti - I just noticed that you specified an image repository with the value of camunda/camunda-bpm-platform … that repository has images for Camunda 7, I think. I think the Camunda 8 repository is camunda/camunda-platform.

The image: section should not be needed if you are using Camunda’s hosted images. That option is available for customers who have their own private registry that they maintain.

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.