Zeebe cluster helm chart setup support

Hi guys,
I get into some trouble with installing Zeebe cluster following Zeebe-helm-chart guideline.
Firstly, I tried to install zeebe cluster with the dev profile in https://github.com/zeebe-io/zeebe-helm-profiles but it didn’t work. I got some errors as follows:

getpod .
Even when Zeebe is in ready I still get this error:

Secondly, I want to install Zeebe cluster on 3 servers with separate resources, how can I setup it? I watched @salaboy 's tutorial video, it seems that 3 brokers are on the same environment. I wonder whether I can deploy them on many servers or I have to merge resources of three servers into one and setup on that. Please give me some instructions, by the way I’m quite new to Kubernetes. Thanks a lot!

hi @Niklaus thanks for trying out the helm charts. Can you share a bit more details about where do you have your Kubernetes cluster installed? Is this an On-Prem installation?
We have seen issues with services starting… as you can see the problem in your installation is ElasticSearch not booting up so you might need to take a look at what is happening there. ElasticSearch and the Zeebe Brokers will use volumes, so if you are on an On-Prem installation you need to make sure that your Kubernetes Storage providers are set up correctly for that chart to work.
Can you share the logs for both ElasticSearch and the Zeebe Brokers?

If you are using Kubernetes, Kubernetes will decide where to run the services, no matter how many servers (Kubernetes Nodes) you have.

HTH

Thanks for your prompt reply. Yeah it’s an On-Premise installation. I’m using Kubernetes Kind to create cluster. I found out that when I changed the option “–values x.yaml” into “-k kind_values.yaml” then it worked. Maybe it’s because I’m using kubernetes kind to create kubernetes cluster(I’m not sure but it worked).
Currently, we have 3 servers (let’s say 192.168.26.99, 192.168.26.100, 192.168.26.101), I want to setup Zeebe-cluster and distribute workload to 3 of them. Can you give me some suggestions or an example on how to config. I see in Helm-chart config https://github.com/zeebe-io/zeebe-helm-profiles/raw/master/zeebe-core-team.yaml, I only need to config the number of clusters then automatically 3 brokers are spawned,… I don’t know how to deal with traffic and communication when deploying them on different servers, the contact points or something else. We used to work with zeebe-cluster-docker-compose.yml for developing but now we want to deploy it for production environment. With zeebe docker-compose version, I used to try to deploy 3 zeebe brokers on 3 servers and routing contact points to each other but problems of communication, grpc, and raftserver were thrown and I think Helm chart with K8s will help us handle those. Can you show me how to setup the Zeebe-cluster correctly? I will need to build 3 nodes on 3 servers or something like that. As I have noted, I’m new to K8s and I’m learning more about it.

@Niklaus great that it worked with KIND.
If you have 3 servers, then you need to install Kubernetes in those 3 servers. Helm then will use that kubernetes instance, and the kubernetes scheduler will deploy your workloads as it see fit.
I would recommend you to read more about Kubernetes, as it seems that you have some specific requirements that I am not fully understanding. Once you manage and understand Kubernetes Basics, Helm will just help you to install things, but not to define where the services will run, as that abstraction is one of the main Kubernetes features.

HTH

1 Like

@salaboy Thanks for the information. It’s very helpful. I have read all about Kubernetes Basics and a lot of things are clearer. But there are a few things I don’t still understand:

  1. Back to the basic Zeebe cluster topo:

    I want to ask whether it is possible to setup like this:
  • suppose we have 3 nodes (instead of 5 as in the figure)
  • Zeebe cluster theory:

Zeebe can operate as a cluster of brokers, forming a peer-to-peer network. In this network, all brokers have the same responsibilities and there is no single point of failure.

  • I have 3 servers, let say: 192.168.26.99, 192.168.26.100, 192.168.26.101, then following the theory can I distribute 3 brokes to 3 servers?
    Here is how I setup before: from your sample docker-compose: zeebe-cluster-docker-compose
    I created 3 docker-compose files and deploy 3 Zeebes on 3 servers. They look like as follows:

server 1 (192.168.26.99):


networks:
  zeebe_network:
    driver: bridge

services:
  node0:
    container_name: zeebe_broker_1
    image: camunda/zeebe:0.23.1
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_NODE_ID=0
      - ZEEBE_PARTITIONS_COUNT=2
      - ZEEBE_REPLICATION_FACTOR=3
      - ZEEBE_CLUSTER_SIZE=3
      - ZEEBE_CONTACT_POINTS=node0:26502
    ports:
      - "26500:26500"
    # volumes:
    #   - ./zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
    networks:
      - zeebe_network

server 2 (192.168.26.100):


networks:
  zeebe_network:
    driver: bridge

services:
  node1:
    container_name: zeebe_broker_2
    image: camunda/zeebe:0.23.1
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_NODE_ID=0
      - ZEEBE_PARTITIONS_COUNT=2
      - ZEEBE_REPLICATION_FACTOR=3
      - ZEEBE_CLUSTER_SIZE=3
      - ZEEBE_CONTACT_POINTS=192.168.26.99:26502
    ports:
      - "26500:26500"
    # volumes:
    #   - ./zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
    networks:
      - zeebe_network

server 3 (192.168.26.101):


networks:
  zeebe_network:
    driver: bridge

services:
  node2:
    container_name: zeebe_broker_3
    image: camunda/zeebe:0.23.1
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_NODE_ID=0
      - ZEEBE_PARTITIONS_COUNT=2
      - ZEEBE_REPLICATION_FACTOR=3
      - ZEEBE_CLUSTER_SIZE=3
      - ZEEBE_CONTACT_POINTS=192.168.26.99:26502
    ports:
      - "26500:26500"
    # volumes:
    #   - ./zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
    networks:
      - zeebe_network

However, it not work. Connection errors occured. Is my setting up correct?

  1. When using helm chart, I config clusterSize: 3 then it creates one service Zeebe and 3 contanerized zeebe apps (3 pods). Is each pod here an independent zeebe container with 3 brokers or each is a zeebe container with 1 broker then 3 brokers (3 pods) routing to one another via ZEEBE_CONTACT_POINTS. clusterSize means 3 brokers, 3 replicas of zeebe container app when deploying?
  2. Can you give more details about how I can install zeebe cluster after I have installed Kubernetes on 3 server? (I will create 3 K8s cluster on each sever and install zeebe cluster there and make them connected??? - I’m not experienced with K8s so…)

@Niklaus you are still confusing a lot of K8s basics…
The whole point of using K8s is managing your 3 servers as a single machine. So you don’t create a Kubernetes cluster in each of your servers… you just have a single Kubernetes cluster that spans to 3 Kubernetes nodes (which are your servers).
Then you install the Zeebe Cluster with HELM, if you configure it for 3 brokers, that creates 3 PODs and Kubernetes will decide in which servers to place those pods. Kubernetes Services are logical load balancers, the main idea is to point your applications to this logical load balancer, so you don’t need to know the IP addresses of your pods that can be moving around the cluster, as kubernetes see fit.

HTH

1 Like

For the docker-compose example, you need to use the external IP interface addresses to connect, not the internal docker network IP on the machine. That address is not reachable from other machines.

1 Like

Do you have a link to a good getting started resource for understanding K8s?

This one is quite good: https://medium.com/google-cloud/kubernetes-101-pods-nodes-containers-and-clusters-c1509e409e16

1 Like

@jwulf It’s a great note. Please take a look at my configs:
server 1

version: "2"
 
networks:
  zeebe_network:
    driver: bridge
 
services:
  node0:
    container_name: zeebe_broker_1
    image:  harbor.viettelcyber.com/sirc/soar/camunda/zeebe:0.22.5
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_NODE_ID=0
      - ZEEBE_PARTITIONS_COUNT=2
      - ZEEBE_REPLICATION_FACTOR=3
      - ZEEBE_CLUSTER_SIZE=3
      - ZEEBE_CONTACT_POINTS=node0:26502
      - ZEEBE_GATEWAY_SECURITY_ENABLED=false
    ports:
      - "8500:26500"
      - "8502:26502"
    volumes:
      - ./data/zeebe_0.22.5_b1:/usr/local/zeebe/data # mount each broker data to a separate location
      #- /opt/soar/workflow_engine/configs/oren/zeebe/zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
      #- /home/sf/zeebe/zeebe_config/zeebe-kafka-exporter-1.2.0-SNAPSHOT-uber.jar:/usr/local/zeebe/lib/zeebe-kafka-exporter-1.2.0-SNAPSHOT-uber.jar
    extra_hosts:
      - kafka01:192.168.26.100
      - es.oa.soar.com:192.168.26.100
    mem_limit: 12g
    logging:
      options:
        max-size: "200m"
        max-file: "5"
      driver: "json-file"
    restart: always
    network_mode: "host"
    #networks:
     # - zeebe_network

server 2

 
networks:
  zeebe_network:
    driver: bridge
 
services:
  node1:
    container_name: zeebe_broker_2
    image:  harbor.viettelcyber.com/sirc/soar/camunda/zeebe:0.22.5
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_NODE_ID=1
      - ZEEBE_PARTITIONS_COUNT=2
      - ZEEBE_REPLICATION_FACTOR=3
      - ZEEBE_CLUSTER_SIZE=3
      - ZEEBE_CONTACT_POINTS=192.168.26.100:8502
      - ZEEBE_GATEWAY_SECURITY_ENABLED=false
    ports:
      - "8500:26500"
      - "8502:26502"
    volumes:
      - ./data/zeebe_0.22.5_b2:/usr/local/zeebe/data # mount each broker data to a separate location
      #- /opt/soar/workflow_engine/configs/oren/zeebe/zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
      #- /home/sf/zeebe/zeebe_config/zeebe-kafka-exporter-1.2.0-SNAPSHOT-uber.jar:/usr/local/zeebe/lib/zeebe-kafka-exporter-1.2.0-SNAPSHOT-uber.jar
    extra_hosts:
      - kafka01:192.168.26.101
      - es.oa.soar.com:192.168.26.101
    mem_limit: 12g
    logging:
      options:
        max-size: "200m"
        max-file: "5"
      driver: "json-file"
    restart: always
    network_mode: "host"
    #networks:
      #- zeebe_network

server 3

version: "2"
 
networks:
  zeebe_network:
    driver: bridge
 
services:
  node2:
    container_name: zeebe_broker_3
    image:  harbor.viettelcyber.com/sirc/soar/camunda/zeebe:0.22.5
    environment:
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_NODE_ID=2
      - ZEEBE_PARTITIONS_COUNT=2
      - ZEEBE_REPLICATION_FACTOR=3
      - ZEEBE_CLUSTER_SIZE=3
      - ZEEBE_CONTACT_POINTS=192.168.26.100:8502
      - ZEEBE_GATEWAY_SECURITY_ENABLED=false
    ports:
      - "8500:26500"
      - "8502:26502"
    volumes:
      - ./data/zeebe_0.22.5_b2:/usr/local/zeebe/data # mount each broker data to a separate location
      #- /opt/soar/workflow_engine/configs/oren/zeebe/zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml
      #- /home/sf/zeebe/zeebe_config/zeebe-kafka-exporter-1.2.0-SNAPSHOT-uber.jar:/usr/local/zeebe/lib/zeebe-kafka-exporter-1.2.0-SNAPSHOT-uber.jar
    extra_hosts:
      - kafka01:192.168.26.102
      - es.oa.soar.com:192.168.26.102
    mem_limit: 12g
    logging:
      options:
        max-size: "200m"
        max-file: "5"
      driver: "json-file"
    restart: always
    network_mode: "host"
    #networks:
     # - zeebe_network

Here are container env when running:


Server ip info:

They are connected to one another:
connected (1)
I tried many different way but they seem not to understand one another and connection errors were thrown. As you said is there any env variable config for them to be connected? Do I need to advertise zeebe host? It’ll be nice if you give a brief example.

@salaboy Yeah I read more about K8s and understand what you mean. I’m installing k8s cluster on 3 servers, pick one to be Master the rest nodes will join in. I create a cluster then use zeebe helm to install Zeebe on that then K8s will handle the rest. It’s what I understand. Is that right?

@Niklaus that’s pretty much it

1 Like

@jwulf thanks for your suggestion, I made docker-compose version work.