How are zeebe brokers connected and how does the deployment happen?

Vineet Gupta: Please clarify this question how zeebe brokers are connected how they will know about the (bpmn workflow ) deployment made to other zeeber broker that is sharing same elastic search:

  1. If i have two zeebe broker nodes(docker containers ) connected to a same elastic search. then if i deploy a bpmn workflow to broker1 then the depolyment is saved in elastic search(shared database) and then broker2 can also know about the deployement from elastic search. (I know this is the case in camunda plaform).
  2. Or both zeebe brokers are not connected by elastic search they are connected via some other method , how they are connected. (how to make two zeebe brokers (docker containers) know about the bpmn deployment that is done to other zeebe broker).

Josh Wulf: The brokers distribute deployments using the RAFT protocol.

Vineet Gupta: initialContactPoints : [ 192.168.1.22:26502, 192.168.1.32:26502 ] does i need to set initialcontactpoints so that zeebe brokers can connect.??

Vineet Gupta: I just want to know if i had three docker_compose files for three zeebe_bokers how to connect them.

Vineet Gupta: what is default initial contact point, and what this initial contact point signifies?? and what is gateway network where gateway network is used.

zell: Deployment distribution is currently not well documented, since it is more an internal thing. I tried to summarize it a bit here https://zeebe-io.github.io/zeebe-chaos/2021/01/26/deployments/

Anyway regarding your question of initial contact points. Currently it is wise to set the IP’s of all nodes on all nodes as initial contact points, which makes the bootstrap easier, and faster. The gateway needs also a contact point and will find the other brokers via gossip.

Vineet Gupta: You said it is internal thing but how to create zeebe cluster then , where nodes(brokers) are supposed to be in different machines, the only way without knowing internal thing is to use helm charts (but those helm charts are not understandable).
In helm charts it is not written that how many much big kubernetees cluster should i have to install helm

zell: Configuring the brokers and how deployments are distributed are two completely different things. Take a look at the self managed guide for docker to get more information about the configs https://docs.camunda.io/docs/self-managed/zeebe-deployment/docker/install/

zell: Additional question: What is not understandable from the helm charts? Maybe something we can improve here?

Vineet Gupta: Like some basic configurations should be documented, about pre requesites size of kubernetes cluster. what configurations to change when we increase our kubernetes cluster size.

Vineet Gupta: and what size of memory should be available on cluster nodes, as helm chart is not running on my mac it shows some memmory insufficient. I run the yml that is for local development.

zell: Maybe you take a look at the newest chart https://github.com/camunda/camunda-cloud-helm/blob/main/charts/ccsm-helm/README.md#configuration we improved the documentation a lot. We adjusted the default to match our smalles cluster configuration in SaaS.

Vineet Gupta: ok

Vineet Gupta: @zell Hi zell after reading above documents, I mainly have two doubts.
Requirement: want to setup two zeebe brokers on two different machines, and will run another broker if needed, these brokers will belong to same cluster. The cluster should export the data to elasticsearch.
Steps I am doing:

  1. Firstly i am running docker-compose up which runs the first broker in a ec2 machine , i have set the elastic search exporter in this broker.
  2. after first broker is up , i ran another docker-compose up with different configs ( elasticsearch exporter is not used, )
    Questions:
  3. did i need to use elastic search exporter on all brokers that are in cluster or have to specify in only first broker gateway connects to.
  4. does by default configuration different gateways are created for all brokers.
  5. also i want to experiment this setup in javascript, so how would i know that proccess instances are distributed between different brokers if my setup is correct.
  6. please answer these questions as i am not getting these after reading.
  7. Or this is not correct way to setup zeebe cluster, we should go with kubernetes with helm charts.

zell: This is one of the reason why we recommend to use kubernetes and our helm charts for production, everything is configured for you already and makes it easier to adjust. https://docs.camunda.io/docs/self-managed/zeebe-deployment/kubernetes/

Nevertheless, if you want to use docker compose you could also use networks and aliases to determine the contact points. https://docs.docker.com/compose/compose-file/compose-file-v2/#networks Set the name in the contact point array and they will be resolved then.

Question why do you want to use two brokers? It is wise to use an odd number of replicas, since Zeebe uses raft. See https://stackoverflow.com/a/69050416/2165134

I don’t get your second point. after first broker is up , i ran another docker-compose up with different configs ( elasticsearch exporter is not used, ) Please use the same configuration for all broker nodes, otherwise it is uncertain what will happen on a leader change.

To you questions:

  1. Yes on all brokers same configs
  2. Per default embedded gateways are enabled yes. They are running in each broker.
  3. Process instances are distributed over partitions not brokers. If you have for example three partitions, and one node is leader for all it might happen that the process instance hit the same node, but likely different partitions. Partitions are distributed/replicated based on the replication factor.
  4. It is recommended to use helm and kubernetes yes

Hope this helps.

Vineet Gupta: ok

Vineet Gupta: Thnaks very much for the detailed reply, now only left with two questions: I will use odd number of broker it is just for testing

  1. let first node has ip X.X.X.X and second has Y.Y.Y.Y ,then on ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT for all brokers i should set (X.X.X.X) right.
  2. and if for all brokers i have elastic search config then , there is not duplicacy of data on elastic search ( i think all leader export data to elastic search)
  3. And initialcontact_point list = [X.X.X.X, Y.Y.Y.Y, …] for all brokers.

zell: 1. Yes correct you just configure one contact point for the gateway.
2. No there is no duplicate data. There is always only one leader for a specific partition. Each partition has their own key range, to make them unique in the cluster.

Note: This post was generated by Slack Archivist from a conversation in the Zeebe Slack, a source of valuable discussions on Zeebe (get an invite). Someone in the Slack thought this was worth sharing!

If this post answered a question for you, hit the Like button - we use that to assess which posts to put into docs.