I want to setup two zeebe broker(docker containers) connected to same elastic search,
one broker is at localhost:26500 and other is at localhost:9000 and supose there is a load balancer between them which transfers he request to an of them.
If i deploy a bpmn workflow to a zeebe broker then how would other zeebe broker will know about deployment.
I think zeebe broker does not exchange data using elastic search, they exchange information between each other using the broker gateway. correct me please.
Okay but how to connect them. where to specify the connection point.
If two zeebe broker are running on different ec2 machines where to specify connection url. or contact point.
Requirement: want to setup two zeebe brokers on two different machines, and will run another broker if needed, these brokers will belong to same cluster. The cluster should export the data to elasticsearch.
Steps I am doing:
Firstly i am running docker-compose up which runs the first broker in a ec2 machine , i have set the elastic search exporter in this broker.
after first broker is up , i ran another docker-compose up with different configs ( elasticsearch exporter is not used, )
Questions:
did i need to use elastic search exporter on all brokers that are in cluster or have to specify in only first broker gateway connects to.
does by default configuration different gateways are created for all brokers.
also i want to experiment this setup in javascript, so how would i know that proccess instances are distributed between different brokers if my setup is correct.
please answer these questions as i am not getting these after reading.
Or this is not correct way to setup zeebe cluster, we should go with kubernetes with helm charts.
Hi @korthout the docs u shared does not clarify on how do we setup gateway, does in both zeebe_broker’s application.yml file we have to set gateway??
suppose ip of first node is X.X.X.X second node ip Y.Y.Y.Y
If we set gateway for both broker does “ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT. = X.X.X.X” in both??
or we had to set gateway in both broker with their own different host addresses.
or we had to set gateway in only first broker with ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT. set to first node ip (X.X.X.X)
In all these cases it is supposed that in both broker config i have set intial_cluster list =[X.X.X.X, Y.Y.Y.Y]
I am using two pc for two brokers and running each broker with different docker-compose.
let first node has ip X.X.X.X and second has Y.Y.Y.Y ,then on ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT for all brokers i should set (X.X.X.X) right.
then i can connect the cluster using all gateways from my client?? the request will goes to first node only and then distributed. Now here i can use a load balancer which then forward request to any of gateway.
process instances are distributed over multiple partitions not necessarily over different brokers. Only when multiple brokers are the leader for different partitions, this would happen. First, make sure you’ve set up multiple partitions, then check the topology using zbctl status which tells you which broker has which partitions and whether they are the leader or follower for that partition. If the topology looks correct, and all partitions are healthy you can be sure that process instances are distributed over the different partitions. The rebalancing guide discusses the advanced topic of rebalancing the broker leadership of partitions.
These helm charts are the recommended way to set up a cluster. Please give that a try.
I think trying out the helm charts will help you more, than me trying to answer the leftover questions. So, I’ll leave it at that.
let first node has ip X.X.X.X and second has Y.Y.Y.Y ,then on ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT for all brokers i should set (X.X.X.X) right.
then i can connect the cluster using all gateways(X.X.X.X:25600 and using Y.Y.Y.Y:25600) from my client?? the request will goes to first node only and then distributed.
Now here i can use a load balancer which then forward request to any of gateway. or i can forward all load to one gateway only then this gateway will forward to first node and then first node will distribute.
I am asking as i want to know that is loadbalancers required before gateways or one gateway is ok.
To get started you can either use the embedded gateway or set up a single standalone gateway (which will need a single ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT). In both cases, the gateway learns about the other brokers from communication with the broker specified by the contactpoint.
