Cluster partitions distribute unevenly on cluster nodes in zeebe v0.24.x / v0.25.0-alpha1

I set up the zeebe cluster with zeebe-docker-compose on 3 hosts, not k8s.
The cluster configuration:

clusterSize: 3
replicationFactor: 3
partitionsCount: 12

With zeebe v0.23.x, it performs well. Every zeebe node has 4 leader partitions, and 8 follower partitions.

host-1-docker-zeebe-node-leader-partitionId: {1, 4 , 7, 10}
host-2-docker-zeebe-node-leader-partitionId: {2, 5 , 8, 11}
host-3-docker-zeebe-node-leader-partitionId: {3, 6 , 9, 12}

However, when it comes to zeebe v0.24.x / v0.25.0-alpha1, the partitions distribute unevenly on 3 cluster nodes. Node-1 always has 11 leader partitions, while node-2 has 1 leader and node-3 has none.
Just for example:

host-1-docker-zeebe-node-leader-partitionId: {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11}
host-2-docker-zeebe-node-leader-partitionId: {12}
host-3-docker-zeebe-node-leader-partitionId: {none}    

I start the cluster with “docker-compose up -d”, and try my best to execute the command on 3 hosts at same time.

Hey @ruozhixian

this is expected because we had to remove the evenly leader distribution, because it causes some other issues and was not save how it was done. You can take a look at the related PR https://github.com/zeebe-io/zeebe/pull/4298

Greets
Chris

hi @Zelldon

then gateway transforms all client requests to the leader partitions (host-1), and host-2/host-3 just watch host-1 performs as the workflow engine core lonely?

Hey @ruozhixian

sorry but I didn’t get it. Could you elaborate on your last post, please?

Greets
Chris

hey, @Zelldon

I set up a zeebe cluster with v0.24.2 on 3 hosts, each zeebe node with an operate.
Cluster configuration:

clusterSize: 3
replicationFactor: 3
partitionsCount: 12 

Leader partitions deployment:

host-1-docker-zeebe-node-leader-partitionId: {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}
host-2-docker-zeebe-node-leader-partitionId: {none}
host-3-docker-zeebe-node-leader-partitionId: {none}   

When I deploy an workflow to the cluster, only operate-1 shows the result.

Then the client requests to create instances of the workflow, continuously.

It seems like all client requests are transferred to the leader partitions(host-1), and host-2/host-3 just do nothing, maybe, except backup.

What’s more, just a moment ago, host-2 became partition-5 leader, and now operate-2 home page shows there are some running instances. Because the deployed workflow was not shown on this page, I can not see more details.

1 Like

Yes, this is correct. Partitions have exactly one broker as the leader, but not every broker is guaranteed to be a leader for any partition.

You have set replication factor 3, so each partition has 2 replicas, in addition to the leader. That means that the other two brokers are acting as followers for all partitions.

Depending on how many threads you have available on each node, you may get better performance with less partitions, given that - as you can see - a single broker can end up as the leader for all partitions.

Josh