How to scale the brokers?

Hi,

So what is the process of scaling broker? I have read in various post that you cannot scale broker by just increasing the number but you need to always start with clean slate? Is that true?

Is there a guide on how to scale?

Thanks

The cluster does not dynamically scale. You cannot change the partition or node count. The only thing you could do is vertically scale nodes by taking a node down, and bringing it back up with more resources.

On Kubernetes, dynamic scaling can be achieved by combining vertical pod auto-scaling and horizontal auto-scaling of the k8s cluster. Important is the number of partitions because that determines the amount of parallelism and therefore how many vCPUs Zeebe can utilize under full load. In times of low load, multiple partitions can time-share a vCPU and thus multiple brokers could be scheduled an a single k8s node allowing the k8s cluster to scale down. For fault tolerance, you should however configure a pod anti-affinity that prevents multiple replicas of a partition to be on the same k8s node.

3 Likes

Thank you very much. Do you have an example kubernetes manifest for these?

the thing I currently do not understand is that. What is the relation between the number of pods and number of partitions? Do they need to be same or ?