We have a cluster of 3 Zeebe brokers v8.0.2 with 2 Zeebe Gateways in front of them. The brokers are configured with 3 partitions and 3 replicas. we monitor the cluster using the Grafana dashboard provided by Camunda.
Every thing seems fine until one or two weeks of worflows running. At some point we can see that one of the three partition is having backpressure Dropping requests as follow:
- partition 1 : 0%
- partition 2: 8.32%
- partition 3: 36.5%
See screenshot attached
When we are such a situation, the workflows are slower than before. We want to understand why only one partition is saturated and what can we do solve it.
Any advises are welcome…
Thanks in advance,