Hey @vasyl
thanks for your post and that you share your findings!
I agree that mostly it is the disk which slows things down. But of course there are also other parameters which need to be considered. Maybe this post also helps here Questions about zeebe performance test - #2 by Zelldon
My normal advice would be to use a large enough SSD, use multiple partitions, turn on replication (as you have seen we currently have issues with replication factor one), give the brokers enough resources (per partition ~1-2 CPU). Make sure to also increase the CPU and io thread count, otherwise the available CPU’s aren’t used. Setup an standalone gateway to scale the gateway independently (and give it own resources). Embedded gateway will also slow down the broker. Reduce the worker count, many workers can also slow down processing since all of them will send regularly Activation commands.
I recommend to use our helm charts to setup a Zeebe cluster https://helm.camunda.io/ here you can find also already quite good defaults, and can configure them more easily.
Hope that helps. 