I have more than three thousand of instance running with 8cpus and 32gb memories. I got timeout error recently and can’t start new instance any more (got the same timeout). I tried to set ‘ZEEBE_GATEWAY_CLUSTER_REQUESTTIMEOUT’ while got nothing work. I’m wondering how to fix this problem?
Hi @Edward-Shaw
the problem you’re facing is that your have to much load on your brokers, which means the processing queue (log) is too long. If it is too long the broker can’t sent responses in time back, which will cause such timeouts you see.
Reduce the load, or increase the partitions. Be aware that zeebe right now don’t support dynamic scaling so you would need to setup a new cluster.
Greets
Chris