Zeebe Cluster Setup & Errors; RESOURCE_EXHAUSTED?

Hey All,

I have been trying to get zeebe up and running for about a month now. trying to achieve some load testing numbers…

currently I’m running the stand-alone broker docker compose configuration with 1 gateway and 3 brokers.
I am running two instances of the java client using NGinx to load balance

my flow has 3 tasks, each of which makes a simple GET call and returns the response body.

Do I need to add more gateways ?? how would one go about this using the docker-compose file…?

I am using Jmeter to load-test and I keep getting the following error :

2020-12-18 15:18:48.837 ERROR 2608 — [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is io.zeebe.client.api.command.ClientStatusException: Expected to execute the command on one of the partitions, but all failed; there are no more partitions available to retry. Please try again. If the error persists contact your zeebe operator] with root cause

io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Expected to execute the command on one of the partitions, but all failed; there are no more partitions available to retry. Please try again. If the error persists contact your zeebe operator
at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-api-1.33.0.jar!/:1.33.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:478) ~[grpc-stub-1.33.0.jar!/:1.33.0]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:616) ~[grpc-core-1.33.0.jar!/:1.33.0]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69) ~[grpc-core-1.33.0.jar!/:1.33.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:802) ~[grpc-core-1.33.0.jar!/:1.33.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:781) ~[grpc-core-1.33.0.jar!/:1.33.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.33.0.jar!/:1.33.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.33.0.jar!/:1.33.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_261]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_261]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_261]

Running 10 threads 100 instances leads to this error with a 56/sec throughput and 88% error rate

I have tired increasing the partitions number from 2 → 5 → 10 → 100 problem still arises

Client code:

To understand partition configuration, try this video: https://www.youtube.com/watch?v=AETSxfR6UUc

Caused by: io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: Expected to execute the command on one of the partitions, but all failed; there are no more partitions available to retry. Please try again. If the error persists contact your zeebe operator
at io.grpc.Status.asRuntimeException(Status.java:533)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:449)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:689)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$900(ClientCallImpl.java:577)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:751)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:740)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
… 1 common frames omitted

What is the reason for this error and how to solve it?