Recently we have scaled up the self managed zeebe cluster from single node to 3 node cluster and we have set 3 partition and 3 replicas.
Since we have not passed the Node-id, Zeebe gateway created process instance ids on partition 4 But in the config we have only 3.
we have fixed the issue by passing node-id in the zeebe broker. But before fixing we have crated around 50 process instances.
These instances are again and again trying to execute in job workers.
question:
we are unable to cancel the process instances from Operate as well as ZeebeCtl or Java zeebe clients failed with below error
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: Expected to execute command on partition 4, but either it does not exist, or the gateway is not yet aware of it
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52)