Force-sync partitions

Hi there,

I’m currently struggling with getting the partitions of my cluster back in sync. Creating a new process instance randomly fails depending on the partion it tries to operate on. In this case, there are only 2 partitions - 1 has the definition, 2 is not having it.

Before this error started to occur, I tested reducing and increasing the partition count - so I assume this caused the problem.

Strangely, the Grafana dashboard says that both of those partitions are healthy and I can’t see any error in the logs.

This issue seems to be similar to Command 'CREATE' rejected with code 'NOT_FOUND & Zeebe is not starting new instance: ClientStatusException - Command 'CREATE' rejected with code 'NOT_FOUND'.
If I understood correctly, according to Deployment Distribution | Zeebe Chaos it should recover eventually - obviously it doesn’t.

Example from trying to create the process via zbctl:

$ zbctl create instance Process__5e26fde75d2f357107601e81__test
  "processDefinitionKey":  "2251799831629657",
  "bpmnProcessId":  "Process__5e26fde75d2f357107601e81__test",
  "version":  1,
  "processInstanceKey":  "2251799838745659"
$ # ... next command
$ zbctl  create instance Process__5e26fde75d2f357107601e81__test
Error: rpc error: code = NotFound desc = Command 'CREATE' rejected with code 'NOT_FOUND': Expected to find process definition with process ID 'Process__5e26fde75d2f357107601e81__test', but none found

Is there a way to enforce syncing the partions? Any hint appreciated.

Best regards,