Atomic Cluster Scaling from 1/1/1 to 3/3/3 got stuck during PARTITION_JOIN and left the cluster in an inconsistent state after recovery

Camunda 8 Topics
1

Hi,
We are validating the migration path for increasing Zeebe high availability.
Our goal was to migrate a cluster from:
clusterSize = 1
partitionsCount = 1
replicationFactor = 1
to
clusterSize = 3
partitionsCount = 3
replicationFactor = 3
using the official Atomic Cluster Scaling API.
Environment
Camunda Platform 8.9.13 Enterprise
Kubernetes
Elasticsearch 8
Official Camunda Helm Chart
Steps performed
Increased the Zeebe StatefulSet to 3 replicas.
Waited until all broker pods were created(actually, were not healthy)
Invoked the official /actuator/cluster API with a single request that:
added broker 1;
added broker 2;
increased partitionsCount from 1 to 3;
increased replicationFactor from 1 to 3.
The scaling operation started successfully but became permanently stuck during PARTITION_JOIN.
Broker logs repeatedly contained errors similar to:
Bootstrap step Partition 1 - Joining Raft failed
Sent join request to all known members, but all failed
Failed to send request to command-api-1, no remote address found
The cluster scaling operation remained in IN_PROGRESS indefinitely and could not be cancelled.
Recovery
To recover the cluster we eventually had to delete the PVC of zeebe-0.
After that the cluster became healthy again and all three brokers are running successfully.
However, after the recovery we also had to manually restore missing Identity configuration (roles/authorizations and related configuration), because part of the Identity configuration became unavailable.
Elasticsearch data was not deleted during the recovery.
Current issues
Although the cluster is healthy now, we have several inconsistencies:

  1. Orphaned Process Instances
    Operate still displays some Process Instances.
    However, cancelling them always returns NOT_FOUND because Zeebe no longer knows these process instances.
  2. Duplicate Process Definition versions
    For the same BPMN process we now have two Process Definitions with version = 1.
    It looks like Zeebe recreated version 1 after the recovery while Operate/Elasticsearch still contains the original version 1.
  3. Identity inconsistency
    Some Identity roles/authorizations and related configuration had to be recreated manually after the recovery.
    Questions
    Is migrating directly from 1/1/1 to 3/3/3 in a single /actuator/cluster request a supported scenario?
    Is this a known issue in Camunda 8.9.13?
    Is deleting the PVC of a single broker a supported recovery procedure after a failed Atomic Cluster Scaling operation?
    What is the recommended recovery procedure to avoid inconsistent state between Zeebe, Operate and Identity?
    Is there a supported way to remove orphaned Process Instances and duplicate Process Definitions without rebuilding Elasticsearch indices?
    Any guidance would be greatly appreciated.
    Thank you.
2

This looks related to known Zeebe dynamic scaling and Raft partition-join issues, as well as some documented behaviors around duplicate process instances/definitions after recovery. I found the following relevant resources:

Does this help? If not, can anyone from the community jump in? :waving_hand:

:light_bulb: Hints: Use the Ask AI feature in Camunda’s documentation to chat with AI and get fast help. Report bugs and features in Camuda’s GitHub issue tracker. Trust the process. :robot: