Can I resize an existing cluster (Feb 2021 tldr; “No”)

Dag H. Baardsen: When “migrating” from a single node broker to a 3-node broker cluster using docker-compose, how should I migrate the data storage? Is it enough to copy the single node data directory to the first node’s data volume or should it be copied to all three node data directories? Or am I missing something? Help appreciated.

korthout: Hi @Dag H. Baardsen Zeebe does not support changing the clusterSize, partitionCount or replicationFactor for an existing cluster. It is not possible to copy the data.

Dag H. Baardsen: Ah, OK. Thanks for the answer!

Dag H. Baardsen: Does any other strategy for recreating data between cluster configuration changes exist? Like export/import, re-running workflows internally, third party software. Or is there something in the roadmap what will cater for these needs? I guess I’m not the only one with these concerns.

korthout: I’m not aware of any available strategies for this. It is a valid concern, but at this time we’re not working on a solution for it. Generally, we recommend to keep your development and production environment as similar as possible. The issue might become more important to us in the future, so if it is important to you, please create an issue for it on github

Dag H. Baardsen: I will do. BTW, this is not a “bring my data from development to production”, but more of a “my resources are exhausted in production, so I have to expand my cluster” concern.

korthout: Ah clear. Thanks for bringing it up. We do have an <|open feature request> for dynamic resizing of an existing cluster. It might make sense to add your use case as a comment there. It’s likely that this feature will not be implemented any time soon. However, I recommend commenting your use case, because perhaps a workaround can be made available more easily while we don’t have this massive feature yet. It’s worth discussing IMO.

Josh Wulf: See also:

Note: This post was generated by Slack Archivist from a conversation in the Zeebe Slack, a source of valuable discussions on Zeebe (get an invite). Someone in the Slack thought this was worth sharing!

If this post answered a question for you, hit the Like button - we use that to assess which posts to put into docs.

Dag H. Baardsen: Gotta love that slack-archivist profile photo!

1 Like

See this open feature request, and feel free to upvote: