Hi all,
Upfront, some context on where this sits: this is exploratory work, not a decision that has been taken. We are at the assessment stage, mapping out possible routes off an unsupported version so we can put a properly costed set of options in front of our stakeholders. Nothing here is committed, no target version is locked, and no approach is favoured to the point that we would defend it.
Posting partly to sanity-check our reasoning against people who have actually done this, and partly because our understanding of what is possible between Camunda 8 versions is based on documentation and inference rather than experience. Some of the assumptions below may well be wrong, and corrections are more useful than agreement.
Environment details are kept generic, and figures are indicative rather than exact.
1. Current environment
We run Camunda 8.1.14 Self-Managed on EKS:
- Zeebe: 3 brokers, 3 partitions, replication factor 3
- Elasticsearch 7.17
- Keycloak 21, with Identity
- Operate, Tasklist, Optimize and Connectors all enabled
- A custom connector runtime and Spring Boot workers on the old ZeebeClient line
It is out of support and our licence is up for renewal, so we need to reach a supported version. Our working target is 8.9, but that is not finalised. Depending on testing effort, timelines and support considerations we may stop at 8.7 instead. The strategy questions below apply either way.
2. What we are considering, and why
The documented path is sequential: 8.1 → 8.2 → … → 8.9, with a data migration at every hop.
That is many upgrades, each needing its own testing and release window, and it delivers little business value until the final target version is reached.
So we are looking at building a fresh cluster at the target version instead, redeploying our BPMN and DMN after a compatibility review, and pointing the application at it. This is mainly a time argument: one build and one cut-over rather than seven or eight upgrades.
In both options below, no runtime data is copied between clusters. Audit history stays where it is. We would keep the old Elasticsearch read-only for retention purposes, then shut down the brokers and web applications. The open question is what happens to running process instances.
3. Option 1 — cut over to the new cluster
Build the new cluster, switch the application across, retire 8.1. Instances still running on the old cluster are lost.
We can live with some loss, but want to keep it as small as possible. Our idea is to stop starting new processes on 8.1 a week or two before cut-over, allowing short-lived processes to complete naturally so that only the long-running tail remains.
For whatever is still running at that point:
3.1. Is there a supported way to create an instance on the new cluster at a specific element in the model, with variables seeded from the old instance? In other words, can start-instructions / process-modification style capabilities be used to restart an instance roughly where it left off?
3.2. Has anyone exported in-flight state from Operate or Elasticsearch and used it to reconstruct instances on a new cluster? What worked well, and what were the pitfalls?
3.3. Anything else people have done to reduce what gets thrown away in a fresh-cluster cut-over?
4. Option 2 — run both clusters in parallel and let the old one drain
New processes start on the new cluster, existing ones finish on 8.1, and we retire 8.1 once it is empty. Nothing is lost, but we do not know how to build it, and we have found few real-world examples of this being done between Camunda 8 versions.
To be clear about what we are asking on this option: cost is not the deciding factor here. We want to understand whether it is architecturally possible at all, and if so, what the design actually looks like, even if the answer is that it works but is expensive to run. If it is viable, we can have the cost conversation internally; what we cannot do is cost something we do not yet know how to build.
The difficulty is that it is one application. During the overlap it has to start new instances on the new cluster while old instances keep running on 8.1, and both have to keep working.
We do not think we can serve both from one runtime. Our current application sits on Spring Boot 2.x with the older ZeebeClient over gRPC, while the 8.9 Camunda Spring Boot Starter defaults to Spring Boot 4.0, and two Spring Boot major versions cannot share one application context. We are aware the Spring Boot version can be pinned, but that would mean running outside supported production configurations, which we would rather not do.
4.1. Is that understanding correct, or has anyone run an old and a new Camunda client generation in a single application successfully? If it is more achievable than we think, that changes our sizing considerably.
Assuming it is not, our working design is two separate deployables:
- The current application, frozen, serving only in-flight instances until they drain
- A new application for the new cluster
- A routing layer in front that sends new requests to the new application and anything touching an existing instance to the old one
4.2. Is this parallel-drain approach feasible in practice between Camunda 8 versions, or is there a fundamental limitation that makes it impractical?
4.3. Where should the routing decision live?
- API gateway
- Dedicated routing service
- Inside the application layer
- Somewhere else
4.4. Job workers: the same job types exist on both clusters and cannot share a runtime either. Is it simply two worker deployments for the duration?
4.5. Correlation: our understanding is that a message published to one cluster cannot correlate with an instance on the other. If so, do teams maintain their own mapping, for example:
Business Key → Owning Cluster → Process Instance Key
Or is there a more established pattern?
4.6. For those who have done this:
- How long did the old cluster take to drain, and what made it take longer than you expected?
- What surprised you in the design or the running of it?
- Would you do it again?
We are conscious that the frozen application cannot be patched while it is still serving production traffic, so a long overlap becomes a security and maintenance question as well as a cost one.
5. Where this leaves us
We are essentially trying to decide between three approaches:
- Sequential upgrades
- A fresh-cluster cut-over
- A fresh-cluster parallel drain
For context, part of the value of asking here is educational as much as practical. We want to understand what risks the sequential path is protecting us from, and whether those same risks simply reappear elsewhere if we attempt a fresh-cluster migration strategy.
If the answer is that the sequential upgrades are the right call and the alternatives carry risks we have not accounted for, that is a genuinely useful answer and we would rather hear it now than later.
Any real-world experience, lessons learned, recommendations or warnings would be greatly appreciated, including “we tried the parallel run and went back to sequential”.
Thanks in advance.
Hints: Use the Ask AI feature in 