After migration, I am experiencing issues with the old Camunda instances (instances from 8.1.5).
Every shard of the Elastic Search is migrated successfully, but when I publish a message which should trigger the old instances I’ve got just intent:PUBLISHED document under zeebe-record-message there is no intent:CORRELATED.
Can you please provide the checklist of what should be done to upgrade the Zeebe engine version?
What step in the migration process did I miss?
Thank you for reaching out about your migration from Camunda 8.1.5 to 8.5.17. The issue you’re experiencing with message correlation (seeing PUBLISHED but not CORRELATED intents) can have several causes. Let me provide you with a comprehensive checklist and troubleshooting steps.
Migration Checklist for Camunda 8.1.5 to 8.5.17
1. Component Version Compatibility
Ensure all Camunda components (Zeebe, Operate, Tasklist, Optimize) are running on the exact same version (8.5.17)
From version 8.6.0 forward, components must run on the same minor and patch level, but this is also a best practice for earlier versions
If you upgraded from Elasticsearch 7 to 8, ensure you replaced Elasticsearch Curator with Index Lifecycle Management (ILM) - Camunda 8 is not compatible with Elasticsearch 8 Curator
Verify all Elasticsearch indices are healthy and accessible
3. Upgrade Process Steps
When upgrading from earlier versions, you must delete Deployments/StatefulSets for Zeebe, Operate, Tasklist before upgrading due to label changes
Restart all components after the upgrade, including Elasticsearch
Ensure proper configuration updates are applied
4. Existing Process Instances
Important: By default, running process instances continue on their original process definition version after upgrade. You have two options:
Let them run naturally (recommended): Old instances finish on 8.1.5 version, new instances use 8.5.17
Migrate them: Use the Process Instance Migration feature in Operate if you need to apply critical changes
Troubleshooting Message Correlation Issues
1. Understanding PUBLISHED vs CORRELATED
PUBLISHED: Message was accepted for delivery (fire-and-forget)
CORRELATED: Message was actually matched to a waiting process instance
If you need immediate feedback on correlation success, use the message correlation endpoint instead of the publish endpoint
2. Common Causes for Missing CORRELATED Intents
A. Process Instance State
Verify that process instances are actually waiting at message catch events when messages are published
Check if instances are in the correct state to receive messages
B. Correlation Key Mismatch
Double-check that correlation keys in your BPMN model match exactly with the keys in your message publishing code
This is a common issue after upgrades
C. Message TTL Settings
If TTL = 0, messages are discarded if no subscription is open at publish time
If TTL > 0, messages are buffered for later correlation
3. Technical Verification Steps
Check Elasticsearch Health: Even if shards show as migrated successfully, verify index accessibility
Review Component Logs: Look for errors in Zeebe, Operate, and Elasticsearch logs
Verify Component Communication: Ensure all components can communicate properly after the upgrade
Test Message Correlation: Try correlating messages to new process instances to isolate if the issue is specific to old instances
Recommended Next Steps
Verify your Elasticsearch version and ILM configuration
Check if old process instances are actually waiting for messages (they should continue working on their original version)
Test message correlation with new process instances to confirm the upgrade was successful
Review correlation keys for any mismatches
Consider using the correlation endpoint instead of publish for better feedback
Thank you for the clarification. I found the root cause of your issue!
The Problem: Unsupported Direct Version Jump
Direct upgrades from Camunda 8.1.x to 8.5.x are NOT supported. According to the official Camunda documentation, you must perform a sequential upgrade through each minor version:
Decide on your approach (sequential upgrade vs fresh install)
If doing sequential upgrade: Start with restoring your 8.1.5 backup
Test thoroughly after each upgrade step
Document the process for your production environment
This should resolve your message correlation issues with the old instances. Let me know which approach you’d like to take, and I can provide more specific guidance for each upgrade step!
@Jovan_Zoric - apologies for the over-excited bot, but the latest reply is correct: you need to upgrade to each version in between or you will run into issues. 8.1.x has been unsupported for a long time, but you can find archives of the docs here:
I would work through each version incrementally, following the upgrade instructions for each version: 8.1 to 8.2 to 8.3 to 8.4 then finally 8.5.
The Problem: Unsupported Direct Version Jump
The Solution