So I have hit a wall with that and cannot find a right solution.
What is the design pattern when you have to maintain some sort of ordering between process instances?
My usecase is that I receive data on Kafka topic, my consumer takes it all and spawns process instance in Zeebe for each of those requests. Then the flow goes on for all items orchestrated by Zeebe. But some items depends in some way on each other.
For one id we can have multiple versions. All the versions are sent to one kafka partition in order. How can we maintain ordering for those items? Is it possible at all?
Best would be to have some sort of ConsistentHashing for Workers, so that for designated key jobs would be assigned to very same worker always, guaranteeing order of processing of those items.
We found in docs that when you have a message start event process instance, you can add correlationKey for that and it guarantees that only one processInstance will exists in time for that correlationKey.
Docs: Messages | Camunda Platform 8
But it has a drawback it stops the world for the items in the queue. And I just wish it to process simultaneously but in order of receving (so the flow progress for all items in the meantime, and not wait for the very end of the oldest).
Is there any design pattern for that case?