Ordering of messages or ConsistentHashing for Workers

Camunda 8 Topics
1

Hey,
So I have hit a wall with that and cannot find a right solution.
What is the design pattern when you have to maintain some sort of ordering between process instances?

My usecase is that I receive data on Kafka topic, my consumer takes it all and spawns process instance in Zeebe for each of those requests. Then the flow goes on for all items orchestrated by Zeebe. But some items depends in some way on each other.
For one id we can have multiple versions. All the versions are sent to one kafka partition in order. How can we maintain ordering for those items? Is it possible at all?
Best would be to have some sort of ConsistentHashing for Workers, so that for designated key jobs would be assigned to very same worker always, guaranteeing order of processing of those items.

We found in docs that when you have a message start event process instance, you can add correlationKey for that and it guarantees that only one processInstance will exists in time for that correlationKey.
Docs: Messages | Camunda Platform 8

But it has a drawback it stops the world for the items in the queue. And I just wish it to process simultaneously but in order of receving (so the flow progress for all items in the meantime, and not wait for the very end of the oldest).

Is there any design pattern for that case?

2

Hi @jt89,

that is an interesting question.

The idea with the message start event sounds valid but has its downsides with the delay.

Another idea would be an orchestration process to coordinate the work.

In general, the job workers can’t control the order of the jobs. Usually, a worker picks the jobs in the order as they are created. But it doesn’t work if there are more than one job worker for a job type. Or, a job worker fail a job, or a job times out.

Best regards,
Philipp

3

Hello,
Sorry for late response.
I’m really missing something like consistent hashing between workers. It would solve all my problems if I could create some sort of key, that would hash the jobs accordingly and send those grouped jobs to workers in FIFO manner. I’m really looking for scalability so multiple workers is a must.

Are there any plans to introduce some pattern for ordering/hashing of jobs in Zeebe platform?

4

Here is one way to do it:

You have a worker that does nothing except pull the jobs, hash them, and stick them on a queue / queues.

Your actual workers then pull work from the queues.

You implement the pinning of jobs to workers using your queue system and routing outside Zeebe.

Josh

1 Like
5

Well, of course I can, but it was Zeebe that was supposed to orchestrate my workflows, not additional orchestration services x] That’s why for now I’m seeking out-of-the-box solution.

And what is wrong with this solution is that I would have to have orchestration service for each step in bpmn… which could complicate a lot of arch and processing.

6