Camunda 8: Message correlation with a filter to multiple processes

We’ve created an event driven microservice based application. Everything that happens generates an event, that can be picked up by other microservices, or by camunda. I think this is the exact use case where camunda is focusing on (orchestration for microservice based applications).

Current setup in Camunda 7

We currently use Camunda 7, and have created a Java plugin to integrate camunda with our AMQP message bus (RabbitMQ). We can publish an AMQP message (for example “order.updated”) to any running processes that might be waiting on an order confirmation. As the message name, we use the amqp topic, and with input mappings, we can specify filters for what events we want to handle.

An example of a event can be like this:

  "orderId": 123,
  "status": 1,
  "confirmation_id": "ABCDEF123456",
  "validated_by": "some_user"

When we receive an AMQP message, our plugin will loop over all message subscriptions, and match them with the amqp message (so we do our message correlation ourself). This way, we can have a BPMN process that will initiate an order, and since an order might take a while to be completed, we wait for the order.completed event, and filter based on orderId = 123 and status = 1 (from the input mappings, see below).

Now, Camunda Platform 8

In Camunda Platform 8, this will have to change. We like the approach to be able to just subscribe on a topic from a Receive Task, and filter based on some variables, but this won’t work anymore, since a message can’t have input variables, only a correlation key.

Also, we can only use the PublishMessage RPC, so we can’t do the message correlation ourselves anymore, and by design, Camunda Platform 8 will correlate a message only to one process, as explained here: Messages | Camunda Platform 8

The only solution I see is for our system to generate specific events, with a correlationKey, like order-updated-123-1, but then we aren’t using the event driven way anymore and created a lot of overhead.

Is there a way to do this in Camunda Platform 8?

Maybe a solution

A solution could be to add input filtering in addition to the correlationKey, and allow a message to be handled by multiple receive tasks in multiple processes (but Message Cardinality is by design, so this won’t change probably). Some related discussions here: Multiple identical Message Events are not honoured with Publish Message · Issue #4644 · camunda/zeebe · GitHub

This way, we can publish the above message, and add input mappings for orderId and status, and allows us to extract the confirmation_id with an output mapping.

Hi @MichaelArnauts :wave:

That’s an interesting case :thinking:

Would it be possible to have your AMQP message consumer check whether the status is 1 and if so, then publish a Zeebe message named order-updated with the orderId as correlationKey (e.g. 123)? As you mention, all that data is available in the AMQP message already. Note that I’m assuming you only have at most 1 instance of each process for a given orderId.

Sure, that would be possible, but this would require a change to my amqp-zeebe-connector for every message I want to react upon in a BPMN process. I hope to be able to make this as generic as possible, so I can opensource it.

Thinking about this more, I will probably need to use a signal for my use case, but that’s not yet supported in Zeebe. This won’t have the limitation of the “one instance per message”, but then I still would need to be able to filter the messages somehow.

I see, thanks for the details. It helps us to understand your use case for such a feature.

Like you’ve already mentioned, at the moment signals or messages with high cardinality are not available in Camunda Platform 8. AFAIK, such a feature is requested more often, but so far it has not yet been on our planning.

The filter concept you describe is an interesting way to reduce the performance impact that might be involved with such a feature. We’ll have to keep that in mind if we do start work on this.

Note that process instances are spread over all partitions, so broadcasted messages (e.g. signals) would have to be cross-partition (defeating the horizontal scalability of partitioning). In addition, they would affect all active instances subscribed to it on each partition. We expect that there might be situations where this could clog up the engine. When we do get to it, it’ll be an interesting technical challenge.