Hello,
I have a question about the number of connectors instantiated for one BPMN with multiple connector elements.
We have developed a connector for the AMPS broker from 60East. It is almost identical to Kafka’s one. So for further discussion, we could assume we are talking about Kafka.
We have a queue/topic with events that signal creation (field in message body called status=NEW), update (status=UPDATE), and closing of the workflow (status=CLOSE).
The BPMN starts with a Connector element configured with Activation (value.status=NEW).
The intermediate catch Connector element is configured with Activation (value.status=UPDATE) and correlation key to match the business key of the message. The last Connector is configured the same way as the previous but the Activation is (value.status=CLOSE).
When the BPMN is deployed it triggers activation of 3 independent connectors.
Each of those 3 connectors creates their own connection and subscription (background thread).
As a result, they are stealing messages from each other and throwing away the ones that do not match the activation.
SIDENOTE: Also currently it is not possible to know whether the message activated the workflow or not due to this issue.
And even if it was possible, then issuing nack for the message would be very inefficient and could cause reordering of the messages (which is happening anyway due to 3 different subscriptions/threads.)
Adding a filter to the subscription so the NEW, UPDATE, and CLOSE events are not even consumed by the connector that is not interested in them does not solve the issue of potential reordering
In the ideal case, only one subscription would be needed per BPMN as only that would guarantee the order of the messages and only the activation condition would change for different elements in the BPMN.
Is this achievable with connectors or is it more case for the Job Worker?
(I like how connectors fetch the process definitions/configuration.)
Thanks
1 Like
Bump. Experiencing similar issue.
Thank you for bringing this up!
This is a design limitation currently. There is no deduplication logic in place to support this usage scenario.
I can see 2 approaches here, an implicit and an explicit one.
Implicit: the Connector runtime compares the properties of each connector instance and decides based on certain rules whether 2 elements represent different connector instances or the same one. Some properties are handled by the runtime (e.g. activation condition, error & response mapping), so the runtime would still have to remember the original elements and map them to the actual connector instances (threads) it created.
Explicit: same as the above, but the runtime does not rely on implicit rules and property comparison. Instead, we could introduce a new property deduplicationId that would allow to define the desired behavior during process modeling. Elements with the same deduplicationId and connector type lead to the creation of the same connector instance. This provides more flexibility, but we might want to introduce additional validation to make sure connectors are actually equal in terms of properties.
We will discuss within the team and decide how we could implement this and which approach we prefer. I will update this thread when I have news.
In the meantime, I can suggest that you use one element per connector in your process, e.g. by modeling only one event and then analyzing its payload using a Script Task to route the message based on its status.
Good news: we decided to proceed with the implementation, combining the implicit and explicit approaches defined in the message above in a single solution.
When this feature is implemented, you will be able to set a custom deduplication ID for connectors that must share the same subscription. The runtime will also be capable of deriving a sensible default value for the deduplication ID if it’s not provided.
Feel free to share your thoughts and track the progress here: Support deduplication ID for inbound connectors · Issue #2132 · camunda/connectors · GitHub
1 Like