External Tasks Pattern and Messaging: Best of both worlds

Hello,

We want to setup Camunda in order to process our computation tasks using workers that are notified through messaging (JMS or similar).

We see two choices:

The first using a classic BPMN style with send and receive messages.
It has various flaws:

  • Business work is not explicit
  • Could not leverage on Camunda retry/incidents facilities to handle worker runtime failures
  • Could not leverage on BPMN Error to handle business errors

So we are seduced by the second option: the External Tasks Pattern.
All the benefits described in the documentation of the External Tasks Pattern are exactly what we try to reach.

But it seems that External Tasks Pattern is made for: HTTP + Polling.

If we want to use it with messaging the API has some flaws:

  • Our “new task” message is send in the start listener of the external task.
    There the External Task object is not yet created so we can’t notify the External Task Id. We have to use the Execution Id (which can be linked to multiple External Tasks??).
    We could not find any better hook for the “new task” notification.

  • The API does not allow to complete the task without locking it first.
    In our case, we rely on the messaging middleware to handle the lock/unicity.
    There is also no way to explicitly lock a task given its external id (or execution id).

We manage to find a way through those issues writing an extended ExternalTaskService, but it feels like we are missing something.

This blog post says that External Tasks Pattern is a good option to “replace JMS queue between the Service Task and the Service Implementation”.
Great news! But what is a good implementation if I like the pain of the messaging architecture?

Hello again …

external tasks are imho not just another way of implementing a message queue. They are an inversion of control, anew way of thinking how external services interact with your process.
You would not notify an external worker that a task is active. You would just wait until a worker comes by (possible scheduled) and executes it. Think of “tasklist for services”.

I am not saying that this is the “new way” to go, it is a new approach to a certain kind of problem. If you require tight interaction, direct feedback, no waittime and full control on the process side, the external worker is not for you. If you can deal with “I am loosly coupled, resilient and my external task will eventually be completed if I just sit and wait”, it’s perfect.

Jan

1 Like

I agree with Jan that external tasks are not the solution if you want to stick to the message queue.

In addition to his remarks, have a look at the asynchronous service invocation pattern: https://github.com/camunda/camunda-bpm-examples/tree/master/servicetask/service-invocation-asynchronous. It allows you to use your current approach and model a single service task at the same time.

Cheers,
Thorben

1 Like

I completely agree with you! I knew something was wrong.
The asynchronous service invocation pattern seems a perfect fit for us.
Many thanks!