Arbitrary execution of JobExecutor jobs in a distributed manner

Suppose i have a process instance with asynch service tasks and a number of independent services (potentially on different JVMs or even hosts), which all share in the process instance execution (so each of the services employs Embedded process engines, but pointing to the shared database).

Is there a way to introduce a kind of “content-based” arbitrated job execution, as in: only the JobExecutor of a particular service will execute the next task on the process, based on some condition, e.g. a context variable value?

The reasoning for this being simple: in a distributed workflow each task is “associated” with a particular service (the association being maintained outside the workflow in general, but context variables may also be used for it) and may not be executed by any other service.

Is it something reasonable to ask, or there is some other way of achieving my goal?

Hi @yzt,

It’s not possible to set specific JobExecutors on certain tasks. However, you can use ExternalTasks to achieve the same goal. With an ExternalTask you can specify a topic, and you can have separate workers dedicated to each topic. You can read more about it here: https://docs.camunda.org/manual/latest/user-guide/process-engine/external-tasks/

Best,
Nikola

Yep, that’s what i ended up doing.

Workers, however, seem to try executing all of the tasks, so some filtering must be implemented, which, sadly, Camunda doesn’t provide, apart from specifying the topic and businessKey. In other words, there’s not a meaninful way to distribute tasks between workers, unless some content-based filtering is introduced. In my case it had to be an informal mnemonic “binding” between activity names and worker-specific IDs and even then i had to fiddle with immediate unlocking of tasks that shouldn’t be executed by “this” worker.

I wonder, if there’s some other, more intelligent way to introduce this filtering?

Hi @yzt,

By using the Rest API’s fetchAndLock endpoint, you can specify processVariables for which to filter by. You can specify the name and the value of the variable (you can find the docs here: https://docs.camunda.org/manual/latest/reference/rest/external-task/fetch/#request-body).

However, this is currently not implemented in the External Task client. The corresponding ticket can be found here: https://app.camunda.com/jira/browse/CAM-9350. I’m not sure when this would be implemented, but if you want, you can add this feature to the client and provide a PR for it.

Best,
Nikola

My engines are embedded, i don’t use and don’t plan to use REST API. I tried to use processVariables, but there’s no way to specify them from the BPMN file - in my case “bindings” must be static and declared in design-time. There was a post somewhere here how this can be achieved by using a script, but this also contradicts my use-case: no scripting is allowed in BPMN file.

I will leave the question unanswered, as clearly there’s no satisfactory answer atm, which doesn’t imply noticeable restrictions on design-time and a certain amount of advanced coding in the runtime.

In addition to that, does any CDI event (e.g. @StartActivity or @CompleteTask) fire when ExternalTaskService#fetchAndLock fetches tasks or ExternalTaskService#complete completes a task?

Hi @yzt,

One idea that comes to mind is to use an Execution listener with an expression on the External Task to set a Process Variable. But this would cover some simple use-cases only.

In addition to that, does any CDI event (e.g. @StartActivity or @CompleteTask ) fire when ExternalTaskService#fetchAndLock fetches tasks or ExternalTaskService#complete completes a task?

As far as I know, no CDI event is fired when an ExternalTaskService#fetchAndLock or ExternalTaskService#complete is called.

Best,
Nikola

I checked and it’s fortunately not true: ExternalTaskService#fetchAndLock does indeed fire @StartActivity event, while ExternalTaskService#complete fires @EndActivity.