We are currently prototyping a process solution based on camunda 7.8.
The goal is to run a homogenous cluster (multiple nodes with the same software and processes) against a shared database. Each cluster node has its own JobExecutor (for acquisition and execution).
What I found from reading through the code: Each AcquireJobsCmd would acquire some jobs and check if they belong to the same process instance and execute those sequentially, as described in the documentation.
However, if there are multiple jobs for the same instance, and each is acquired by a different JobExecutor (on a different cluster node), they would run in parallel. Say each node acquires 3 jobs and (after sorting) job 1 and 4 are for the same instance, then job 1 would be executed on one node and job 4 on another.
Did I miss something? Is this intended behaviour? How could we mitigate this?
I assumed that any one job executor would check if there were other exclusive jobs to be executed (for the same instance) and lock all of them. That would probably need some acquisition lock, so that only one JobExecutor acquires at a time.
If I understand you correctly, you appear to want all activities from a particular process instance to run only on a specific node. For example, process ABC would run all of its activities (jobs) on node #1, process DEF would run them on node #2, etc.
If that is what you are trying to do, then I’m not sure there’s a way to do that out of the box (at least in the 7.6.2 version we’re using). The point of the cluster architecture as Camunda have implemented it is the sharing of resource utilization across multiple nodes without requiring the developer to manage any of it. This means you can scale vertically (faster nodes) and horizontally (more nodes) very easily as long as the shared database can handle the load. In other words, this is the intended behavior.
Others with more experience may have a different answer, but to the best of my knowledge the only way to “force” a particular process to execute on particular node would be to set the jobExecutorDeploymentAware value in the processes.xml file to “true”. This will force the job executor to only acquire jobs for processes actually “deployed” to that node, though what is meant by “deployed” is a bit unclear. This would not achieve what I believe you are describing above, however.
I don’t want to achieve that. The docs make a compelling argument for why exclusive tasks are a good idea. And I thought the implementation would take care of only ever running one (exclusive) job per process instance at a time. With a clustered setup I currently see no way the engine accomplishes that.
Could anyone please clarify how exclusive task behave in a clustered environment?