Does Camunda 8 engine preserve order of processing external tasks?
Lets say I have 10 External Service Tasks to perform. Tasks was created in certain order (one at a time).
If I have 1 Zeebe client which fetch 1 task at a time, does order of fetched tasks is preserved?
The oldest task is fetched first, then second, …
?
Each job is handled independently.
The order will depend on several factors and one of them is the execution time of the tasks.
Therefore, it is not safe to assume that you will have the tasks returned in order… if you really need to guarantee this order, you may have to create logic to guarantee this.
Thanks for the quick reply!
I don’t need the EXACT order, I need to keep more or less the order of creation time.
For example, if hundreds of tasks queue for a while (the zeebee client), there will be no situation that tasks created first get ‘stuck’ for a long time blocking some process to go on. This is a necessary condition for KPI fulfilment
Could you point me in the right direction to find a solution (or some documentation describing the algorithm)?
A.
Unfortunately, to guarantee an order in the execution of activities, we only need to implement logic to do this as I mentioned, as it depends on a series of factors, such as processing size, resource availability, among others.
I’m leaving the link to Camunda’s official documentation regarding external tasks below. It’s from Camunda 7, but the behavior on Camunda 8 should be similar
As for a possible solution, I don’t know your level of technical programming knowledge, but you could implement a queue as SQS (FIFO) for external tasks, and create the worker searching for tasks from this queue.
Hi again,
I’ve done some test.
Simple process with single external task + spring worker (1 thread).
I’ve created 1000 processes first (the id at the end of each line is the ProcessInstanceKey):
2023-10-19 22:22:07,461 [grpc-default-executor-1] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813710949
2023-10-19 22:22:07,480 [grpc-default-executor-0] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813710956
2023-10-19 22:22:07,491 [grpc-default-executor-1] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813710963
2023-10-19 22:22:07,502 [grpc-default-executor-0] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813710970
2023-10-19 22:22:07,512 [grpc-default-executor-1] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813710977
2023-10-19 22:22:07,522 [grpc-default-executor-0] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813710984
2023-10-19 22:22:07,531 [grpc-default-executor-1] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813710991
2023-10-19 22:22:07,540 [grpc-default-executor-0] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813710998
...
2023-10-19 22:22:15,085 [grpc-default-executor-1] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813717929
2023-10-19 22:22:15,092 [grpc-default-executor-0] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813717936
2023-10-19 22:22:15,098 [grpc-default-executor-1] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813717943
2023-10-19 22:22:15,104 [grpc-default-executor-0] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813717950
2023-10-19 22:22:15,114 [grpc-default-executor-1] INFO a.m.c.v.RestControllerVerticle - Created process 2251799813717957
and then I started worker to consume this external tasks (with Thread.sleep of 10ms)
2023-10-19 22:23:53,240 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813710949
2023-10-19 22:23:53,254 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813710956
2023-10-19 22:23:53,264 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813710963
2023-10-19 22:23:53,275 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813710970
2023-10-19 22:23:53,286 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813710977
2023-10-19 22:23:53,296 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813710984
2023-10-19 22:23:53,307 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813710991
2023-10-19 22:23:53,318 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813710998
...
2023-10-19 22:24:03,519 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813717929
2023-10-19 22:24:03,529 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813717936
2023-10-19 22:24:03,539 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813717943
2023-10-19 22:24:03,549 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813717950
2023-10-19 22:24:03,560 [pool-2-thread-1] INFO ai.minte.core.worker.TestWorker - Task consumed: 2251799813717957
and the order of consuming tasks is exact the same as order of creation the processes.
Moreover, here
I found the fragment:
“Whenever a process instance arrives at a service task, a new job is created and pushed to an internal persistent queue within Camunda 8”
Soo…
Am I doing something wrong?
I think maybe I wasn’t able to explain it well… A single-threaded worker means that it will do one thing at a time…
so it will certainly do one processing at a time. When the instance reaches the worker, a job is created and the worker does the fetch and lock to perform the work… but imagine that you started 100 instances with different types of payloads and processing… and imagine that before reaching the worker, you have 5 other Services tasks, some instances will arrive before the others…
imagine that instance number 2 arrives at the worker before instance 1… it will be processed by the worker before instance 1… that is, In this case you have already lost the order guarantee.