Hi,
I have been experimenting a lot with Camunda and what is close to what we need our purposes is using Java delegates.
I also started looking into Zeebe. Though I am a bit confused atm how these jobs, workers etc work?
Is it like External Task in camunda ? As External task was not a good fit for us because it was sequential or blocking in nature. So is Zeebe working on the same notion? We have multiple workers and they are work sequentially accepting jobs?
Yes, it is the external task worker pattern.
I’m not sure what you mean by sequential or blocking - that is a function of the BPMN process: if you put tasks in parallel, they can be executed in parallel. If they are in series, they will be executed in series.
That is a function of the orchestration.
Josh
Zeebe works a lot better on parallel and multitask patters, because do not have camunda’s optimistic lock problem.
And yes, every task is locking for lock interval when fetched, and is unlocking on commit.
Thanks for the replies. And apologies on replying late.
I have explained my answer about why it’s kinda of a bummer this external task pattern in the vanilla Camunda forum .org
So our usually process is that we have an event coming to the system which results in 100 to 1000 process running simultaneously. So if we start implementing each task as an external task then in our case where the process can have 50 to 100 tasks in a process whcih would mean we need to implement as many external tasks?
Then if they are being executed sequentially then the only way our 100 to 1000 process can run simultaneously is by somehow creating quite many or as many external tasks?
That’s the reason we started to look into Camunda Delegates.
But could you perhaps may be throw some more light if my understanding is correct about the way how external task work and might be an issue in our case?
Hmm I did not understand the locking pattern now. So each task gets locked till it gets fully executed no? That’s why I have been thinking they are sequential in nature in receiving tasks to be performed
Just look and examples:
GitHub - MaximMonin/camunda-plugins - camunda external pattern with stress tests
almost the same with zeebe:
GitHub - MaximMonin/zeebe-docker: Zeebe in Docker also with stress tests
Thanks.
I will have still read how these workers work and scale. Any good links?
Also my original question is here What is the best way for parallel process and task execution - Process Engine - Camunda Platform Forum
And the recommendation there was to have multiple instance of workers running. For which I chose to check javadelegates
But with your example of workers tasks , I see it does not have to say it’s completed at the end of of the function ‘ externalTaskService!!.complete‘ so I assume 1 worker can process multiple requests parallely?
sure, in these project 1 worker uses max 200 parallel tasks
and many workers can be setuped on different hosts to execute code
make sure increase database connection for camunda to 500+, because every commit and task that works in parallel use it own db connection
for zeebe it is not applied, check maxJobsToActivate: 200
You could write a single worker that can service 50 different task types, and that has 15 threads and can execute 15 jobs (task instances) in parallel. Then you run two instances of that worker, and you have a parallel capacity of 30.
Or you could write it in Node.js, using a single thread and the event loop, and give it a capacity of 100, and run one instance, and you have a parallel capacity of 100.
I recommend this video to understand how the Zeebe engine works: https://www.youtube.com/watch?v=JbXKgQQmukE
Hey Sherry. I recently wrote a blog post (https://blog.bernd-ruecker.com/writing-good-workers-for-camunda-cloud-61d322cad862) that looks at how to scale workers - if I don’t get the question wrong, using the reactive way of programming your worker code should also help you parallelize things (and you can still use a thread pool additionally). Or did I misunderstand the problem?
Best
Bernd
Thank you @berndruecker and others. 