Help with designing and implementing of scheduling multiple process instances

I have the following scenario

I have to do create multiple instances(count determined by call to DB) of 5 different business processes which are modelled in separate workflows

For illustration naming them

  1. Workflow 1
  2. Workflow 2
  3. Workflow 3
  4. Workflow 4
  5. Workflow 5

I will have to start multiple instances of the above workflows based on some data in the database

I will also need a parent workflow(to manage creating the above workflows) which will essentially do the below steps
Note: This workflow should never die, unless stopped externally. I want this workflow as a scheduler to create instances of other workflows. This workflow will be started when the container starts

Step1. Read data from the database using a REST API in a service task.

. The data from step1 will tell the following information

Workflow 1 -> create 5 instances 
Workflow 2 -> create 2 instances 
Workflow 3 -> create 1 instances 
Workflow 4 -> nothing yet to create
Workflow 5 -> nothing yet to create

Note: We have some thresholds set, which ensures not many PI are being created by this process

Step 2: I am trying to start these process instances using the java API of the RuntimeService in the next service task

runtimeService.startProcessInstanceByKey("workflow1");. * 5 times
runtimeService.startProcessInstanceByKey("workflow2");. * 2 times
runtimeService.startProcessInstanceByKey("workflow3");. * 1 time
Not starting workflow 4 and workflow 5 as there is no need in this iteration

I am calling this number of times based on the data in step2

I am expecting that all these process instances will be started async
The purpose of this workflow is only to start the other workflows

Step 3 . After i have finished starting all the process instances for workflow1 to workflow5
I am doing some process cleanup and sending the flow back to Step 1

This keeps going in a loop and does the same steps again

I observed that the execution of the workflows(workflow1 to workflow5) are not triggering at all . until the main workflow is stopped

I have tried different mechanism but unsuccessful in achieving the use case

What is the best approach to model this? I am not sure what has to be done to achieve this.Can someone please help with the same?

I am using the spring boot camunda starter to do the same

I have attached the master workflow which has 3 service tasks
1.Get Data (explained earlier)
2.Schedule Workflow(Start child workflows)
3.Cleanup

Hi,

You seem to have created an endless loop. Have you considered using a timer start event? Hence you could use a cycle timer and run the process say every minute…

Regards

Rob

Thanks @Webcyberrob
I can include a timer, but not able to understand the fact that , the child processes do not start executing until the main process is stopped(through a timer)
How are the aysnc jobs executed ? i have read the documentation and looks like the JOB executor will execute them
For example , i start 10 sub processes async, are they all executed in parallel in separate threads?

How can i configure all this ?

Need to understand the following

  1. Hoe does the Job executor decide which have to be executed and what order
  2. How many threads does it allocate
  3. If the child processes are executing and the main workflow comes back after the time finishes, what happens to all the child processes?

Hi,

to answer some of your questions;

How does the Job executor decide which have to be executed and what order?

The job executor will select jobs based on their due date, ie ready to run now. The order can be influenced by configuration items. Details can be found here [1]

How many threads does it allocate?

This could depend on the way the engine is deployed. In an app container such as Tomcat, the number of threads is the number of threads enabled in the executor pool. Detail for Tomcat can be found here [2]

If the child processes are executing and the main workflow comes back after the time finishes, what happens to all the child processes?

This depends on how you start the child processes. If you use a call activity, the master process call activity will block until the child returns. Your sample code is not using a call activity. You are starting independent process instances with no affinity to the master process. If the child processes use asynchronous continuations, then your mast process start thread will run up until the async continuation. If you want to use the call activity, you may need to research multi-instance activities here [3].

BTW - a quick review of the Springboot documentation indicates that the job executor is configured with 1 thread by default. Hence if your master process consumes this thread, no other process instances will run until you master process finishes or reaches an async continuation. (Note I am assuming that you master process is started with an async continuation. Sprinboot config docs can be found here [4]

regards

Rob

[1] https://docs.camunda.org/manual/7.7/user-guide/process-engine/the-job-executor/
[2] https://docs.camunda.org/manual/7.7/user-guide/runtime-container-integration/tomcat/#core-pool-size
[3] https://docs.camunda.org/manual/7.7/reference/bpmn20/tasks/task-markers/#multiple-instance
[4] https://camunda.github.io/camunda-bpm-spring-boot-starter/docs/current/index.html#_default_configurations

Thanks @Webcyberrob for such a detailed explanation.
I will look at how i can configure the same
Appreciate your effort and time for the same

I cannot use call activity , as they should not block the master workflow. I am using the master workflow as a scheduler, which keeps polling for data after every 1 minute and spawn new instances of child workflows

I am starting all the child processes with async continuation

If i configure the job executor thread pool to 6 for example, i assume this will be the flow

  1. The master workflow kicks off on start up
  2. Gets data for starting the following
    Workflow 1 - 2 instances
    Workflow 2 - 3 instances
    Workflow 3 - 2 instances
    If i am using async continuation to start these workflows , they will be added as jobs and will be executed by the Job executor with the threads available (6) in my case
  3. When the master workflow returns after 1 minute , will it get a turn only if there are any available threads(will it be blocked by the executing child processes)? Should i also configure the master workflow with async continuation ? I am not clear on this bit so it is not blocked

Do you think this approach is scalable ?