How can we perform the batch jobs in camunda ..as i have multiple records which are waiting for due date say 10k+ records ..i want them to process in a batch of say 500 / 15min of time span , how can we do it in cmaunda and java application

Camunda 8 Topics
1

How can we perform the batch jobs in camunda …as i have multiple records which are waiting for due date say 10k+ records …i want them to process in a batch of say 500 / 15min of time span , how can we do it in camunda and java application.

please note …we cannot use the dashboard batch option to run it because we have to perform at certain level in middle of the flow that is at due dates (as the number of records are waiting at due date are huge for that particular date). how can we do it is there any option we can use here with respect to camunda or database job table or java here please help

2

any help around this?

3

@Niall @hassang please help

4

You can use a multi-instance subprocess to get and process the data your want.

5

Thanks for reply @Niall , we cant use it in Time- > Date section for due date right we might need to create the service task/user task before the timer section right to use the multi instance either parallel or sequential ?

6

any update? @Niall could you pls help

7

any update here

8

Hi @vinodgaikwad

I don’t fully understand yet what you’re trying to acheive.
Can you answer some questions.

  1. What’s the limitation on fetching records exactly?
  2. Once fetched can the records be processes in parallel?
  3. You mentioned a 15 min time frame - is that limited to how often you can poll for new records?
9

HI @Niall

  1. we have more than 15k + records at table called "Camunda.act_ru_job "which are waiting for due to to process them…for same day due date there will be 10k+ records in one go. so we wanted to avoid this records to process them in one go as it causing heap memory n GC overhead.

so we wanted to split it across the different configurable batch size and configurable time like within first 15m we can process about 500 records n remaining from resp next 15 -15 minutes time frame with 500 records each.

so to fetch the records do we use the GET /process-instance this api with process definition key as request param so will i get the all the records of 15k ?

n then i can process them either parallel or sequential i am not sure here what’s best in these two for my use case?

records are lying inside the database we need to retrieve them n process them in batch wise to avoid memory issues

10

So, you’re using Camunda 7 then?

Your issue is that you have lots of throughput in your current Camunda 7 instance and you cant process them all because it’s causing memory issues.

Tell me - are you running a single Camunda 7 engine here? Or have you clustered the engine in any way?

11

yes we are with camunda 7 with openshift cluster instance

12

how can we resolve this now?

13

Seems like this problem would be solved by simply increasing the size of the cluster.

14

we tried with increasing the heap memory of it but still facing the same issue

15

Can you let me know how many instances of your camunda 7 engine you are running in the cluster - and also let me know what the process engine settings are?

16

we have 2 pods instances in the cluster @Niall which are running

17

And the engine settings? Have you tried to tweak them at all to improve performance. This page might be able to help you understand what you could do to improve the performance.