Camunda with file processing

Camunda 8 Topics
1

Hi Team,

I’m attempting to incorporate camunda in my project. I have a requirement here that I receive a csv file from the user. The file may include over 5 lakh records. Each record would necessitate an outbound call, such as Rest, to a third-party system.

How can we accomplish this with Camunda? I’ve seen the sample below: https://github.com/camunda-community-hub/camunda-8-examples/tree/main/large-multi-instance-example. However, if the total number of bucket sizes is large, this will present an issue.

Do we have any possibilities, like in Kafka, to stream everything?

Thanks & Regards,
Raja Gandharaw.

2

Hi @GRajaMca,

why not distribute the work outside and start a process instance for each dataset?

Hope this helps, Ingo

3

Hi @Ingo_Richtsmeier,

If we distribute the work outside, we cannot gather all of the process instances for a particular job.

Assume we have 1000 records in the csv file. How can we group them? How can we indicate how many records were successfully processed?

Here my primary concerns is about the scalability of the job. csv records can be more and more over the period of time.

Thanks & Regards,
Raja Gandharaw

4

Hi @GRajaMca,

for example, you could start the process instances with a process variable containing the value of the CSV file name or any other ID.

You can filter in Operate for process instances with the value of a process variable to get all process instances that are started from a single CSV file.

Hope this helps, Ingo