I am new to Camunda and using Camunda platform 7 to create a multi instance workflow.
I am trying to test how many concurrent processes can be run through a Camunda Workflow.
My workflow has a Script Task that calls a shell script that does nothing but sleep for 20 sec. I am changing the loop cardinality values for a multi instance script task to check the maximum level of parallelism that we can achieve through Camunda. I have also added a catch event next to the Script Task. This is what the workflow looks like:
I am able to run around 800 parallel tasks after which I get the error: java.lang.OutOfMemoryError: unable to create new native thread
On running the script manually in the server without Camunda, we are able to run till 2000 parallel tasks. Is there any configuration that can be done in Camunda to change the thread limit?
I ran the same script manually on the server without using Camunda workflow and I could get a greater number of parallel processes (2000) as compared to running it through the workflow (800). Is there a reason why running the script through Camunda allows fewer parallel processes?
Our max thread limit set at server is 126749.
Also, Is it possible to connect over teams or google meet so that we can show you the problem that we are facing?
We have increased the following limits in our application server but we still don’t see an improvement in the number of concurrent processes using our parallel multi-instance workflow:
Could you please specify how you start your process? and What distribution you are using (Camunda Run, Tomcat …)?
I believe that the list of parameters you shared is related to Linux and has nothing to do with it.
And keep in mind that there are two types of threads executing processes in the Camunda platform.
Two types of threads execute processes in Camunda BPM:
Application threads
Job Executor threads
Application threads are threads that call the Camunda Java API from an application context. For example, if we have a REST endpoint that starts a process instance, the HTTP worker thread calls RuntimeService#startProcessInstanceByKey and runs the process instance until wait states are reached. The calling thread receives immediate feedback: If the call fails, we can catch and handle an exception. If the call does not return, we can debug directly into our process’s delegate code and see where execution gets stuck.
The second type of threads is job execution threads. A job in the process engine is a piece of work that needs to be done at some time in the future. In the case of BPMN timer events, the BPMN model defines this point in time. In the case of asynchronous continuations, the time is as soon as possible. Job execution threads take care of this work and are managed by the job executor. Being rather detached from an application, it is sometimes hard to understand when job execution happens or, more importantly, when it does not happen or when it is slow.
We have hosted this distribution on our linux server.
The BPMN workflow is deployed to the REST endpoint pertaining to our host server and after which we start a new process instance to run the workflow.
The workflow has a script task with an inline Groovy script that is executing a shell script present in our host server.
Script Task details:
• Script format: Groovy
• Script Type: Inline Script
• Script: [/test_script.sh].execute()
Since we are new to Camunda, we are not sure how to check the process’s delegate code. Could you provide more details on this so we that we can check where exactly our execution is stuck?
The error you are getting happens due to your sdk internal system running out of memory. this happens with all java applications at some point, here are some older infos, which are kind of still valid Java: What is the limit to the number of threads you can create?
Basically your standard java programm is limited to 4Gb of resources. in This case these resources are used by camunda and everything that is started from camunda (as far as i know). What you have to change is the call of the camunda main application and add more rsources there. i dont know if camunda supports this via administration or config though
