camunda 7 process engine taking too much memory

Camunda 7 Topics Discussion & Questions
1

Hi , I am using camunda for an application where I am facing heap taking too much memory. Spikes go from 950 MB to 1700MB per pod. TPS is not that high around 1 TPS max.

Heap analysis indicates
‘org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl’
One instance of org.apache.ibatis.session.defaults.DefaultSqlSessionFactory loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0xcd80cb68 occupies 4,12,33,400 (36.14%) bytes. The memory is accumulated in one instance of org.apache.ibatis.session.Configuration, loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0xcd80cb68, which occupies 4,12,33,384 (36.14%) bytes.

Taking too much heap.

Is this expected for an application with this traffic? how can further tuning or debugging can be done?

2

Couple of questions.

  1. Would you please share your environment details.
  2. What is your backend database? Is this running on same pod or separate pod.
  3. Have you tuned your environment or running with default configuration?

There are couple of blogs, you can refer for Camunda on Kubernetes setup.

  1. https://camunda.com/blog/2019/06/camunda-bpm-on-kubernetes/
  2. https://camunda.com/blog/2019/06/camunda-bpm-with-session-manager/
1 Like
3
  1. Camunda is being used as embedded engine in a sprint boot application which is using postgres DB on GCP. Application is connecting to postgres using cloudSQL proxy.
  2. Camunda is not using postgres, it is using H2 database internally which might be causing memory issues. Since camunda is embedded in spring boot app it is in same pod as application. Hence H2 in memory database will be in same pod as well.
  3. Camunda is running on default configurations.

Links seem to point to camunda cloud option, any resources for camunda embedded in spring boot app would useful.

4

Your points 1 and 2 are confusing.

If you are running with default configurations, you need tuning . If you could clarify point 1 and 2, then we can suggest the possible solution.

5

Camunda is using H2 database internally has no connection to postgresDB. However the application performs CRUD opertaions on business data residing in a postgres DB. I hope this clarifies.

6

Hey @cpbpm more investigation confirms that H2 is the culprit.

333 instances of org.h2.mvstore.tx.TransactionStore$TxMapBuilder$TMVMap, loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0x7670399b0 occupy 15,75,76,768 (33.53%) bytes.

One instance of org.h2.mvstore.tx.TransactionStore$TxMapBuilder$TMVMap loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0x7670399b0 occupies 15,07,53,384 (32.07%) bytes.

The memory is accumulated in one instance of org.h2.mvstore.Page$PageReference, loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0x7670399b0, which occupies 15,07,52,880 (32.07%) bytes.

So H2 database which is used camunda is causing the issue. Can you help me with the configurations you talked about

7

H2 is recommended for local development. Consider integrating Camunda with supported databases.

8

Hi @cpbpm , I am a teammate of @varun .
TLDR: How to clean up the database after the process instance execution completes?
I would like to give you more context on our use case.

  1. We are using Camunda in embedded mode inside a SpringBoot Application - camunda-bpm-spring-boot-starter.
  2. We are using Camunda as an ephemeral task executor. Once the process instance execution completes(either in success or in failure state), we are saving the state in a separate PostgreSQL DB. Camunda is not connected to this DB.
  3. So, as we already have a separate PostgreSQL DB to manage the long-term persistent state, we have chosen to use the ephemeral H2 DB that runs with the service inside the pod. We think managing a DB just for ephemeral Camunda states is unnecessary and would increase the latency as Camunda would be connecting to a DB that isn’t inside the same pod. Is this correct?
  4. Summarizing the above, how can we delete the history of a process instance as soon as the execution completes(success or failure state)? And, I think the answer to this wouldn’t differ if we are using either H2 or some other recommended DB.
9

You need to go through the Camunda docs {History and Audit Event Log | docs.camunda.org }

and apply the settings. Based on your question, ACTVITIY level would be fine.

Regarding running H2, it’s not recommended for production application. Consider tuning your kubernetes cluster for production environment.

1 Like