Hi , I am using camunda for an application where I am facing heap taking too much memory. Spikes go from 950 MB to 1700MB per pod. TPS is not that high around 1 TPS max.
Heap analysis indicates
‘org.camunda.bpm.engine.impl.cfg.ProcessEngineConfigurationImpl’
One instance of org.apache.ibatis.session.defaults.DefaultSqlSessionFactory loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0xcd80cb68 occupies 4,12,33,400 (36.14%) bytes. The memory is accumulated in one instance of org.apache.ibatis.session.Configuration, loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0xcd80cb68, which occupies 4,12,33,384 (36.14%) bytes.
Taking too much heap.
Is this expected for an application with this traffic? how can further tuning or debugging can be done?
Camunda is being used as embedded engine in a sprint boot application which is using postgres DB on GCP. Application is connecting to postgres using cloudSQL proxy.
Camunda is not using postgres, it is using H2 database internally which might be causing memory issues. Since camunda is embedded in spring boot app it is in same pod as application. Hence H2 in memory database will be in same pod as well.
Camunda is running on default configurations.
Links seem to point to camunda cloud option, any resources for camunda embedded in spring boot app would useful.
Camunda is using H2 database internally has no connection to postgresDB. However the application performs CRUD opertaions on business data residing in a postgres DB. I hope this clarifies.
Hey @cpbpm more investigation confirms that H2 is the culprit.
333 instances of org.h2.mvstore.tx.TransactionStore$TxMapBuilder$TMVMap, loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0x7670399b0 occupy 15,75,76,768 (33.53%) bytes.
One instance of org.h2.mvstore.tx.TransactionStore$TxMapBuilder$TMVMap loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0x7670399b0 occupies 15,07,53,384 (32.07%) bytes.
The memory is accumulated in one instance of org.h2.mvstore.Page$PageReference, loaded by org.springframework.boot.loader.LaunchedURLClassLoader @ 0x7670399b0, which occupies 15,07,52,880 (32.07%) bytes.
So H2 database which is used camunda is causing the issue. Can you help me with the configurations you talked about
Hi @cpbpm , I am a teammate of @varun .
TLDR: How to clean up the database after the process instance execution completes?
I would like to give you more context on our use case.
We are using Camunda as an ephemeral task executor. Once the process instance execution completes(either in success or in failure state), we are saving the state in a separate PostgreSQL DB. Camunda is not connected to this DB.
So, as we already have a separate PostgreSQL DB to manage the long-term persistent state, we have chosen to use the ephemeral H2 DB that runs with the service inside the pod. We think managing a DB just for ephemeral Camunda states is unnecessary and would increase the latency as Camunda would be connecting to a DB that isn’t inside the same pod. Is this correct?
Summarizing the above, how can we delete the history of a process instance as soon as the execution completes(success or failure state)? And, I think the answer to this wouldn’t differ if we are using either H2 or some other recommended DB.