Hello Camunda Community,
We are running a multitenant application using Camunda BPM version 7.15. Recently, we have been experiencing performance degradation, application slowness, and connection leaks, especially during load testing with 100 concurrent users.
Issue Observed
During profiling and heap dump analysis with JProfiler, we noticed that a significant number of connections were not being released properly, leading to leaks. The stack traces for the leaked connections consistently point to iBATIS-related dependencies within Camunda’s internals. These iBATIS components are also taking up a substantial amount of heap space.
Sample Connection Leak Stack Trace (JProfiler) :
org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(java.lang.String, java.lang.Object)
org.camunda.bpm.engine.impl.db.sql.DbSqlSession.lambda$selectById$2(java.lang.String, java.lang.String)
org.camunda.bpm.engine.impl.util.ExceptionUtil.doWithExceptionWrapper(java.util.function.Supplier)
org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectById(java.lang.Class, java.lang.String)
org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectById(java.lang.Class, java.lang.String)
org.camunda.bpm.engine.impl.persistence.entity.TaskManager.findTaskById(java.lang.String)
org.camunda.bpm.engine.impl.cmd.CompleteTaskCmd.execute(org.camunda.bpm.engine.impl.interceptor.CommandContext)
org.camunda.bpm.engine.impl.cmd.CompleteTaskCmd.execute(org.camunda.bpm.engine.impl.interceptor.CommandContext)
org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(org.camunda.bpm.engine.impl.interceptor.Command)
org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(org.camunda.bpm.engine.impl.interceptor.Command)
org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(org.springframework.transaction.TransactionStatus)
org.springframework.transaction.support.TransactionTemplate.execute(org.springframework.transaction.support.TransactionCallback)
org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(org.camunda.bpm.engine.impl.interceptor.Command)
org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(org.camunda.bpm.engine.impl.interceptor.Command)
org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(org.camunda.bpm.engine.impl.interceptor.Command)
org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(org.camunda.bpm.engine.impl.interceptor.Command)
org.camunda.bpm.engine.impl.TaskServiceImpl.complete(java.lang.String, java.util.Map)
org.camunda.bpm.engine.impl.TaskServiceImpl.complete(java.lang.String)
com.rrssi.ient.camunda.service.CamundaServiceImpl.updateTask(java.lang.String, java.util.Map, boolean, boolean)
com.rrssi.ient.camunda.service.CamundaServiceImpl$$EnhancerBySpringCGLIB$$af41c28f.updateTask(java.lang.String, java.util.Map, boolean, boolean)
com.rrssi.ient.camunda.utils.service.CamundaUtilsServiceImpl.createOrUpdateTask(com.rrssi.ient.common.dto.ApplicationDTO, java.lang.String, com.rrssi.ient.common.formio.dto.ExecuteRequestDTO, boolean, boolean)
Service Logs Sample :
WARN Connection has been abandoned PooledConnection[org.postgresql.jdbc.PgConnection@712a3e50]:
java.lang.Exception
at org.apache.tomcat.jdbc.pool.ConnectionPool.getThreadDump(ConnectionPool.java:1163)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:816)
…
at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection
Heap
Configuration Details
application.yml (excerpt):
datasource:
url: jdbc:postgresql://localhost/XXXXX?useSSL=false&useUnicode=true
username: postgres
password: postgres
driverClassName: org.postgresql.Driver
tomcat:
initial-size: 0
min-idle: 0
max-idle: 0
validation-query: SELECT 1
test-while-idle: true
test-on-borrow: true
time-between-eviction-runs-millis: 90000
min-evictable-idle-time-millis: 90000
camunda:
min-connections.per.tenant: 5
max.connections.per.tenant: 50
max.idletime.in.secs: 30
max.lifetime.in.secs: 60
camunda.bpm.job-execution:
core-pool-size: 10
max-pool-size: 50
queue-capacity: 100
Datasource Java Configuration (excerpt):
p.setTestWhileIdle(true);
p.setTestOnBorrow(true);
p.setValidationQuery(“SELECT 1”);
p.setTimeBetweenEvictionRunsMillis(30000);
p.setMaxIdle(minConnections);
p.setMaxActive(maxConnections);
p.setInitialSize(minConnections);
p.setMaxWait(60000);
p.setRemoveAbandonedTimeout(60);
p.setMinEvictableIdleTimeMillis(60000);
p.setMinIdle(minConnections);
p.setLogAbandoned(true);
p.setRemoveAbandoned(true);
p.setJdbcInterceptors(
“org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;” +
“org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer;” +
“org.apache.tomcat.jdbc.pool.interceptor.ResetAbandonedTimer;” +
“org.apache.tomcat.jdbc.pool.interceptor.SlowQueryReport(threshold=10000)”);
Request for Help
- Has anyone experienced similar connection leak issues tied to iBATIS within Camunda 7.x?
- Are there recommended best practices for tuning database connection pools in a multitenant Camunda setup?
- Is upgrading to a newer Camunda version known to resolve such leaks or improve memory utilization in similar scenarios?
Any guidance or suggestions would be greatly appreciated.
Thank you!