High Velocity Messages Overwhelm Camunda Server Database

mppfor_manu · October 25, 2016, 1:57am

When I fire a large number (essentially a continuous stream) of start requests at a Camunda server for a very simple service (basically two tasks that each print a small log entry, with a 5 second delay inserted in the class associate with the first task, it throws errors and “drops” requests (i.e. it never processes them). I’ve included a stack trace below which tells me that Camunda can’t get enough connections to the back end database to process the requests. Rather than simply queuing them up, it discards them.

Is this the expected behavior? Have we already hit the wall on the server? How many transactions can we reasonably expect Camunda to handle?

My configuration:

12 Core VMware Guest
64 GB RAM allocated
SAN attached storage
Camunda 7.5.3-ee
Red Hat EL 6.8 Linux
WildFly 10.0.0-FINAL application server
MySQL (5.7.13 Commercial) on separate server, but the same subnet as persistence database
Connection pooling:

Min Pool Size = 10
Initial Pool Size = 10
Max Pool Size = 100
Prefill = True
Allocation Retry = 10
Allocation Retry Wait Millis = 500

We operate in an environment with high message volume and we cannot drop even a single message. What can be done about this?

STACK TRACE:

2016-10-24 18:56:45,350 WARNING [ExceptionHandler] (default task-76) org.camunda.bpm.engine.ProcessEngineException: Process engine persistence exception
at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.rethrow(CommandInvocationContext.java:148)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:213)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
at org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:58)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
at org.camunda.bpm.engine.impl.AbstractQuery.singleResult(AbstractQuery.java:130)
at org.camunda.bpm.engine.rest.impl.ProcessDefinitionRestServiceImpl.getProcessDefinitionByKey(ProcessDefinitionRestServiceImpl.java:49)
at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.resteasy.core.ResourceLocatorInvoker.createResource(ResourceLocatorInvoker.java:79)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:106)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:395)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:202)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:221)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:85)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:60)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:131)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ibatis.exceptions.PersistenceException:

Error querying database. Cause: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/ProcessEngine

The error may exist in org/camunda/bpm/engine/impl/mapping/entity/ProcessDefinition.xml

The error may involve org.camunda.bpm.engine.impl.persistence.entity.ProcessDefinitionEntity.selectProcessDefinitionsByQueryCriteria_mysql

The error occurred while executing a query

Cause: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/ProcessEngine

at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:111)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:102)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectList(DbSqlSession.java:87)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectListWithRawParameter(DbEntityManager.java:171)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:163)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:155)
at org.camunda.bpm.engine.impl.persistence.entity.ProcessDefinitionManager.findProcessDefinitionsByQueryCriteria(ProcessDefinitionManager.java:95)
at org.camunda.bpm.engine.impl.ProcessDefinitionQueryImpl.executeList(ProcessDefinitionQueryImpl.java:303)
at org.camunda.bpm.engine.impl.AbstractQuery.evaluateExpressionsAndExecuteList(AbstractQuery.java:186)
at org.camunda.bpm.engine.impl.AbstractQuery.executeSingleResult(AbstractQuery.java:207)
at org.camunda.bpm.engine.impl.AbstractQuery.execute(AbstractQuery.java:167)
at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:104)
… 50 more
Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/ProcessEngine
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146)
at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:66)
at org.apache.ibatis.transaction.managed.ManagedTransaction.openConnection(ManagedTransaction.java:87)
at org.apache.ibatis.transaction.managed.ManagedTransaction.getConnection(ManagedTransaction.java:61)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:279)
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:72)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:59)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:137)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:96)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:77)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:108)
… 62 more
Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/ProcessEngine
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:656)
at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:429)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:747)
at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138)
… 73 more
Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (0 [ms])
at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:564)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:708)
at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:607)
at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:626)

aakhmerov · October 25, 2016, 8:35am

Hi @mppfor_manu,

could you attach your process definition please? I assume you have to have exception handling in your process.

Cheers,
Askar

mppfor_manu · October 25, 2016, 12:46pm

Askar,

These are processes that we built specifically to test “hot” deployment and versioning in the Camunda server itself. They don’t really do anything, so there’s no exception handling.

My concern here is how Camunda handles really heavy loads or spikes. I have attached the BPMN code. The Java for the first class is:

package com.att.gcs.bizops.cop.core.template.messagehandler;

import org.camunda.bpm.engine.delegate.DelegateExecution;
import org.camunda.bpm.engine.delegate.JavaDelegate;

public class printCounter implements JavaDelegate {

public void execute(DelegateExecution execution) {

	System.out.println("Version 0.0.4 - Counter Value is " + execution.getVariable("timestamp").toString() + "|" + execution.getVariable("counter").toString());
	try {
		Thread.sleep(5000);
	} catch (InterruptedException e) {
		// TODO Auto-generated catch block
		e.printStackTrace();
	}

}

}

The Java for the second class is:

package com.att.gcs.bizops.cop.core.template.messagehandler;

import org.camunda.bpm.engine.delegate.DelegateExecution;
import org.camunda.bpm.engine.delegate.JavaDelegate;

public class printAnother implements JavaDelegate {

public void execute(DelegateExecution execution) {

	System.out.println("Version 0.0.4 - Another Value is " + execution.getVariable("timestamp").toString() + "|" + execution.getVariable("counter").toString() + ".1");

}

}

BPMN:

printCounter1.bpmn (4.2 KB)

praveen_buya · September 14, 2018, 11:55pm

Hi Team,

Even we are running into this issue , is there any solution available for this issue.

we are using wildfly 10.1.0 and camunda 7.8.0 and mssql 13.0

Webcyberrob · September 15, 2018, 12:37am

Hi,
My advice would be to examine and tune the resources allocated through the entire request pipeline. Consider the request pipeline as;

App Server->DB Server

However the app server really has two resources, foreground request threads and a DB connection pool. hence it more likely looks like;

Request Handler Pool -> DB Connection Pool -> DB Server

Thus you want to ensure that the DB server is configured to accept at least as many connections as the DB Connection Pool. In addition, you want to ensure the DB Connection pool has at least as many connections as request handlers.

The next item to consider is the job executor. Its threads will also use connections from the DB connection pool, hence your DB connection pool will need additional connections for job executor threads.

On the assumption you get resource management appropriately sized through all the layers, you still need to ensure that the DB node is sized to handle the expected load.

Note that if the system is at saturation, allocating more connections etc may actually reduce throughput as the system may start to thrash. In this case, its probably better to refuse client connections and force the client to retry.

regards

Rob

mppfor_manu · September 28, 2018, 12:14pm

The other way to mitigate this is to turn process history levels down or completely off if you can afford to do so.

StephenOTT · September 28, 2018, 12:46pm

Check out: https://github.com/StephenOTT/camunda-spring-boot-rabbitmq-messaging. You can route all your messages into a queue and have them consumed by a worker. Thus preventing overload. In this example I have reused the rest API format as the content of the message. So you just need to submit your regular Json POST for a message and the rest is taken care of