Unable to Delete Deployments with Cascade=true

Stephen_Bucholtz · January 26, 2017, 8:29pm

We are unable to delete deployments when setting the Cascade option to true.

Our environment:
- Camunda 7.5
- Apache Tomcat
- stand-alone deployment
- SQL database running on Amazon RDS

We have tested deleting deployments from the Cockpit and directly using the REST API from Postman. In both cases, the behavior is the same. The first time we try deleting a deployment, after a minute we receive a Gateway Timeout error, as seen below. (Note that we can successfully delete deployments when Cascade=false.)

Subsequent attempts to delete the deployment result in the following errors in the Catalina log.

26-Jan-2017 17:34:53.002 SEVERE [http-nio-8080-exec-10761] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: ENGINE-03004 Exception while executing Database Operation 'DELETE_BULK deleteExceptionByteArraysByIds {deploymentId=a52dd1cb-e39e-11e6-98f7-0242b5568089}' with message '
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: delete     from         ACT_GE_BYTEARRAY      WHERE ID_ in (     select         JOB_EXCEPTION_STACK_ID_     from         ACT_HI_JOB_LOG     WHERE JOB_EXCEPTION_STACK_ID_ is not null                                                                                 and DEPLOYMENT_ID_ = ?   )
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction'. Flush summary: 
 [
  INSERT UserOperationLogEntryEventEntity[a0c6dbbb-e3ed-11e6-98f7-0242b5568089]
  DELETE_BULK deleteIdentityLinkByProcDef a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteExceptionByteArraysByIds {deploymentId=a52dd1cb-e39e-11e6-98f7-0242b5568089}
  DELETE_BULK deleteExceptionByteArraysByIds {processDefinitionId=a53301ed-e39e-11e6-98f7-0242b5568089}
  DELETE_BULK deleteCaseDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteProcessDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteResourcesByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteDeployment a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteDecisionDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteAuthorizationsForResourceId {resourceId=a52dd1cb-e39e-11e6-98f7-0242b5568089, resourceType=9}
  DELETE_BULK deleteHistoricIdentityLinksByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricIncidentsByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricJobLogByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricJobLogByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteJobDefinitionsByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
]
 org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation 'DELETE_BULK deleteExceptionByteArraysByIds {deploymentId=a52dd1cb-e39e-11e6-98f7-0242b5568089}' with message '
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: delete     from         ACT_GE_BYTEARRAY      WHERE ID_ in (     select         JOB_EXCEPTION_STACK_ID_     from         ACT_HI_JOB_LOG     WHERE JOB_EXCEPTION_STACK_ID_ is not null                                                                                 and DEPLOYMENT_ID_ = ?   )
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction'. Flush summary: 
 [
  INSERT UserOperationLogEntryEventEntity[a0c6dbbb-e3ed-11e6-98f7-0242b5568089]
  DELETE_BULK deleteIdentityLinkByProcDef a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteExceptionByteArraysByIds {deploymentId=a52dd1cb-e39e-11e6-98f7-0242b5568089}
  DELETE_BULK deleteExceptionByteArraysByIds {processDefinitionId=a53301ed-e39e-11e6-98f7-0242b5568089}
  DELETE_BULK deleteCaseDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteProcessDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteResourcesByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteDeployment a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteDecisionDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteAuthorizationsForResourceId {resourceId=a52dd1cb-e39e-11e6-98f7-0242b5568089, resourceType=9}
  DELETE_BULK deleteHistoricIdentityLinksByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricIncidentsByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricJobLogByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricJobLogByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteJobDefinitionsByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
]
	at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:115)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:297)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:283)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:321)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:249)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deleteDeployment(RepositoryServiceImpl.java:108)
	at org.camunda.bpm.engine.rest.sub.repository.impl.DeploymentResourceImpl.deleteDeployment(DeploymentResourceImpl.java:145)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
	at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:159)
	at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:107)
	at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:154)
	at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:92)
	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilterSecure(SecurityFilter.java:67)
	at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilter(SecurityFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:59)
	at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:56)
	at org.camunda.bpm.webapp.impl.security.SecurityActions.runWithAuthentications(SecurityActions.java:38)
	at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:56)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: delete     from         ACT_GE_BYTEARRAY      WHERE ID_ in (     select         JOB_EXCEPTION_STACK_ID_     from         ACT_HI_JOB_LOG     WHERE JOB_EXCEPTION_STACK_ID_ is not null                                                                                 and DEPLOYMENT_ID_ = ?   )
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.delete(DefaultSqlSession.java:165)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeDelete(DbSqlSession.java:187)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.deleteBulk(DbSqlSession.java:201)
	at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:48)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:294)
	... 60 more
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1379)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	... 65 more

26-Jan-2017 17:34:53.034 WARNING [http-nio-8080-exec-10761] null.null org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation 'DELETE_BULK deleteExceptionByteArraysByIds {deploymentId=a52dd1cb-e39e-11e6-98f7-0242b5568089}' with message '
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: delete     from         ACT_GE_BYTEARRAY      WHERE ID_ in (     select         JOB_EXCEPTION_STACK_ID_     from         ACT_HI_JOB_LOG     WHERE JOB_EXCEPTION_STACK_ID_ is not null                                                                                 and DEPLOYMENT_ID_ = ?   )
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction'. Flush summary: 
 [
  INSERT UserOperationLogEntryEventEntity[a0c6dbbb-e3ed-11e6-98f7-0242b5568089]
  DELETE_BULK deleteIdentityLinkByProcDef a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteExceptionByteArraysByIds {deploymentId=a52dd1cb-e39e-11e6-98f7-0242b5568089}
  DELETE_BULK deleteExceptionByteArraysByIds {processDefinitionId=a53301ed-e39e-11e6-98f7-0242b5568089}
  DELETE_BULK deleteCaseDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteProcessDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteResourcesByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteDeployment a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteDecisionDefinitionsByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteAuthorizationsForResourceId {resourceId=a52dd1cb-e39e-11e6-98f7-0242b5568089, resourceType=9}
  DELETE_BULK deleteHistoricIdentityLinksByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricIncidentsByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricJobLogByDeploymentId a52dd1cb-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteHistoricJobLogByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
  DELETE_BULK deleteJobDefinitionsByProcessDefinitionId a53301ed-e39e-11e6-98f7-0242b5568089
]
	at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:115)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:297)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:283)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:321)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:249)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
	at org.camunda.bpm.engine.impl.RepositoryServiceImpl.deleteDeployment(RepositoryServiceImpl.java:108)
	at org.camunda.bpm.engine.rest.sub.repository.impl.DeploymentResourceImpl.deleteDeployment(DeploymentResourceImpl.java:145)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
	at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:159)
	at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:107)
	at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:154)
	at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:92)
	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilterSecure(SecurityFilter.java:67)
	at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilter(SecurityFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:59)
	at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:56)
	at org.camunda.bpm.webapp.impl.security.SecurityActions.runWithAuthentications(SecurityActions.java:38)
	at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:56)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error updating database.  Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
### The error may involve defaultParameterMap
### The error occurred while setting parameters
### SQL: delete     from         ACT_GE_BYTEARRAY      WHERE ID_ in (     select         JOB_EXCEPTION_STACK_ID_     from         ACT_HI_JOB_LOG     WHERE JOB_EXCEPTION_STACK_ID_ is not null                                                                                 and DEPLOYMENT_ID_ = ?   )
### Cause: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.delete(DefaultSqlSession.java:165)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeDelete(DbSqlSession.java:187)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.deleteBulk(DbSqlSession.java:201)
	at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:48)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:294)
	... 60 more
Caused by: java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1074)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4074)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4006)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2155)
	at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1379)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
	at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
	at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
	... 65 more

Here are some statistics on the number of rows in various Camunda DB tables:

ACT_HI_PROCINST - 11,264
ACT_HI_TASKINST - 28,480
ACT_HI_VARINST - 231,684
ACT_HI_ACTINST - 147,771
ACT_HI_DETAIL - 278,185
ACT_HI_JOB_LOG - 25,904
ACT_RE_PROCDEF - 2259
ACT_RE_DEPLOYMENT - 2259
ACT_GE_BYTEARRAY - 34,260

Does this seem like a large number of items in the database? Given these statistics, would you expect any problems deleting a deployment? Do you have any suggestions for how to get this working?

If you need any further data, please let me know.

roman.smirnov · January 27, 2017, 8:04am

Hi @Stephen_Bucholtz,

It looks like you have a deadlock.

Could you try to add an index on JOB_EXCEPTION_STACK_ID_?

create index ACT_IDX_HI_JOB_LOG_STACK_ID on ACT_HI_JOB_LOG(JOB_EXCEPTION_STACK_ID_);

But I am not sure if this will fix that. Maybe you give that a try?

As an alternative, you have to delete first each process instance manually, and then you could delete the whole deployment. Please note, that with 7.6 it is possible to delete process instances in a bulk manner.

Cheers,
Roman

aakhmerov · January 27, 2017, 8:44am

Hi @Stephen_Bucholtz,

which database are your running in RDS? Is Multi A-Z enabled?

Cheers,
Askar

Stephen_Bucholtz · January 27, 2017, 1:34pm

Hi @aakhmerov,

We are using the Amazon Aurora DB engine. It is NOT multi-AZ. The Class is db.r3.xlarge.

aakhmerov · January 27, 2017, 1:50pm

@Stephen_Bucholtz,

one more question, even if multi az is not enabled, is it a clustered setup of some sort? Or replicated in some way.

additionally could you please show us your jdbc connection settings.

Cheers,
Askar.

Stephen_Bucholtz · January 27, 2017, 3:55pm

@aakhmerov,

There is no clustering or replication of the database. It is a single DB instance. (Although in the future we will likely use a clustered setup).

We are using Camunda in a docker container. So the jdbc connection settings are configured via the docker setup. Here is the resulting jdbc configuration as shown in the server.xml file on the running container:

Resource name="jdbc/ProcessEngine" auth="Container" type="javax.sql.DataSource" factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" uniqueResourceName="process-engine" driverClassName="com.mysql.jdbc.Driver" url="jdbc:mysql://<redacted>?autoReconnect=true" username=<redacted> password=<redacted> maxActive="20" minIdle="5"/>

Note: sensitive settings have been replaced in the above with “redacted”.

Stephen_Bucholtz · January 30, 2017, 4:17pm

@roman.smirnov,

I have tried your suggestion of adding the index on JOB_EXCEPTION_STACK_ID. So far, this appears to be working. I can now delete the deployments which previously resulted in a failure. Should this index be permanently added to the DB Schema?

roman.smirnov · February 1, 2017, 8:09am

Hi @Stephen_Bucholtz,

Thanks for trying that index.

I will create a bug issue for that and we will investigate it, whether the index should added to the DB schema.

Cheers,
Roman