Deadlock when completing external task

Camunda 7 Topics Discussion & Questions
1

Hi ,

When I complete an external task a dead lock occurs. Please help.

06-Apr-2018 01:53:45.912 WARNING [http-nio-8282-exec-7] org.camunda.bpm.engine.rest.exception.ProcessEngineExceptionHandler.toResponse org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation ‘DELETE ExecutionEntity[fc11661b-3977-11e8-ab47-001c42968f92]’ with message ’

Error updating database. Cause: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

The error may involve org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.deleteExecution-Inline

The error occurred while setting parameters

SQL: delete from ACT_RU_EXECUTION where ID_ = ? and REV_ = ?

Cause: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.’. Flush summary:

[
INSERT AuthorizationEntity[fe26ff83-3977-11e8-ab47-001c42968f92]
INSERT HistoricTaskInstanceEventEntity[fe224485-3977-11e8-ab47-001c42968f92]
INSERT HistoricIdentityLinkLogEventEntity[fe26ff84-3977-11e8-ab47-001c42968f92]
INSERT HistoricExternalTaskLogEntity[fe1b1888-3977-11e8-ab47-001c42968f92]
INSERT HistoricActivityInstanceEventEntity[Task_10cev7g:fe224484-3977-11e8-ab47-001c42968f92]
INSERT TaskEntity[fe224485-3977-11e8-ab47-001c42968f92]
DELETE ExternalTaskEntity[fc11662b-3977-11e8-ab47-001c42968f92]
DELETE VariableInstanceEntity[fc11661d-3977-11e8-ab47-001c42968f92]
DELETE VariableInstanceEntity[fc11661f-3977-11e8-ab47-001c42968f92]
DELETE VariableInstanceEntity[fc116621-3977-11e8-ab47-001c42968f92]
DELETE VariableInstanceEntity[fc116623-3977-11e8-ab47-001c42968f92]
DELETE VariableInstanceEntity[fc116625-3977-11e8-ab47-001c42968f92]
DELETE VariableInstanceEntity[fc116627-3977-11e8-ab47-001c42968f92]
DELETE VariableInstanceEntity[fc116629-3977-11e8-ab47-001c42968f92]
UPDATE ExecutionEntity[fc1165e7-3977-11e8-ab47-001c42968f92]
DELETE ExecutionEntity[fc11661b-3977-11e8-ab47-001c42968f92]
UPDATE HistoricActivityInstanceEventEntity[Task_1kfmj3p:fc11661c-3977-11e8-ab47-001c42968f92]
]
at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:115)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:305)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:281)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:203)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:132)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
at org.camunda.bpm.engine.impl.ExternalTaskServiceImpl.complete(ExternalTaskServiceImpl.java:48)
at org.camunda.bpm.engine.rest.sub.externaltask.impl.ExternalTaskResourceImpl.complete(ExternalTaskResourceImpl.java:106)
at sun.reflect.GeneratedMethodAccessor371.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:159)
at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:107)
at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:154)
at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:92)
at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
at org.jboss.resteasy.core.SynchronousDispatcher.invokePropagateNotFound(SynchronousDispatcher.java:169)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:212)
at org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.doFilter(FilterDispatcher.java:59)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.camunda.bpm.engine.rest.filter.EmptyBodyFilter.doFilter(EmptyBodyFilter.java:95)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.ibatis.exceptions.PersistenceException:

Error updating database. Cause: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

The error may involve org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.deleteExecution-Inline

The error occurred while setting parameters

SQL: delete from ACT_RU_EXECUTION where ID_ = ? and REV_ = ?

Cause: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.

at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:154)
at org.apache.ibatis.session.defaults.DefaultSqlSession.delete(DefaultSqlSession.java:165)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.executeDelete(DbSqlSession.java:188)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.deleteEntity(DbSqlSession.java:173)
at org.camunda.bpm.engine.impl.db.AbstractPersistenceSession.executeDbOperation(AbstractPersistenceSession.java:45)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:303)
... 47 more

Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Transaction (Process ID 67) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction.
at com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:258)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1535)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:467)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:409)
at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:7151)
at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:2478)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:219)
at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:199)
at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.execute(SQLServerPreparedStatement.java:393)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:44)
at org.apache.ibatis.executor.statement.RoutingStatementHandler.update(RoutingStatementHandler.java:69)
at org.apache.ibatis.executor.SimpleExecutor.doUpdate(SimpleExecutor.java:48)
at org.apache.ibatis.executor.BaseExecutor.update(BaseExecutor.java:105)
at org.apache.ibatis.executor.CachingExecutor.update(CachingExecutor.java:71)
at org.apache.ibatis.session.defaults.DefaultSqlSession.update(DefaultSqlSession.java:152)
… 52 more

2

Hi @deepti,

Deadlocks can have a lot of reasons. Some simple things to check:

Cheers,
Thorben

3

Hi Throben,

I have run the below script :

ALTER DATABASE [xx]
SET ALLOW_SNAPSHOT_ISOLATION ON

ALTER DATABASE [xx]
SET READ_COMMITTED_SNAPSHOT ON

and I am using mssql-jdbc-6.2.1.jre7.jar driver on my sql server version : 12.0.2000.8
which is compatible according to this link
https://docs.microsoft.com/en-us/sql/connect/jdbc/microsoft-jdbc-driver-for-sql-server-support-matrix

But I am still facing the deadlocks.

Please help .

4

Please find screen shot of my database version.sqlversion

5

Please find screen shot of my jdbc driver.jdbcdriver

6

Ok, looks good to me so far.

When there is a deadlock, you have two transactions that each hold a write lock that the other tries to acquire. We need to identify what the transactions are doing (i.e. which entities do they update/delete) in order to find the root cause of the deadlock. Furthermore, the stacktrace shows us which operations are performed by one of the transactions (the one that was rolled back), so we have an idea which write locks it held at that time. Now we have to find out what the second transaction did and which statements it performed in which order. The following may help us get there:

  • An educated guess. Maybe you know what happened at the time the deadlock occurred.
  • Database-side diagnostics. Often times, databases store information about deadlocking transactions after they occurred. Not sure what SQL server offers here.
  • Tweaking engine logging. You could set the levels of the following loggers to DEBUG (or FINE on Tomcat): org.camunda.bpm.engine.persistence, org.camunda.bpm.engine.cmd. This will log every command the engine performs along with the database operations it made. Then, reproduce the problem and see in the log what happened in parallel to the deadlocking transaction.

Cheers,
Thorben

1 Like