[External Task] Exception(Deadlock found when trying to get lock) while processing the tasks

Hi All,

We use external task to handle Async tasks, so at a given time atleast 10 external task clients ( threads) will be running.We see below exception while processing the tasks.

Camunda version : 7.15
Database : Mariadb

Please suggest if there are any solution to resolve this

Exception details :
[pool-2-thread-53] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace.
org.camunda.bpm.engine.ProcessEngineException: An exception occurred in the persistence layer. Please check the server logs for a detailed message and the entire exception stack trace.
at org.camunda.bpm.engine.impl.util.ExceptionUtil.wrapPersistenceException(ExceptionUtil.java:263)
at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:132)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:364)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:323)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:295)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:272)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:188)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:119)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:57)
at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:110)
at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:71)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation ‘DELETE VariableInstanceEntity[92dfe9fe-7b19-11ed-994c-628e16bf6343]’ with message ’

Error flushing statements. Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.deleteVariableInstance (batch index #7) failed. 6 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn=292850) Deadlock found when trying to get lock; try restarting transaction

Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.deleteVariableInstance (batch index #7) failed. 6 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: (conn=292850) Deadlock found when trying to get lock; try restarting transaction

java.sql.BatchUpdateException: (conn=292850) Deadlock found when trying to get lock; try restarting transaction
java.sql.SQLTransactionRollbackException: (conn=292850) Deadlock found when trying to get lock; try restarting transaction
org.mariadb.jdbc.internal.util.exceptions.MariaDbSqlException: Deadlock found when trying to get lock; try restarting transaction
java.sql.SQLException: Deadlock found when trying to get lock; try restarting transaction
'. Flush summary:
[
INSERT HistoricVariableInstanceEntity[92e28162-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableInstanceEntity[92e28165-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableInstanceEntity[92e31da7-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableInstanceEntity[92e31da9-7b19-11ed-994c-628e16bf6343]
INSERT HistoricJobLogEventEntity[92e3b9f9-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableUpdateEventEntity[92e28164-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableUpdateEventEntity[92e28166-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableUpdateEventEntity[92e31da8-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableUpdateEventEntity[92e31daa-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableUpdateEventEntity[92e392e7-7b19-11ed-994c-628e16bf6343]
INSERT HistoricVariableUpdateEventEntity[92e392e8-7b19-11ed-994c-628e16bf6343]
INSERT HistoricActivityInstanceEventEntity[Activity_08dvyzg:92e28161-7b19-11ed-994c-628e16bf6343]
INSERT HistoricActivityInstanceEventEntity[Activity_1knrcri:92e344bc-7b19-11ed-994c-628e16bf6343]
INSERT HistoricActivityInstanceEventEntity[Event_15s0f9w:92e344bd-7b19-11ed-994c-628e16bf6343]
INSERT HistoricActivityInstanceEventEntity[Event_1cwfjtt:92e344bf-7b19-11ed-994c-628e16bf6343]
INSERT HistoricActivityInstanceEventEntity[Event_1uypogg:92e344c0-7b19-11ed-994c-628e16bf6343]
DELETE MessageEntity[92e01123-7b19-11ed-994c-628e16bf6343]
UPDATE VariableInstanceEntity[92834a52-7b19-11ed-994c-628e16bf6343]
UPDATE VariableInstanceEntity[92834a54-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371e0-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371e2-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371e4-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371e6-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371e8-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371ea-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371ec-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371ee-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[928371f0-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92839902-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92839904-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92839906-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92839908-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[9295c13d-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[9295e853-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92da1d39-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9ee-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9f0-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9f2-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9f4-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9f6-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9f8-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9fa-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9fc-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfe9fe-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfea00-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfea02-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfea04-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfea06-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfea08-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfea0b-7b19-11ed-994c-628e16bf6343]
DELETE VariableInstanceEntity[92dfea0f-7b19-11ed-994c-628e16bf6343]
DELETE ByteArrayEntity[92da1d38-7b19-11ed-994c-628e16bf6343]
DELETE ByteArrayEntity[92dfea0a-7b19-11ed-994c-628e16bf6343]
UPDATE ExecutionEntity[92834a59-7b19-11ed-994c-628e16bf6343]
UPDATE ExecutionEntity[92834a4e-7b19-11ed-994c-628e16bf6343]
DELETE ExecutionEntity[928371df-7b19-11ed-994c-628e16bf6343]
UPDATE HistoricActivityInstanceEventEntity[Activity_1h2bh7m:928371de-7b19-11ed-994c-628e16bf6343]
UPDATE HistoricProcessInstanceEventEntity[928371df-7b19-11ed-994c-628e16bf6343]
DELETE_BULK deleteAuthorizationsForResourceId {resourceId=928371df-7b19-11ed-994c-628e16bf6343, resourceType=8}
UPDATE HistoricVariableInstanceEntity[92834a52-7b19-11ed-994c-628e16bf6343]
UPDATE HistoricVariableInstanceEntity[92834a54-7b19-11ed-994c-628e16bf6343]
]
at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationException(EnginePersistenceLogger.java:131)
… 15 more
Caused by: org.apache.ibatis.exceptions.PersistenceException:

Regards,
Kavitha

Hi Kavitha,
Can you try increasing your lock duration? Looks like there is some issue in acquiring lock.

Here is the sample of External task I am using.

  ExternalTaskClient client = ExternalTaskClient.create()
                    .baseUrl("http://localhost:8080/engine-rest")
                    .asyncResponseTimeout(20000)
                    .lockDuration(10000)
                    .maxTasks(1)
                    .build();

-Manash

Hi Manash,

We are using below configurations in the external task client

lockDuration = 2700000 ( 45 minutes)
asyncResponseTimeout = 120000( 2 minutes)

Regards,
KAvitha