Database deadlock errors while deleting deployments

Hello fellow Camunda users,

I’m getting database deadlock errors when attempting to delete a deployment. We’re currently using:

  • Camunda version 7.8.0
  • Apache Tomcat
  • MySQL DB on Amazon RDS (v 5.6.10)
  • Hosted on Docker containers (AWS ECS - Fargate)

We currently have over 1k deployments, each with at least 1 active program instance.

This is the rest call that causes the errors:
DELETE /engine-rest/deployment/[deploymentId]?cascade=true

These are all the things I’ve tried:

  • Updating to the latest stable version (Camunda and DB to 7.11.0)
  • Setting the isolation level to READ_COMMITTED
  • Deleting one deployment at a time
  • Setting the cascade option to false (which was not allowed)

None of these items have resolved the issue.

Below are the error logs that I get while deploying:

### Error flushing statements. Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.HistoricJobLogEntity.deleteHistoricJobLogByDeploymentId (batch index #8) failed. 7 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.HistoricJobLogEntity.deleteHistoricJobLogByDeploymentId (batch index #8) failed. 7 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
'. Flush summary:
[
DELETE_BULK deleteDeployment 918f3d90-6250-11e9-adb5-0a58a9feac2a
UPDATE HistoricActivityInstanceEventEntity[05b08512-c24f-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[1537433c-c256-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[163308b7-c256-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[22d4d416-c256-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[2724cd6c-c256-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[3abed632-c253-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[3d848612-c253-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[4752f89b-c251-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[49be52a7-c251-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[4db8b580-c255-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[4ea0eeee-c251-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[519049ca-c255-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[82912e4a-c252-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[83a75845-c252-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[96134fa4-c254-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[a0939aa9-c254-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[cb27234a-c251-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[d0c128f6-c251-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[e1d1d543-c253-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[ed7fcaf9-c253-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[31de8002-9d15-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[4267efe6-9d16-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[682a619b-9d16-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[748e611e-9d16-11e9-b66e-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[91dfad34-9d15-11e9-b66e-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[b125f69c-9d15-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[bcc96553-9d15-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[e8b60dc8-9d15-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricProcessInstanceEventEntity[04c4f42f-6251-11e9-9287-0a58a9feac2a]
DELETE_BULK deleteAuthorizationsForResourceId {resourceId=918f3d90-6250-11e9-adb5-0a58a9feac2a, resourceType=9}
DELETE_BULK deleteAuthorizationsForResourceId {resourceId=04c4f42f-6251-11e9-9287-0a58a9feac2a, resourceType=8}
DELETE_BULK deleteHistoricIdentityLinksByProcessDefinitionId 91924ad2-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteHistoricIncidentsByProcessDefinitionId 91924ad2-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteHistoricJobLogByDeploymentId 918f3d90-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteHistoricJobLogByProcessDefinitionId 91924ad2-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteJobDefinitionsByProcessDefinitionId 91924ad2-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteHistoricDetailByteArraysByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricDetailsByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricVariableInstanceByteArraysByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricVariableInstanceByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteCommentsByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteAttachmentByteArraysByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteAttachmentByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricDetailByteArraysByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricDetailsByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteCommentsByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteAttachmentByteArraysByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteAttachmentByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricIdentityLinksByTaskProcessInstanceIds [04c4f42f-6251-11e9-9287-0a58a9feac2a]
]

Has anyone seen this issue before? Are there any other items I can try?

Thanks!

@katbntz i hope you are using MySQL database, and make sure the isolation level for DB was set to READ_COMMITTED and while deleting do cascade delete of process instances.

Also make sure same version for database drivers and Database are been used. For example, MySQL v5.6, then mysql jdbc driver version also should be 5.6

Are you using rest api to delete the process instances? If rest api doesn’t return any response, the response would be like Status 204. No content. , in that case if you try to expect response body, it will throw Unexpected token < in JSON at position 0’, instead you can check for status code from response entity.