Hello fellow Camunda users,
I’m getting database deadlock errors when attempting to delete a deployment. We’re currently using:
- Camunda version 7.8.0
- Apache Tomcat
- MySQL DB on Amazon RDS (v 5.6.10)
- Hosted on Docker containers (AWS ECS - Fargate)
We currently have over 1k deployments, each with at least 1 active program instance.
This is the rest call that causes the errors:
DELETE /engine-rest/deployment/[deploymentId]?cascade=true
These are all the things I’ve tried:
- Updating to the latest stable version (Camunda and DB to 7.11.0)
- Setting the isolation level to READ_COMMITTED
- Deleting one deployment at a time
- Setting the cascade option to false (which was not allowed)
None of these items have resolved the issue.
Below are the error logs that I get while deploying:
### Error flushing statements. Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.HistoricJobLogEntity.deleteHistoricJobLogByDeploymentId (batch index #8) failed. 7 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.HistoricJobLogEntity.deleteHistoricJobLogByDeploymentId (batch index #8) failed. 7 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
java.sql.BatchUpdateException: Deadlock found when trying to get lock; try restarting transaction
com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction
'. Flush summary:
[
DELETE_BULK deleteDeployment 918f3d90-6250-11e9-adb5-0a58a9feac2a
UPDATE HistoricActivityInstanceEventEntity[05b08512-c24f-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[1537433c-c256-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[163308b7-c256-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[22d4d416-c256-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[2724cd6c-c256-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[3abed632-c253-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[3d848612-c253-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[4752f89b-c251-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[49be52a7-c251-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[4db8b580-c255-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[4ea0eeee-c251-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[519049ca-c255-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[82912e4a-c252-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[83a75845-c252-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[96134fa4-c254-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[a0939aa9-c254-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[cb27234a-c251-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[d0c128f6-c251-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[e1d1d543-c253-11e9-99ed-0a58a9feac2a]
UPDATE HistoricActivityInstanceEventEntity[ed7fcaf9-c253-11e9-a1d1-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[31de8002-9d15-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[4267efe6-9d16-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[682a619b-9d16-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[748e611e-9d16-11e9-b66e-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[91dfad34-9d15-11e9-b66e-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[b125f69c-9d15-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[bcc96553-9d15-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricIncidentEventEntity[e8b60dc8-9d15-11e9-b9e7-0a58a9feac2a]
UPDATE HistoricProcessInstanceEventEntity[04c4f42f-6251-11e9-9287-0a58a9feac2a]
DELETE_BULK deleteAuthorizationsForResourceId {resourceId=918f3d90-6250-11e9-adb5-0a58a9feac2a, resourceType=9}
DELETE_BULK deleteAuthorizationsForResourceId {resourceId=04c4f42f-6251-11e9-9287-0a58a9feac2a, resourceType=8}
DELETE_BULK deleteHistoricIdentityLinksByProcessDefinitionId 91924ad2-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteHistoricIncidentsByProcessDefinitionId 91924ad2-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteHistoricJobLogByDeploymentId 918f3d90-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteHistoricJobLogByProcessDefinitionId 91924ad2-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteJobDefinitionsByProcessDefinitionId 91924ad2-6250-11e9-adb5-0a58a9feac2a
DELETE_BULK deleteHistoricDetailByteArraysByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricDetailsByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricVariableInstanceByteArraysByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricVariableInstanceByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteCommentsByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteAttachmentByteArraysByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteAttachmentByIds {processInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricDetailByteArraysByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricDetailsByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteCommentsByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteAttachmentByteArraysByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteAttachmentByIds {taskProcessInstanceIds=[04c4f42f-6251-11e9-9287-0a58a9feac2a]}
DELETE_BULK deleteHistoricIdentityLinksByTaskProcessInstanceIds [04c4f42f-6251-11e9-9287-0a58a9feac2a]
]
Has anyone seen this issue before? Are there any other items I can try?
Thanks!