We are seeing first the optimistic lock exception and then after some time deadlock on the ACT_RU_VARIABLE table. Due to these deadlock exceptions the database connections get max out and the upstream application (Camunda engine) is unable to make any new connection as the connection pool has exhausted. This makes the whole application to go into a very bad state.
This is happening on our customer production environment. The database is Oracle 19c.
I am attaching optimistic lock trace, deadlock stack trace and database trace log.
Any help on how to resolve this is much appreciated.
Thanks!
<< Begin -- Server-OptimisticLock.log
2024-08-05 02:00:41,167 INFO [stdout] (default task-30) activitySize=1
2024-08-05 02:00:41,383 WARN [org.camunda.bpm.engine.jobexecutor] (pool-6-thread-4) ENGINE-14006 Exception while executing job 376982c0-5308-11ef-b1ba-a0d3c1fb179f: : org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of 'DELETE MessageEntity[376982c0-5308-11ef-b1ba-a0d3c1fb179f]' failed. Entity was updated by another transaction concurrently.
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.concurrentUpdateDbEntityException(EnginePersistenceLogger.java:135)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.handleOptimisticLockingException(DbEntityManager.java:499)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.checkFlushResults(DbEntityManager.java:451)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:367)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:325)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:297)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:208)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:137)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:116)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:61)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:51)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:44)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:92)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:61)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
2024-08-05 02:00:41,400 WARN [org.camunda.bpm.engine.jobexecutor] (pool-6-thread-4) ENGINE-14006 Exception while executing job 376982c0-5308-11ef-b1ba-a0d3c1fb179f: : org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of 'DELETE MessageEntity[376982c0-5308-11ef-b1ba-a0d3c1fb179f]' failed. Entity was updated by another transaction concurrently.
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.concurrentUpdateDbEntityException(EnginePersistenceLogger.java:135)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.handleOptimisticLockingException(DbEntityManager.java:499)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.checkFlushResults(DbEntityManager.java:451)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:367)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:325)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:297)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:208)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:137)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:116)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:61)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:51)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:44)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:92)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:61)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
2024-08-05 02:00:41,461 ERROR [stderr] (pool-6-thread-6) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:41,461 ERROR [stderr] (pool-6-thread-5) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:41,659 ERROR [stderr] (pool-6-thread-6) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:41,661 ERROR [stderr] (pool-6-thread-5) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:41,787 ERROR [stderr] (pool-6-thread-5) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:41,804 ERROR [stderr] (pool-6-thread-6) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:42,036 WARN [org.camunda.bpm.engine.jobexecutor] (pool-6-thread-6) ENGINE-14006 Exception while executing job 3646fc38-5308-11ef-a3ad-a0d3c1fb179f: : org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of 'DELETE MessageEntity[3646fc38-5308-11ef-a3ad-a0d3c1fb179f]' failed. Entity was updated by another transaction concurrently.
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.concurrentUpdateDbEntityException(EnginePersistenceLogger.java:135)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.handleOptimisticLockingException(DbEntityManager.java:499)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.checkFlushResults(DbEntityManager.java:451)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:367)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:325)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:297)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:208)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:137)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:116)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:61)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:51)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:44)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:92)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:61)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
2024-08-05 02:00:42,047 WARN [org.camunda.bpm.engine.jobexecutor] (pool-6-thread-6) ENGINE-14006 Exception while executing job 3646fc38-5308-11ef-a3ad-a0d3c1fb179f: : org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of 'DELETE MessageEntity[3646fc38-5308-11ef-a3ad-a0d3c1fb179f]' failed. Entity was updated by another transaction concurrently.
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.concurrentUpdateDbEntityException(EnginePersistenceLogger.java:135)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.handleOptimisticLockingException(DbEntityManager.java:499)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.checkFlushResults(DbEntityManager.java:451)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:367)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:325)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:297)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:208)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:137)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:116)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:61)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:51)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:44)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:92)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:61)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
2024-08-05 02:00:42,112 ERROR [stderr] (pool-6-thread-7) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:42,224 ERROR [stderr] (pool-6-thread-7) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:42,359 ERROR [stderr] (pool-6-thread-7) Warning: Nashorn engine is planned to be removed from a future JDK release
2024-08-05 02:00:42,569 WARN [org.camunda.bpm.engine.jobexecutor] (pool-6-thread-7) ENGINE-14006 Exception while executing job 34de6b2c-5308-11ef-b1ba-a0d3c1fb179f: : org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of 'DELETE MessageEntity[34de6b2c-5308-11ef-b1ba-a0d3c1fb179f]' failed. Entity was updated by another transaction concurrently.
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.concurrentUpdateDbEntityException(EnginePersistenceLogger.java:135)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.handleOptimisticLockingException(DbEntityManager.java:499)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.checkFlushResults(DbEntityManager.java:451)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:367)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:325)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:297)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:208)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:137)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:116)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:61)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:51)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:44)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:92)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:61)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:809)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
<< end -- Server-OptimisticLock.log
<< Begin -- server-deadlock
2024-08-05 02:06:33,329 ERROR [org.camunda.bpm.engine.context] (pool-6-thread-285) ENGINE-16004 Exception while closing command context: ENGINE-03083 Exception while executing Batch Database Operations with message '
### Error flushing statements. Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
'. Flush summary:
[
INSERT HistoricActivityInstanceEventEntity[fefab9e6-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[fefed7bd-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff031d8d-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff031d8f-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff0344a2-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff089bdc-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff089bdf-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff0e1a22-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff0e1a24-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[fefed7bb-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff028144-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff02a855-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff02a856-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff02f677-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff02f678-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff031d89-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff031d8a-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff031d8b-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff0344a0-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff089bda-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff089bdd-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff0df310-5309-11ef-867a-98f2b32157c3]
INSERT MessageEntity[ff0e4136-5309-11ef-867a-98f2b32157c3]
DELETE MessageEntity[376982c0-5308-11ef-b1ba-a0d3c1fb179f]
DELETE_BULK deleteByteArrayNoRevisionCheck 3791cad1-5308-11ef-b1ba-a0d3c1fb179f
UPDATE ExecutionEntity[1efc323f-5309-11ef-90f1-98f2b32157c3]
]: org.camunda.bpm.engine.ProcessEngineException: ENGINE-03083 Exception while executing Batch Database Operations with message '
### Error flushing statements. Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
java.sql.SQLException: ORA-00060: deadlock detected while waiting for resource
'. Flush summary:
[
INSERT HistoricActivityInstanceEventEntity[fefab9e6-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[fefed7bd-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff031d8d-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff031d8f-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff0344a2-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff089bdc-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff089bdf-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff0e1a22-5309-11ef-867a-98f2b32157c3]
INSERT HistoricActivityInstanceEventEntity[ff0e1a24-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[fefed7bb-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff028144-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff02a855-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff02a856-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff02f677-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff02f678-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff031d89-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff031d8a-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff031d8b-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff0344a0-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff089bda-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff089bdd-5309-11ef-867a-98f2b32157c3]
INSERT VariableInstanceEntity[ff0df310-5309-11ef-867a-98f2b32157c3]
INSERT MessageEntity[ff0e4136-5309-11ef-867a-98f2b32157c3]
DELETE MessageEntity[376982c0-5308-11ef-b1ba-a0d3c1fb179f]
DELETE_BULK deleteByteArrayNoRevisionCheck 3791cad1-5308-11ef-b1ba-a0d3c1fb179f
UPDATE ExecutionEntity[1efc323f-5309-11ef-90f1-98f2b32157c3]
]
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.flushDbOperationsException(EnginePersistenceLogger.java:693)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:362)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:325)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:297)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:208)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:137)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:116)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.JtaTransactionInterceptor.execute(JtaTransactionInterceptor.java:61)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:51)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:44)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:92)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:61)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:29)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:789)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:44)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:830)
at java.base/java.lang.Thread.run(Thread.java:829)
at org.jboss.threads@2.3.3.Final//org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: org.apache.ibatis.exceptions.PersistenceException:
### Error flushing statements. Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
### Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
at org.mybatis.mybatis@3.5.3//org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:30)
at org.mybatis.mybatis@3.5.3//org.apache.ibatis.session.defaults.DefaultSqlSession.flushStatements(DefaultSqlSession.java:254)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.sql.DbSqlSession.flushOperations(DbSqlSession.java:90)
at org.camunda.bpm.camunda-engine@7.11.0//org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:357)
... 18 more
Caused by: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.VariableInstanceEntity.insertVariableInstance (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
at org.mybatis.mybatis@3.5.3//org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:149)
at org.mybatis.mybatis@3.5.3//org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:129)
at org.mybatis.mybatis@3.5.3//org.apache.ibatis.executor.BaseExecutor.flushStatements(BaseExecutor.java:122)
at org.mybatis.mybatis@3.5.3//org.apache.ibatis.executor.CachingExecutor.flushStatements(CachingExecutor.java:114)
at org.mybatis.mybatis@3.5.3//org.apache.ibatis.session.defaults.DefaultSqlSession.flushStatements(DefaultSqlSession.java:252)
... 20 more
Caused by: java.sql.BatchUpdateException: ORA-00060: deadlock detected while waiting for resource
at oracle.driver@12.2.0.1.0//oracle.jdbc.driver.OraclePreparedStatement.executeLargeBatch(OraclePreparedStatement.java:10032)
at oracle.driver@12.2.0.1.0//oracle.jdbc.driver.T4CPreparedStatement.executeLargeBatch(T4CPreparedStatement.java:1364)
at oracle.driver@12.2.0.1.0//oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:9839)
at oracle.driver@12.2.0.1.0//oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:234)
at org.jboss.ironjacamar.jdbcadapters@1.4.17.Final//org.jboss.jca.adapters.jdbc.WrappedStatement.executeBatch(WrappedStatement.java:1180)
at org.mybatis.mybatis@3.5.3//org.apache.ibatis.executor.BatchExecutor.doFlushStatements(BatchExecutor.java:123)
... 24 more
>> end server-deadlock
<< Begin database deadlock alert log
Holds resource TX-079B0001-000003CD-00000000-00000000 acquired 36 seconds ago.
sid: 9883 ser: 44152 audsid: 52917762 user: 131/SMP20_LABDBUSER
flags: (0x1000041) USR/- flags2: (0x40009) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
pid: 1598 O/S info: user: oracle, term: UNKNOWN, ospid: 27421
image: oracle@belsmd01
client details:
O/S info: user: smpadmin, term: unknown, ospid: 1234
machine: belsmp01 program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
insert into ACT_RU_VARIABLE
(
ID_,
TYPE_,
NAME_,
PROC_INST_ID_,
EXECUTION_ID_,
CASE_INST_ID_,
CASE_EXECUTION_ID_,
TASK_ID_,
BYTEARRAY_ID_,
DOUBLE_,
LONG_,
TEXT_,
TEXT2_,
VAR_SCOPE_,
SEQUENCE_COUNTER_,
IS_CONCURRENT_LOCAL_,
TENANT_ID_,
REV_
)
values (
:1 ,
:2 ,
:3 ,
:4 ,
:5 ,
:6 ,
:7 ,
:8 ,
:9 ,
:10 ,
:11 ,
:12 ,
:13 ,
:14 ,
:15 ,
:16 ,
:17 ,
1
)
Session 4726:
Holds resource TX-06140003-0000043F-00000000-00000000 acquired 3 seconds ago.
sid: 4726 ser: 43798 audsid: 52918230 user: 131/SMP20_LABDBUSER
flags: (0x1000041) USR/- flags2: (0x40009) -/-/INC
flags_idl: (0x1) status: BSY/-/-/- kill: -/-/-/-
pid: 2062 O/S info: user: oracle, term: UNKNOWN, ospid: 29188
image: oracle@belsmd01
client details:
O/S info: user: smpadmin, term: unknown, ospid: 1234
machine: belsmp01 program: JDBC Thin Client
application name: JDBC Thin Client, hash value=2546894660
current SQL:
insert into ACT_RU_VARIABLE
(
ID_,
TYPE_,....
...
>> end of database deadlock alert log