Many daily exceptions Springboot starter 3.0.0 mysql 5.7

Hello,
It’s been 3 months since we started using camunda springboot, and now we have 3~5 projects using it. Each projects has its own mysql database for camunda, and so none of the projects know each other nor have any type of communication (very different projects from a Business perspective).

In all of this projects, we have the same type of errors everyday and need to keep retrying some process instances til we have no more incidents. I can say that in a project with like 1000 new process instances each day, we have like 200 incidentes each day.

We are testing camunda community version before getting the enterprise version, but can’t continue using it with that many problems. I think it must be some version we are using or some adjustment we have to do in mysql database and haven’t done. Obs.: Isolation Level is already READ COMMITED as documentation says.

Please, if anyone can give me any advice on this. Thankyou!

This is our environment:
Docker image: oracle/serverjre:8
Number of pods in kubernetes: 3
Mysql version: 5.7

Gradle dependencies:
compile(“org.springframework.boot:spring-boot-starter-jdbc”)
compile(‘org.springframework.cloud:spring-cloud-starter-config’)
compile(“org.camunda.bpm.springboot:camunda-bpm-spring-boot-starter:3.0.0”)
compile(“org.camunda.bpm.springboot:camunda-bpm-spring-boot-starter-webapp:3.0.0”)
compile(“org.camunda.bpm.springboot:camunda-bpm-spring-boot-starter-rest:3.0.0”)
compile(“org.camunda.bpm:camunda-engine-plugin-spin:7.9.0”)
compile(“org.camunda.spin:camunda-spin-core”)
compile(“org.camunda.spin:camunda-spin-dataformat-all:1.5.3”)
compile(“com.squareup.okhttp3:okhttp:3.9.1”)
compile(“com.jayway.jsonpath:json-path:2.0.0”)
compile(‘mysql:mysql-connector-java:8.0.11’)
compile(‘org.camunda.bpm.identity:camunda-identity-ldap:7.9.0’)
compile group: ‘org.springframework.cloud’, name: ‘spring-cloud-starter-openfeign’, version: ‘2.0.1.RELEASE’
testCompile(‘org.springframework.boot:spring-boot-starter-test’)

Below is some of the messages we see everyday in incidents:

Sometimes it says a variable or even a Delegate couldn’t be found, and i keep retrying it til it works.
Sometines its “ENGINE-17004 Cannot add variable instance with name xxx. Variable already exists”
Sometimes it says “ENGINE-03083” “MySQLTransactionRollbackException: Deadlock found when trying to get lock; try restarting transaction”
Sometimes it says “ENGINE-03083” “SQLIntegrityConstraintViolationException: Cannot add or update a child row: a foreign key constraint fails (camunda.ACT_RU_VARIABLE, CONSTRAINT ACT_FK_VAR_BYTEARRAY FOREIGN KEY (BYTEARRAY_ID_) REFERENCES ACT_GE_BYTEARRAY (ID_))”
Sometimes it says: Unknown property used in expression: ${xxx}. Cause: Cannot resolve identifier ‘xxx’
Sometimes it says: at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
at java.util.LinkedList$ListItr.next(LinkedList.java:888)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.fireCommandContextClose(CommandContext.java:185)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:131)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
at org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:42)

Just to answer it for anyone who has the same caotic scenario:

We had a Base class for our delegates, with some private properties. One of those properties holds the Execution and is used to get/set variables. We used this style of base class with success in another projects, but they wasn’t springboot projects. Now that we are using camunda springboot in newer projects, our delegates became Singletons (using @Component with no @Scope defined). So the problem was: if our delegate is a Singleton, and we hold the execution in a private field, when Multiple process instances are using the same delegate in the same time, the execution gets overrided. It can lead to some serious problems like save variables from one instance in another process instance.

Using @Scope(ConfigurableBeanFactory.SCOPE_PROTOTYPE) in our base class resolved this issue.