Processes get stuck on a Timer Intermediate catch event

Hi!
We have migrated schema from Activiti 5.21.0 to Camunda 7.9. We can start our application (schema-update=false) and we see all previous processes from history. But when we create a new instance we will get stuck on a timer. This timer has a duration configuration like PT10S. And there were no exceptions.

After investigating I found out that this timer is stored in ACT_RU_JOB table where:

EXCEPTION_STACK_ID_   - 20012527

and

EXCEPTION_MSG_ - 
Error querying database.  Cause: java.sql.SQLSyntaxErrorException: ORA-00904: "PROC_INST_ID_": invalid identifier
The error may exist in org/activiti/db/mapping/entity/IdentityLink.xml
The error may involve org.activiti.engine.impl.persis... 

It’s strange to have an exception here and why do we have this from activiti. Also, I found the full stacktrace by EXCEPTION_STACK_ID_ in the ACT_GE_BYTEARRAY table:

org.apache.ibatis.exceptions.PersistenceException: 
Error querying database.  Cause: java.sql.SQLSyntaxErrorException: ORA-00904: "PROC_INST_ID_": invalid identifier
The error may exist in org/activiti/db/mapping/entity/IdentityLink.xml
The error may involve org.activiti.engine.impl.persistence.entity.IdentityLinkEntity.selectIdentityLinksByProcessInstance-Inline
The error occurred while setting parameters
SQL: select * from ACT_RU_IDENTITYLINK where PROC_INST_ID_ = ?
Cause: java.sql.SQLSyntaxErrorException: ORA-00904: "PROC_INST_ID_": invalid identifier
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:23)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:107)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:98)
	at org.activiti.engine.impl.db.DbSqlSession.selectListWithRawParameter(DbSqlSession.java:443)
	at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:434)
	at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:429)
	at org.activiti.engine.impl.db.DbSqlSession.selectList(DbSqlSession.java:406)
	at org.activiti.engine.impl.persistence.entity.IdentityLinkEntityManager.findIdentityLinksByProcessInstanceId(IdentityLinkEntityManager.java:51)
	at org.activiti.engine.impl.persistence.entity.IdentityLinkEntityManager.deleteIdentityLinksByProcInstance(IdentityLinkEntityManager.java:103)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.removeIdentityLinks(ExecutionEntity.java:1454)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.remove(ExecutionEntity.java:993)
	at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionDestroyScope.execute(AtomicOperationTransitionDestroyScope.java:101)
	at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:644)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:637)
	at org.activiti.engine.impl.pvm.runtime.AtomicOperationTransitionNotifyListenerEnd.eventNotificationsCompleted(AtomicOperationTransitionNotifyListenerEnd.java:35)
	at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:56)
	at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:644)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:637)
	at org.activiti.engine.impl.pvm.runtime.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:49)
	at org.activiti.engine.impl.interceptor.CommandContext.performOperation(CommandContext.java:97)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:644)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:637)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.take(ExecutionEntity.java:454)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.take(ExecutionEntity.java:432)
	at org.activiti.engine.impl.bpmn.behavior.BpmnActivityBehavior.performOutgoingBehavior(BpmnActivityBehavior.java:140)
	at org.activiti.engine.impl.bpmn.behavior.BpmnActivityBehavior.performDefaultOutgoingBehavior(BpmnActivityBehavior.java:66)
	at org.activiti.engine.impl.bpmn.behavior.FlowNodeActivityBehavior.leave(FlowNodeActivityBehavior.java:44)
	at org.activiti.engine.impl.bpmn.behavior.AbstractBpmnActivityBehavior.leave(AbstractBpmnActivityBehavior.java:47)
	at org.activiti.engine.impl.bpmn.behavior.IntermediateCatchEventActivityBehavior.signal(IntermediateCatchEventActivityBehavior.java:27)
	at org.activiti.engine.impl.persistence.entity.ExecutionEntity.signal(ExecutionEntity.java:411)
	at org.activiti.engine.impl.jobexecutor.TimerCatchIntermediateEventJobHandler.execute(TimerCatchIntermediateEventJobHandler.java:58)
	at org.activiti.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:85)
	at org.activiti.engine.impl.persistence.entity.TimerEntity.execute(TimerEntity.java:96)
	at org.activiti.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:88)
	at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
	at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
	at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
	at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
	at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
	at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.handleMultipleJobs(ExecuteJobsRunnable.java:94)
	at org.activiti.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLSyntaxErrorException: ORA-00904: "PROC_INST_ID_": invalid identifier
	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:494)
	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:446)
	at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1054)
	at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:623)
	at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
	at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:226)
	at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:59)
	at oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe(T4CPreparedStatement.java:747)
	at oracle.jdbc.driver.OracleStatement.executeMaybeDescribe(OracleStatement.java:904)
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1082)
	at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
	at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343)
	at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887)
	at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:56)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:70)
	at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:57)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:259)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:132)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:105)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:81)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:104)
	... 44 more
Caused by: Error : 904, Position : 40, Sql = select * from ACT_RU_IDENTITYLINK where PROC_INST_ID_ = :1 , OriginalSql = select * from ACT_RU_IDENTITYLINK where PROC_INST_ID_ = ?, Error Msg = ORA-00904: "PROC_INST_ID_": invalid identifier
	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:498)
	... 66 more

I set
org.camunda.bpm.engine.impl.persistence.entity
org.camunda.bpm.engine.persistence
org.camunda.bpm.engine.cmd
to DEBUG mode but unfortunately it gave us nothing.

Do you have ideas why it happens? What’s wrong?

Thank you in advance.

Hi @Vyacheslav_Biletskiy ,

are you sure that you run Camunda and not Activiti? :sweat_smile:

In the stacktrace, I see classes from Activiti and not Camunda:
org.activiti.engine.impl.persistence.entity.IdentityLinkEntityManager

You get this exception because the database schema is different and in Camunda the table ACT_RU_IDENTITYLINK doesn’t have a column PROC_INST_ID_.

Best regards,
Philipp

Thank you for response.

We found the reason of this behavior. We kept other running WSO2 server it the same time on the same DB schema. And we assumed that after job is created activity tried to execute this job.