Task after timer event intermittently throws ENGINE-09008 Exception while instantiating class

Dan_Bahir · March 16, 2017, 11:47pm

I have a workflow running on an embedded workflow engine. After adding a timer between two of the tasks I started getting ENGINE-09008 Exception while instantiating class exceptions from the task after the timer. The exception occur 50% of the time. The rest of the time it executes correctly. If I hit retry from the Cockpit it will work correctly again ~50% of the time.

The exception does not show up in the logs of the application, I only see it when I look at the Cockpit

Work flow before adding the timer:

Workflow after adding the timer:

Exception in Cockpit:

Exception details:

org.camunda.bpm.engine.ProcessEngineException: ENGINE-09008 Exception while instantiating class 'com.someCompany.someProduct.workflows.replicationMonitoring.CheckIfClusterIsKdrTask': ENGINE-09017 Cannot load class 'com.someCompany.someProduct.workflows.replicationMonitoring.CheckIfClusterIsKdrTask': com.someCompany.someProduct.workflows.replicationMonitoring.CheckIfClusterIsKdrTask
	at org.camunda.bpm.engine.impl.util.EngineUtilLogger.exceptionWhileInstantiatingClass(EngineUtilLogger.java:78)
	at org.camunda.bpm.engine.impl.util.ClassDelegateUtil.instantiateDelegate(ClassDelegateUtil.java:50)
	at org.camunda.bpm.engine.impl.bpmn.behavior.ClassDelegateActivityBehavior.getActivityBehaviorInstance(ClassDelegateActivityBehavior.java:109)
	at org.camunda.bpm.engine.impl.bpmn.behavior.ClassDelegateActivityBehavior$1.call(ClassDelegateActivityBehavior.java:65)
	at org.camunda.bpm.engine.impl.bpmn.behavior.ClassDelegateActivityBehavior$1.call(ClassDelegateActivityBehavior.java:62)
	at org.camunda.bpm.engine.impl.bpmn.behavior.AbstractBpmnActivityBehavior.executeWithErrorPropagation(AbstractBpmnActivityBehavior.java:108)
	at org.camunda.bpm.engine.impl.bpmn.behavior.ClassDelegateActivityBehavior.execute(ClassDelegateActivityBehavior.java:62)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute$2.callback(PvmAtomicOperationActivityExecute.java:57)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute$2.callback(PvmAtomicOperationActivityExecute.java:46)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.continueIfExecutionDoesNotAffectNextOperation(PvmExecutionImpl.java:1936)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute.execute(PvmAtomicOperationActivityExecute.java:38)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityExecute.execute(PvmAtomicOperationActivityExecute.java:27)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:611)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:587)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl$5.callback(PvmExecutionImpl.java:1875)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl$5.callback(PvmExecutionImpl.java:1872)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.continueExecutionIfNotCanceled(PvmExecutionImpl.java:1942)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1891)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1872)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(PvmAtomicOperationTransitionNotifyListenerStart.java:57)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerStart.eventNotificationsCompleted(PvmAtomicOperationTransitionNotifyListenerStart.java:27)
	at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:65)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:69)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:622)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:597)
	at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:58)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:69)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:622)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:597)
	at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:58)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:611)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:587)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionCreateScope.scopeCreated(PvmAtomicOperationTransitionCreateScope.java:34)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationCreateScope.execute(PvmAtomicOperationCreateScope.java:50)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationCreateScope.execute(PvmAtomicOperationCreateScope.java:24)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:112)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:611)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:587)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.take(PvmExecutionImpl.java:731)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionDestroyScope.execute(PvmAtomicOperationTransitionDestroyScope.java:85)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionDestroyScope.execute(PvmAtomicOperationTransitionDestroyScope.java:33)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:611)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:587)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerEnd$1.callback(PvmAtomicOperationTransitionNotifyListenerEnd.java:42)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerEnd$1.callback(PvmAtomicOperationTransitionNotifyListenerEnd.java:37)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.continueExecutionIfNotCanceled(PvmExecutionImpl.java:1942)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1891)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerEnd.eventNotificationsCompleted(PvmAtomicOperationTransitionNotifyListenerEnd.java:37)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationTransitionNotifyListenerEnd.eventNotificationsCompleted(PvmAtomicOperationTransitionNotifyListenerEnd.java:24)
	at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:65)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:69)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:622)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:597)
	at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:58)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:69)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:622)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperationSync(ExecutionEntity.java:597)
	at org.camunda.bpm.engine.impl.core.operation.AbstractEventAtomicOperation.execute(AbstractEventAtomicOperation.java:58)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:611)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:587)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.leaveActivityViaTransitions(PvmExecutionImpl.java:999)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.leaveActivityViaTransition(PvmExecutionImpl.java:942)
	at org.camunda.bpm.engine.impl.bpmn.behavior.BpmnActivityBehavior.performOutgoingBehavior(BpmnActivityBehavior.java:101)
	at org.camunda.bpm.engine.impl.bpmn.behavior.BpmnActivityBehavior.performDefaultOutgoingBehavior(BpmnActivityBehavior.java:52)
	at org.camunda.bpm.engine.impl.bpmn.behavior.FlowNodeActivityBehavior.doLeave(FlowNodeActivityBehavior.java:52)
	at org.camunda.bpm.engine.impl.bpmn.behavior.AbstractBpmnActivityBehavior.doLeave(AbstractBpmnActivityBehavior.java:71)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityLeave.execute(PvmAtomicOperationActivityLeave.java:52)
	at org.camunda.bpm.engine.impl.pvm.runtime.operation.PvmAtomicOperationActivityLeave.execute(PvmAtomicOperationActivityLeave.java:28)
	at org.camunda.bpm.engine.impl.interceptor.AtomicOperationInvocation.execute(AtomicOperationInvocation.java:89)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.invokeNext(CommandInvocationContext.java:125)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performNext(CommandInvocationContext.java:104)
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.performOperation(CommandInvocationContext.java:79)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:611)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.performOperation(ExecutionEntity.java:587)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl$5.callback(PvmExecutionImpl.java:1875)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl$5.callback(PvmExecutionImpl.java:1872)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.continueExecutionIfNotCanceled(PvmExecutionImpl.java:1942)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1891)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.dispatchDelayedEventsAndPerformOperation(PvmExecutionImpl.java:1872)
	at org.camunda.bpm.engine.impl.bpmn.behavior.FlowNodeActivityBehavior.leave(FlowNodeActivityBehavior.java:48)
	at org.camunda.bpm.engine.impl.bpmn.behavior.IntermediateCatchEventActivityBehavior.signal(IntermediateCatchEventActivityBehavior.java:46)
	at org.camunda.bpm.engine.impl.pvm.runtime.PvmExecutionImpl.signal(PvmExecutionImpl.java:710)
	at org.camunda.bpm.engine.impl.jobexecutor.TimerCatchIntermediateEventJobHandler.execute(TimerCatchIntermediateEventJobHandler.java:40)
	at org.camunda.bpm.engine.impl.jobexecutor.TimerCatchIntermediateEventJobHandler.execute(TimerCatchIntermediateEventJobHandler.java:23)
	at org.camunda.bpm.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:129)
	at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:99)
	at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:36)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:104)
	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
	at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:35)
	at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:28)
	at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:82)
	at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:56)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.camunda.bpm.engine.ClassLoadingException: ENGINE-09017 Cannot load class 'com.someCompany.someProduct.workflows.replicationMonitoring.CheckIfClusterIsKdrTask': com.someCompany.someProduct.workflows.replicationMonitoring.CheckIfClusterIsKdrTask
	at org.camunda.bpm.engine.impl.util.EngineUtilLogger.classLoadingException(EngineUtilLogger.java:135)
	at org.camunda.bpm.engine.impl.util.ReflectUtil.loadClass(ReflectUtil.java:107)
	at org.camunda.bpm.engine.impl.util.ClassDelegateUtil.instantiateDelegate(ClassDelegateUtil.java:42)
	... 131 more
Caused by: java.lang.ClassNotFoundException: com.someCompany.someProduct.workflows.replicationMonitoring.CheckIfClusterIsKdrTask
	at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:487)
	at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:428)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:340)
	at org.camunda.bpm.engine.impl.util.ReflectUtil.loadClass(ReflectUtil.java:84)
	... 132 more

garysamuelson · March 17, 2017, 8:14am

The error sounds like you have two servers configured in a “round-robin” cluster configuration and one of these servers isn’t getting a complete deployment.

Given the name of the class, “CheckIfClusterIsKdrTask”, I’m leaning towards a problem with cluster deployment configuration.

Are you running JBoss/Wildfly in clustered, “HA” mode?

If you’re setup for fail-over and/or load-balancing, a quick-fix is to just manually push a full deployment at both servers/nodes. Then, maybe later setup a master controller.

Camunda on clustered JBoss/Wildfly also requires “sticky session” configuration for your load-balancer.

For load-balancing, I’ve had success using NGINX. Though some still prefer the old HTTP mod_ configuration.

Dan_Bahir · March 17, 2017, 4:14pm

Thanks for answering Gary.

My configuration is a different but your comment allowed me to understand what went wrong. As I mentioned I have the process engine embedded in my java application. The application is running on jetty but that probably is not important. I am running this on my local machine and had two applications running (AppA, AppB) each with their own different workflows (WorkflowA, WorkflowB). The rest interfaces for each app are bound to different ports.
My guess to what happened is that WorkflowA was started on AppA and then got to the async timer. When the timer was up the job was sometimes picked up by AppB’s embedded engine. But AppB does not have the class for WorkflowA and hence the error.

Let me know if that sounds correct.

If so my follow up question is can I have more then one embedded engine with different workflows connecting to the same database ? It there a setting to make the task execution sticky to the engine instance it was started on ?

Thanks

Dan_Bahir · March 17, 2017, 5:53pm

Think I found it here: https://docs.camunda.org/manual/7.6/user-guide/process-engine/the-job-executor/

Thanks!

gdharley · March 17, 2017, 9:00pm

I was about to point you to: https://docs.camunda.org/manual/7.6/user-guide/process-engine/the-job-executor/#job-execution-in-heterogeneous-clusters

But I think you may have already found it.
If you make the the Job Executor “Deployment Aware” the problem goes away, but you will also impact your scalability as you have effectively locked a job executor by process app.

Cheers,
Greg