"activity is null" & "message is null"

wseaford · September 15, 2023, 4:42pm

Hi, in the past few weeks my Camunda process has been getting two specific incidents rising in cockpit

Cannot invoke “org.camunda.bpm.engine.impl.pvm.process.ActivityImpl.isAsyncBefore()” because “activity” is null
and
message is null

Here is the exception stack trace for the first one.

14-Sep 09:46:17.350 ERROR org.camunda.bpm.engine.context - ENGINE-16004 Exception while closing command context: Cannot invoke "org.camunda.bpm.engine.impl.pvm.process.ActivityImpl.isAsyncBefore()" because "activity" is null
java.lang.NullPointerException: Cannot invoke "org.camunda.bpm.engine.impl.pvm.process.ActivityImpl.isAsyncBefore()" because "activity" is null
	at org.camunda.bpm.engine.impl.pvm.runtime.LegacyBehavior.isAsync(LegacyBehavior.java:541)
	at org.camunda.bpm.engine.impl.pvm.runtime.LegacyBehavior.repairMultiInstanceAsyncJob(LegacyBehavior.java:570)
	at org.camunda.bpm.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:67)
	at org.camunda.bpm.engine.impl.jobexecutor.AsyncContinuationJobHandler.execute(AsyncContinuationJobHandler.java:40)
	at org.camunda.bpm.engine.impl.persistence.entity.JobEntity.execute(JobEntity.java:133)
	at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:110)
	at org.camunda.bpm.engine.impl.cmd.ExecuteJobsCmd.execute(ExecuteJobsCmd.java:43)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
	at org.camunda.bpm.engine.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:72)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
	at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:70)
	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
	at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
	at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobHelper.executeJob(ExecuteJobHelper.java:57)
	at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.executeJob(ExecuteJobsRunnable.java:110)
	at org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable.run(ExecuteJobsRunnable.java:71)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

For the other error I get even less information

15-Sep 16:15:09.023 ERROR org.camunda.bpm.engine.context - ENGINE-16004 Exception while closing command context: null
java.lang.NullPointerException: null

So far I have:

checked each activity in the bpmn has an id and name using modeler.
checked each activity has their asynchronous continuations before, after & exclusive checked.
went into Deployments and deleted past process definitions and checked they were no longer in the database (ACT_RE_PROCDEF).

These two errors still persist, It’s difficult for me to know which activity is sparking the incidents as the incident happens on the call activity. When I go into the process which the call activity passes to from cockpit then no incident or process is seen. Only viewable from the main pool.

Would anybody be able to shed light on what is happening and why?

WilliamR.Alves · September 15, 2023, 4:46pm

Hello my friend!

Can you share your process flow with us?

William Robert Alves

wseaford · September 22, 2023, 10:01pm

Didn’t quite find the root cause but we found a solution in setting up a retry policy as R10/PT5M.

WilliamR.Alves · September 22, 2023, 11:02pm

Great, but i imagine that in this activity you are calling some service, right?

My suggestion is that you check the root cause, as treating a nullPointerException by increasing retries will drastically increase the number of attempted calls to your service, and even “dirty” your logs.

If you increased the number of retries and resolved, probably in the context of your application, at the moment the instance arrives at this activity, the instance still does not have the object or variable necessary for execution, and then as time and attempts run out , your process has time for the object or variable to be loaded into the process and then it is possible to execute this activity.

Or something is trying to start this task, without it being definitively prepared for this at the time it is called.

If you want to share your process or part of it that may be causing this error, I can help you with a definitive solution.

William Robert Alves

ahuet · September 27, 2023, 1:28pm

Hello,

I’ve encountered a similar issue after a migration from version 7.16 to 7.18. I can share two stacktrace but the root cause is the same, one when starting a workflow:

! java.lang.NullPointerException: Cannot invoke “org.camunda.bpm.engine.impl.pvm.process.ActivityImpl.isAsyncBefore()” because “activity” is null
! at org.camunda.bpm.engine.impl.pvm.runtime.LegacyBehavior.isAsync(LegacyBehavior.java:541)
! at org.camunda.bpm.engine.impl.pvm.runtime.LegacyBehavior.migrateMultiInstanceJobDefinitions(LegacyBehavior.java:533)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.updateJobDeclarations(BpmnDeployer.java:184)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.definitionAddedToDeploymentCache(BpmnDeployer.java:139)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.definitionAddedToDeploymentCache(BpmnDeployer.java:71)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.registerDefinition(AbstractDefinitionDeployer.java:293)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.handlePersistedDefinition(AbstractDefinitionDeployer.java:245)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.handlePersistedDefinition(BpmnDeployer.java:162)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.handlePersistedDefinition(BpmnDeployer.java:71)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.loadDefinitions(AbstractDefinitionDeployer.java:237)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.postProcessDefinitions(AbstractDefinitionDeployer.java:206)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.deploy(AbstractDefinitionDeployer.java:66)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.CacheDeployer$2.call(CacheDeployer.java:66)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.CacheDeployer$2.call(CacheDeployer.java:63)
! at org.camunda.bpm.engine.impl.interceptor.CommandContext.runWithoutAuthorization(CommandContext.java:559)
! at org.camunda.bpm.engine.impl.interceptor.CommandContext.runWithoutAuthorization(CommandContext.java:547)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.CacheDeployer.deployOnlyGivenResourcesOfDeployment(CacheDeployer.java:63)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.ResourceDefinitionCache.resolveDefinition(ResourceDefinitionCache.java:124)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.ResourceDefinitionCache.findDeployedDefinitionById(ResourceDefinitionCache.java:56)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.DeploymentCache.findDeployedProcessDefinitionById(DeploymentCache.java:81)
! at org.camunda.bpm.engine.impl.runtime.DefaultCorrelationHandler.correlateStartMessageByEventSubscription(DefaultCorrelationHandler.java:176)
! at org.camunda.bpm.engine.impl.runtime.DefaultCorrelationHandler.correlateStartMessages(DefaultCorrelationHandler.java:155)
! at org.camunda.bpm.engine.impl.cmd.CorrelateMessageCmd$1.call(CorrelateMessageCmd.java:70)
! at org.camunda.bpm.engine.impl.cmd.CorrelateMessageCmd$1.call(CorrelateMessageCmd.java:68)
! at org.camunda.bpm.engine.impl.interceptor.CommandContext.runWithoutAuthorization(CommandContext.java:559)
! at org.camunda.bpm.engine.impl.interceptor.CommandContext.runWithoutAuthorization(CommandContext.java:547)
! at org.camunda.bpm.engine.impl.cmd.CorrelateMessageCmd.execute(CorrelateMessageCmd.java:68)
! at org.camunda.bpm.engine.impl.cmd.CorrelateMessageCmd.execute(CorrelateMessageCmd.java:42)
! at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
! at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
! at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.lambda$execute$0(SpringTransactionInterceptor.java:71)
! at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
! at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:71)
! at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
! at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35)
! at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
! at org.camunda.bpm.engine.impl.interceptor.ExceptionCodeInterceptor.execute(ExceptionCodeInterceptor.java:55)
! at org.camunda.bpm.engine.impl.MessageCorrelationBuilderImpl.execute(MessageCorrelationBuilderImpl.java:322)
! at org.camunda.bpm.engine.impl.MessageCorrelationBuilderImpl.correlateWithResult(MessageCorrelationBuilderImpl.java:233)
! at org.camunda.bpm.engine.impl.MessageCorrelationBuilderImpl.correlateStartMessage(MessageCorrelationBuilderImpl.java:292)
! at org.camunda.bpm.engine.impl.RuntimeServiceImpl.startProcessInstanceByMessage(RuntimeServiceImpl.java:600)

And second when calling the rest api to get the user tasks:

! java.lang.NullPointerException: Cannot invoke “org.camunda.bpm.engine.impl.pvm.process.ActivityImpl.isAsyncBefore()” because “activity” is null
! at org.camunda.bpm.engine.impl.pvm.runtime.LegacyBehavior.isAsync(LegacyBehavior.java:541)
! at org.camunda.bpm.engine.impl.pvm.runtime.LegacyBehavior.migrateMultiInstanceJobDefinitions(LegacyBehavior.java:533)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.updateJobDeclarations(BpmnDeployer.java:184)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.definitionAddedToDeploymentCache(BpmnDeployer.java:139)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.definitionAddedToDeploymentCache(BpmnDeployer.java:71)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.registerDefinition(AbstractDefinitionDeployer.java:293)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.handlePersistedDefinition(AbstractDefinitionDeployer.java:245)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.handlePersistedDefinition(BpmnDeployer.java:162)
! at org.camunda.bpm.engine.impl.bpmn.deployer.BpmnDeployer.handlePersistedDefinition(BpmnDeployer.java:71)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.loadDefinitions(AbstractDefinitionDeployer.java:237)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.postProcessDefinitions(AbstractDefinitionDeployer.java:206)
! at org.camunda.bpm.engine.impl.AbstractDefinitionDeployer.deploy(AbstractDefinitionDeployer.java:66)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.CacheDeployer$2.call(CacheDeployer.java:66)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.CacheDeployer$2.call(CacheDeployer.java:63)
! at org.camunda.bpm.engine.impl.interceptor.CommandContext.runWithoutAuthorization(CommandContext.java:559)
! at org.camunda.bpm.engine.impl.interceptor.CommandContext.runWithoutAuthorization(CommandContext.java:547)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.CacheDeployer.deployOnlyGivenResourcesOfDeployment(CacheDeployer.java:63)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.ResourceDefinitionCache.resolveDefinition(ResourceDefinitionCache.java:124)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.ResourceDefinitionCache.findDeployedDefinitionById(ResourceDefinitionCache.java:56)
! at org.camunda.bpm.engine.impl.persistence.deploy.cache.DeploymentCache.findDeployedProcessDefinitionById(DeploymentCache.java:81)
! at org.camunda.bpm.engine.impl.persistence.entity.TaskEntity.getTaskDefinition(TaskEntity.java:1320)
! at org.camunda.bpm.engine.impl.persistence.entity.TaskEntity.initializeFormKey(TaskEntity.java:1429)
! at org.camunda.bpm.engine.impl.TaskQueryImpl.executeList(TaskQueryImpl.java:1440)
! at org.camunda.bpm.engine.impl.AbstractQuery.evaluateExpressionsAndExecuteList(AbstractQuery.java:219)
! at org.camunda.bpm.engine.impl.AbstractQuery.execute(AbstractQuery.java:193)
! at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
! at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
! at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.lambda$execute$0(SpringTransactionInterceptor.java:71)
! at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
! at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:71)
! at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
! at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35)
! at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
! at org.camunda.bpm.engine.impl.interceptor.ExceptionCodeInterceptor.execute(ExceptionCodeInterceptor.java:55)
! at org.camunda.bpm.engine.impl.AbstractQuery.executeResult(AbstractQuery.java:160)
! at org.camunda.bpm.engine.impl.AbstractQuery.list(AbstractQuery.java:142)
! at org.camunda.bpm.engine.rest.impl.TaskRestServiceImpl.executeTaskQuery(TaskRestServiceImpl.java:116)
! at org.camunda.bpm.engine.rest.impl.TaskRestServiceImpl.queryTasks(TaskRestServiceImpl.java:96)
! at org.camunda.bpm.engine.rest.impl.TaskRestServiceImpl.getJsonTasks(TaskRestServiceImpl.java:71)
! at org.camunda.bpm.engine.rest.impl.TaskRestServiceImpl.getTasks(TaskRestServiceImpl.java:60)

When I access the workflow definition with the cockpit, I got the same error with message: Failed to display process definition **:**Received 500 from server.

And when I access a second time to the workflow definition, it works. It looks like an issue with the cache. Any clue ? Could it be due to bad data in the database, or a bug while building the java objects ?

Best regards,
Alexandre

Sreenidhi_KS · April 3, 2024, 11:40am

Hey @ahuet,
How did you resolve this issue? What was the reason? Can you help with the fix as well? Facing a similar issue. TIA

ahuet · April 3, 2024, 12:07pm

Hello,

There was a unknown property in a bpmn, a messageEventDefinition had the xml property: camunda:async=“true”, probably due to a bad copy/paste or maybe an older version of Camunda, I don’t know.

I have removed the property and also for existing jobs, I have deleted the entries in database as MessageEventDefinition should not be in this table:

delete from ACT_RU_JOBDEF where act_id_ like ‘%MessageEventDefinition%’

Hope it helps.

Alexandre

Sreenidhi_KS · April 4, 2024, 8:45am

Thanks for the quick reply, @ahuet ! Much appreciated. Will check this out.