Unhandled BPMN error raised from external task does not raise an incident

gaviriar · September 17, 2020, 10:41am

Hi There,

I have a bit of an interesting situation. I have configured the process engine to set enableExceptionsAfterUnhandledBpmnError property to true such that any unhandled BPMN errors raised (either by an external task or in the BPMN) cause a Process Engine Exception to be raised and in turn trigger an incident for the given task/element.

Below is an example of 2 BPMN’s one that raises an incident as expected, and the other that is problematic:

The workflow that works: Raises an incident on the last save point when the error is raised

I.e. the BPMN error event is raised, and an incident is raised.

The Problematic Case: When I have a BPMN that has a sequence of external tasks and one of the external tasks riases a BPMN error I end up with this little mess described below

The external task raises an BPMN error via the Java external-task-client. The logs look like this:

2020-09-17 12:21:26.593  INFO 24768 --- [criptionManager] c.r.a.i.h.a.camunda.CamundaTaskGrabber   : The External Task 82770ba4-f8cf-11ea-8eb8-0242ac110002 has been grabbed!
2020-09-17 12:21:26.609  WARN 24768 --- [criptionManager] c.r.a.i.h.a.camunda.CamundaTaskGrabber   : The External Task 82770ba4-f8cf-11ea-8eb8-0242ac110002 failed due to 'internal-system-error' while executing 'zip_files' method on 'local_gaviriar' connector: Failed to zip file 'C:\autolab\test\generic_methods\zipped\files.zip': C:\autolab\test\generic_methods\file_2.csv (The system cannot find the file specified)
2020-09-17 12:21:28.151 ERROR 24768 --- [criptionManager] org.camunda.bpm.client                   : TASK/CLIENT-03004 Exception on external task service method invocation for topic 'local_gaviriar':

org.camunda.bpm.client.exception.NotResumedException: TASK/CLIENT-01009 Exception while notifying a BPMN error: The corresponding process instance could not be resumed. Reason: status code: 500, reason phrase: {"type":"ProcessEngineException","message":"ENGINE-13033 Propagation of bpmn error internal-system-error failed. "}
	at org.camunda.bpm.client.impl.ExternalTaskClientLogger.externalTaskServiceException(ExternalTaskClientLogger.java:124) ~[camunda-external-task-client-1.3.1.jar:1.3.1]
	at org.camunda.bpm.client.task.impl.ExternalTaskServiceImpl.handleBpmnError(ExternalTaskServiceImpl.java:92) ~[camunda-external-task-client-1.3.1.jar:1.3.1]
	at net.jodah.failsafe.Functions.lambda$toSupplier$10(Functions.java:240) ~[failsafe-2.4.0.jar:2.4.0]
	at net.jodah.failsafe.Functions.lambda$get$0(Functions.java:48) ~[failsafe-2.4.0.jar:2.4.0]
	at net.jodah.failsafe.RetryPolicyExecutor.lambda$supply$0(RetryPolicyExecutor.java:62) ~[failsafe-2.4.0.jar:2.4.0]
	at net.jodah.failsafe.Execution.executeSync(Execution.java:129) ~[failsafe-2.4.0.jar:2.4.0]
	at net.jodah.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[failsafe-2.4.0.jar:2.4.0]
	at net.jodah.failsafe.FailsafeExecutor.run(FailsafeExecutor.java:210) ~[failsafe-2.4.0.jar:2.4.0]
	at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.handleExternalTask(TopicSubscriptionManager.java:152) ~[camunda-external-task-client-1.3.1.jar:1.3.1]
	at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.lambda$acquire$0(TopicSubscriptionManager.java:108) ~[camunda-external-task-client-1.3.1.jar:1.3.1]
	at java.base/java.util.Arrays$ArrayList.forEach(Arrays.java:4390) ~[na:na]
	at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.acquire(TopicSubscriptionManager.java:103) ~[camunda-external-task-client-1.3.1.jar:1.3.1]
	at org.camunda.bpm.client.topic.impl.TopicSubscriptionManager.run(TopicSubscriptionManager.java:87) ~[camunda-external-task-client-1.3.1.jar:1.3.1]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

On the camunda logs the BPMN error is received but it seems like it cannot get propagated. Here are the logs:

17-Sep-2020 10:24:32.921 SEVERE [http-nio-8080-exec-1] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context: ENGINE-13033 Propagation of bpmn error internal-system-error failed. 
	org.camunda.bpm.engine.ProcessEngineException: ENGINE-13033 Propagation of bpmn error internal-system-error failed. 
		at org.camunda.bpm.engine.impl.cmd.CommandLogger.exceptionBpmnErrorPropagationFailed(CommandLogger.java:214)
		at org.camunda.bpm.engine.impl.persistence.entity.ExternalTaskEntity.bpmnError(ExternalTaskEntity.java:400)
		at org.camunda.bpm.engine.impl.cmd.HandleExternalTaskBpmnErrorCmd.execute(HandleExternalTaskBpmnErrorCmd.java:62)
		at org.camunda.bpm.engine.impl.cmd.HandleExternalTaskCmd.execute(HandleExternalTaskCmd.java:59)
		at org.camunda.bpm.engine.impl.cmd.HandleExternalTaskCmd.execute(HandleExternalTaskCmd.java:31)
		at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
		at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
		at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
		at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
		at org.camunda.bpm.engine.impl.ExternalTaskServiceImpl.handleBpmnError(ExternalTaskServiceImpl.java:79)
		at org.camunda.bpm.engine.rest.sub.externaltask.impl.ExternalTaskResourceImpl.handleBpmnError(ExternalTaskResourceImpl.java:150)
		at jdk.internal.reflect.GeneratedMethodAccessor552.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
		at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
		at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
		at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138)
		at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107)
		at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133)
		at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101)
		at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406)
		at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)
		at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
		at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
		at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
		at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
		at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
		at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
		at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
		at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
		at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
		at org.camunda.bpm.engine.rest.security.auth.ProcessEngineAuthenticationFilter.doFilter(ProcessEngineAuthenticationFilter.java:155)
		at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
		at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
		at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:45)
		at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
		at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
		at org.camunda.bpm.engine.rest.filter.EmptyBodyFilter.doFilter(EmptyBodyFilter.java:99)
		at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
		at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
		at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
		at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
		at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
		at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
		at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
		at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
		at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
		at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
		at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
		at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
		at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
		at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
		at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
		at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
		at java.base/java.lang.Thread.run(Thread.java:834)
	Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-02042 Execution with id 'Activity_0lql8bs' throws an error event with errorCode 'internal-system-error', but no error handler was defined. 
		at org.camunda.bpm.engine.impl.bpmn.behavior.BpmnBehaviorLogger.missingBoundaryCatchEventError(BpmnBehaviorLogger.java:267)
		at org.camunda.bpm.engine.impl.bpmn.helper.BpmnExceptionHandler.propagateError(BpmnExceptionHandler.java:133)
		at org.camunda.bpm.engine.impl.bpmn.helper.BpmnExceptionHandler.propagateBpmnError(BpmnExceptionHandler.java:94)
		at org.camunda.bpm.engine.impl.persistence.entity.ExternalTaskEntity.bpmnError(ExternalTaskEntity.java:398)
		... 56 more
17-Sep-2020 10:24:32.922 WARNING [http-nio-8080-exec-1] org.camunda.commons.logging.BaseLogger.logWarn ENGINE-REST-HTTP500 org.camunda.bpm.engine.ProcessEngineException: ENGINE-13033 Propagation of bpmn error internal-system-error failed. 
	at org.camunda.bpm.engine.impl.cmd.CommandLogger.exceptionBpmnErrorPropagationFailed(CommandLogger.java:214)
	at org.camunda.bpm.engine.impl.persistence.entity.ExternalTaskEntity.bpmnError(ExternalTaskEntity.java:400)
	at org.camunda.bpm.engine.impl.cmd.HandleExternalTaskBpmnErrorCmd.execute(HandleExternalTaskBpmnErrorCmd.java:62)
	at org.camunda.bpm.engine.impl.cmd.HandleExternalTaskCmd.execute(HandleExternalTaskCmd.java:59)
	at org.camunda.bpm.engine.impl.cmd.HandleExternalTaskCmd.execute(HandleExternalTaskCmd.java:31)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:28)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:110)
	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
	at org.camunda.bpm.engine.impl.ExternalTaskServiceImpl.handleBpmnError(ExternalTaskServiceImpl.java:79)
	at org.camunda.bpm.engine.rest.sub.externaltask.impl.ExternalTaskResourceImpl.handleBpmnError(ExternalTaskResourceImpl.java:150)
	at jdk.internal.reflect.GeneratedMethodAccessor552.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
	at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:138)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:107)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:133)
	at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:101)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
	at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.camunda.bpm.engine.rest.security.auth.ProcessEngineAuthenticationFilter.doFilter(ProcessEngineAuthenticationFilter.java:155)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:45)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.camunda.bpm.engine.rest.filter.EmptyBodyFilter.doFilter(EmptyBodyFilter.java:99)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.camunda.bpm.engine.ProcessEngineException: ENGINE-02042 Execution with id 'Activity_0lql8bs' throws an error event with errorCode 'internal-system-error', but no error handler was defined. 
	at org.camunda.bpm.engine.impl.bpmn.behavior.BpmnBehaviorLogger.missingBoundaryCatchEventError(BpmnBehaviorLogger.java:267)
	at org.camunda.bpm.engine.impl.bpmn.helper.BpmnExceptionHandler.propagateError(BpmnExceptionHandler.java:133)
	at org.camunda.bpm.engine.impl.bpmn.helper.BpmnExceptionHandler.propagateBpmnError(BpmnExceptionHandler.java:94)
	at org.camunda.bpm.engine.impl.persistence.entity.ExternalTaskEntity.bpmnError(ExternalTaskEntity.java:398)
	... 56 more

Ideally what I would like to see is an incident raised in the corresponding external task which raised the BPMN error. Am I missing something here? Or is there some unexpected behaviour in the process engine? I would image it is the former

Thanks in advance for reading this and for your inputs!

Niall · September 17, 2020, 1:17pm

Hi @gaviriar

Welcome along to the forum. So the answer can actually be a little bit complicated and involves understanding how transactions work in the engine. BUT I think i can give you a pretty quick explaining of whats happening. So i built this model.

As you can see by the name - everything will go great until my script at the end fails. After you start the process and complete it, in cockpit will look like this:

So it might strike you as odd that error is on the first task, but it’s just because the error happened on the This will fail task and rolled back to the This is fine task because it was the last known wait state.

So - what if you would like to ensure the error will appear in the right place and not roll back over a bunch of tasks, well you’ll need to create a wait state at the point where you’ll like to commit your state so if an error happens it will roll it back to that point. Which just involves ticking a box.

By adding this it’ll commit it’s state before running the task. so cockpit will now look like this:

Anyway - the answer to your question, in short is tick asynchronous before on your task that throws the error.

fml2 · September 17, 2020, 4:29pm

There is a setting enableExceptionsAfterUnhandledBpmnError in the process engine that controls whether an unhandled BpmnError causes an incident. I also had such a case and was very surprised that the process just completed. This is the default behaviour is false, i.e. just swallow it!

gaviriar · September 21, 2020, 10:28am

Hi @Niall and @fml2,

Thank you for your swift reply.

@fml2, in the case, above I am already setting enableExceptionsAfterUnhandledBpmnError=true.

@Niall thanks for the explanation. I followed your advice, and I have set asynchronous before for all external service tasks. What I am observing is that:

If the error is either an error boundary event then there is an incident in “This will work” external task due to raised error end event.
If the error is raised from an external service task via the handleBpmnError API then the incident isn’t raised on the failing task but rather the token gets stuck at that task

I would have expected case 1 and 2 to behave the same. But that is not quite the case (at least with my setup). Any inputs here would be very valuable

Thanks in advance!

gaviriar · September 29, 2020, 11:38am

Hi @Niall, just checking to see if you saw my message above. I would really appreciate some help here as I am a bit stuck.

Please let me know if an example bpmn and some sample code would be of value and I can prepare some

vinodgaikwad · October 3, 2022, 12:17pm

Hi @Niall , do we have any option to remove these failed task permanently either it could be script issue / java any exception issue ? because in my case couple of tasks which are failed are there and i need to do it manually one by one ? can we process /remove those failed tasks in one go? how we can do this ? how to collect the data for faileds events like id?/history of same? kindly help

vinodgaikwad · October 4, 2022, 11:34am

waiting fr your reply on this pls @Niall

Niall · October 7, 2022, 8:20am

when you say remove - do you mean to cancel the instances where there is an error?

vinodgaikwad · October 7, 2022, 9:03am

yes @Niall…to remove it from flow so that i can not able to see it anymore in the flow

Niall · October 7, 2022, 9:06am

You can query for all the process instances which have an incident and then you can run a batch with this call to remove them all.