I had a situation where inside the process instance, a token got stuck on Message Emit event, but since this was a while ago, and i started a lot of other processes, so now i cant find logs for the original issue (i have camunda in a docker container on a linux machine).
Is there a way for me to store or fetch the exceptions from camunda DB?
Hello my friend!
When you say Message Emit⌠do you mean Send Task?
When you say you stood still⌠did an incident break out in the camunda?
If you are referring to a Receive task or Message Catch Event, it is likely that camunda has not been able to correlate the message for some specific reason, which you will be able to verify in the logs, it could have been a âmismatchingCorrelationExceptionâ or âOptimisticLockingExceptionâ.
If it is one of these errors I mentioned, you can use the Camunda Events API through Postman or Insomnia to fire this event again and release your instance.
Are you unable to check the logs by searching for processInstanceId or some specific key?
William Robert Alves
@Hadi_Hoteit1 You can query those exception details from ACT_RU_INCIDENT and ACT_HI_INCIDENT tables.
Hi William , I have these two exception in my bpmn. I am not sure how to solve these exceptions
Hi @tasmiat ,
MismatchingCorrelationException is when an attempt is made to correlate a message with Camunda, but for some reason it cannot be done, and there can be several reasons, for example:
A correlation was sent but there was no event waiting for it.
There was an event, but the process instance had not yet reached this point to receive the correlation.
The instance was terminated or deleted before receiving the correlation.
The instance was not in the same scope as the message correlation.
Among other reasonsâŚ
The OptimisticLockingException occurs when a transaction is attempted concurrently in the database for the same entity, for example, trying to update 2 or more times the same variable at the same time.
If you want to share your BPM flow I can try to help model it in a better way so we can try to avoid this.
William Robert Alves
rpa_bib_connect_integrated_process_no_parallel.bpmn (20.4 KB)
I am getting this error
INSERT VariableInstanceEntity[6aa16587-3c4b-11ee-ac9d-16d424da2f2f]â failed. Entity was updated by another transaction concurrently.
org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of âINSERT VariableInstanceEntity[6aa16587-3c4b-11ee-ac9d-16d424da2f2f]â failed. Entity was updated by another transaction concurrently.
at org.camunda.bpm.engine.impl.db.EnginePersistenceLogger.concurrentUpdateDbEntityException(EnginePersistenceLogger.java:141)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.handleConcurrentModification(DbEntityManager.java:413)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperations(DbEntityManager.java:356)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flushDbOperationManager(DbEntityManager.java:323)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.flush(DbEntityManager.java:295)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.flushSessions(CommandContext.java:272)
at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:188)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:119)
at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.lambda$execute$0(SpringTransactionInterceptor.java:71)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
at org.camunda.bpm.engine.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:71)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:70)
at org.camunda.bpm.engine.impl.interceptor.CommandCounterInterceptor.execute(CommandCounterInterceptor.java:35)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:33)
at org.camunda.bpm.engine.impl.interceptor.ExceptionCodeInterceptor.execute(ExceptionCodeInterceptor.java:55)
at org.camunda.bpm.engine.impl.MessageCorrelationBuilderImpl.execute(MessageCorrelationBuilderImpl.java:322)
at org.camunda.bpm.engine.impl.MessageCorrelationBuilderImpl.correlateWithResult(MessageCorrelationBuilderImpl.java:233)
at ca.bell.bbm.orchestrator.service.JobMessageReceiveService.correlateMessage(JobMessageReceiveService.java:138)
at ca.bell.bbm.orchestrator.service.JobMessageReceiveService$$FastClassBySpringCGLIB$$a770b3fd.invoke()
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708)
at ca.bell.bbm.orchestrator.service.JobMessageReceiveService$$EnhancerBySpringCGLIB$$a8f2ba4f.correlateMessage()
at ca.bell.bbm.orchestrator.receiver.JobMessageReceiver.receiveMessage(JobMessageReceiver.java:60)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:169)
at org.springframework.messaging.handler.invocation.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:119)
at org.springframework.amqp.rabbit.listener.adapter.HandlerAdapter.invoke(HandlerAdapter.java:77)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandler(MessagingMessageListenerAdapter.java:263)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.invokeHandlerAndProcessResult(MessagingMessageListenerAdapter.java:209)
at org.springframework.amqp.rabbit.listener.adapter.MessagingMessageListenerAdapter.onMessage(MessagingMessageListenerAdapter.java:148)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:1670)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.actualInvokeListener(AbstractMessageListenerContainer.java:1589)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:1577)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:1568)
at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.executeListener(AbstractMessageListenerContainer.java:1512)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doReceiveAndExecute(SimpleMessageListenerContainer.java:994)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.receiveAndExecute(SimpleMessageListenerContainer.java:941)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.access$1600(SimpleMessageListenerContainer.java:85)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.mainLoop(SimpleMessageListenerContainer.java:1319)
at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer.run(SimpleMessageListenerContainer.java:1225)
at java.base/java.lang.Thread.run(Thread.java:833)
Without a doubt, one of the mistakes of your BPM is the one below.
When the correlation attempt is made, its 1 minute timer is interrupted, and it has already removed the instance of the receive task⌠making it impossible to correlate the message, and generating a mismatchingCorrelationException.
William Robert Alves
Consider using Async After for your Receive tasks, to avoid loss of message correlation in case of incidents.
And if you want to share with us the part of the code that is doing this correlation so we can see what might be causing your optimistic locking, that would be great.
William Robert Alves
private static Logger logger=LoggerFactory.getLogger( JobMessageReceiveService.class);
@Transactional
public String correlateMessage(byte[] bytes) throws IOException {
logger.info("+++++++++ JobMessageReceiveService - inside the custom correlatemessage method");
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.enable(DeserializationFeature.ACCEPT_EMPTY_STRING_AS_NULL_OBJECT);
JobMessageRequest jobMessageRequest = objectMapper.readValue(bytes, JobMessageRequest.class);
if (jobMessageRequest.getProcessInstanceId()!=null) {
jobMessageRequest.setProcessInstanceId(jobMessageRequest.getProcessInstanceId().trim());
}
logger.info("++++++ JobMessageReceiveService before map - jobMessageRequest: " + jobMessageRequest.toString());
Map<String, Object> map = new HashMap<String,Object>();
map.put(ProcessConstants.VAR_NAME_JOB_ID, jobMessageRequest.getJobId());
map.put(ProcessConstants.VAR_NAME_INTEGRATION_ID, jobMessageRequest.getIntegrationId());
map.put(ProcessConstants.VAR_NAME_JOB_TYPE, jobMessageRequest.getJobType());
map.put(ProcessConstants.VAR_NAME_MESSAGE, jobMessageRequest.getMessage());
map.put(ProcessConstants.VAR_NAME_PROJECT_CODE, jobMessageRequest.getProjectCode());
//Convert json message string to messageMap and combine the maps together
if (jobMessageRequest.getMessage()!=null) {
Map<String, Object> messageMap = objectMapper.readValue(jobMessageRequest.getMessage(), Map.class);
map.putAll(messageMap);
logger.info("++++++ JobMessageReceiveService - made all message fields into a map field");
}
logger.info("++++++ JobMessageReceiveService - GOING TO PRINT ALL MAP VALUES: " + jobMessageRequest.toString());
//print all values in map
Stream.of(map.keySet().toString())
.forEach(System.out::println);
logger.info("++++++ JobMessageReceiveService - FINISHED PRINTING ALL MAP VALUES: " + jobMessageRequest.toString());
//Convert to upper case for variable "jobStatus"
if (jobMessageRequest.getJobStatus()!=null) {
map.put(ProcessConstants.VAR_NAME_JOB_STATUS, jobMessageRequest.getJobStatus().toUpperCase());
logger.info("++++++ JobMessageReceiveService - converted to uppercase");
}
String processInstanceId = null;
MessageCorrelationResult result = null;
if (RpaJobAction.START_PROCESS_ACTION.getValue().equalsIgnoreCase(jobMessageRequest.getAction())) {
logger.info("++++++ JobMessageReceiveService - START_PROCESS_ACTION");
logger.info("+++++++++++++++++++++++++++++++++++++++++++++++++++++Inside START_PROCESS_ACTION and given action: "+jobMessageRequest.getAction());
String startProcessMessageReference=RpaJobType.valueOf(jobMessageRequest.getJobType().trim()).getProcessMessageReference();
logger.info("+++++++++++++++++++++++++++++++++++++++++++++++++++++->startProcessMessageReference="+startProcessMessageReference);
try {
Thread.sleep(5000);
} catch (InterruptedException e) {
e.printStackTrace();
}
result = runtimeService.createMessageCorrelation(startProcessMessageReference)
.processInstanceBusinessKey(jobMessageRequest.getJobId().toString()).setVariables(map)
.correlateWithResult();
processInstanceId = result.getProcessInstance().getProcessInstanceId();
}
else if (RpaJobAction.BOT_JOB_COMPLETED_ACTION.getValue().equalsIgnoreCase(jobMessageRequest.getAction())) {
logger.info("++++++ JobMessageReceiveService - BOT_JOB_COMPLETED_ACTION");
logger.info("+++++++++++++++++++++++++++++++++++++++++++++++++++++Inside BOT_JOB_COMPLETED_ACTION and given processInstanceId: "+jobMessageRequest.getProcessInstanceId());
ProcessInstance currentInstance = runtimeService.createProcessInstanceQuery().processInstanceId(jobMessageRequest.getProcessInstanceId()).singleResult();
if (currentInstance ==null) {
logger.error("++++++ Data error: there is no ProcessInstance for processInstanceId =" + jobMessageRequest.getProcessInstanceId());
}
else {
logger.info("++++++ JobMessageReceiveService - BOT_JOB_COMPLETED_ACTION - ELSE CONDITION");
logger.info("++++++ JobMessageReceiveService - BOT_JOB_COMPLETED_ACTION - ELSE CONDITION - going to call waitForActiviyIdsInProcess()");
int counter = waitForActiviyIdsInProcess(jobMessageRequest);
logger.info("++++++counter=" + counter);
//correlate the receiver task in the process using the global message referenced for receiver
logger.info("++++++ JobMessageReceiveService - BOT_JOB_COMPLETED_ACTION - ELSE CONDITION - going to call createMessageCorrelation()");
runtimeService.createMessageCorrelation(jobMessageRequest.getTaskId()+"_OUT")
.processInstanceId(jobMessageRequest.getProcessInstanceId())
.setVariables(map)
.correlateWithResult();
logger.info("++++++ JobMessageReceiveService - BOT_JOB_COMPLETED_ACTION - ELSE CONDITION - finished createMessageCorrelation()");
processInstanceId=jobMessageRequest.getProcessInstanceId();
logger.info("++++++ JobMessageReceiveService - BOT_JOB_COMPLETED_ACTION - ELSE CONDITION - set processInstanceId");
}
}
else if (RpaJobAction.BACKEND_TASK_COMPLETED_ACTION.getValue().equalsIgnoreCase(jobMessageRequest.getAction())) {
logger.info("++++++ JobMessageReceiveService - BACKEND_TASK_COMPLETED_ACTION");
logger.info("+++++++++++++++++++++++++++++++++++++++++++++++++++++Inside BACKEND_TASK_COMPLETED_ACTION and given processInstanceId: "+jobMessageRequest.getProcessInstanceId());
ProcessInstance currentInstance = runtimeService.createProcessInstanceQuery().processInstanceId(jobMessageRequest.getProcessInstanceId()).singleResult();
if (currentInstance ==null) {
logger.error("++++++ Backend Task Completed Data error: there is no ProcessInstance for processInstanceId =" + jobMessageRequest.getProcessInstanceId());
}
else {
logger.info("++++++ JobMessageReceiveService - BACKEND_TASK_COMPLETED_ACTION - ELSE CONDITION");
logger.info("++++++ JobMessageReceiveService - BACKEND_TASK_COMPLETED_ACTION - ELSE CONDITION - going to call waitForActiviyIdsInProcess()");
int counter = waitForActiviyIdsInProcess(jobMessageRequest);
logger.info("++++++Backend Task Completed counter=" + counter);
//correlate the receiver task in the process using the global message referenced for receiver
logger.info("++++++ JobMessageReceiveService - BACKEND_TASK_COMPLETED_ACTION - ELSE CONDITION - going to call createMessageCorrelation()");
runtimeService.createMessageCorrelation(jobMessageRequest.getTaskId()+"_OUT")
.processInstanceId(jobMessageRequest.getProcessInstanceId())
.setVariables(map)
.correlateWithResult();
logger.info("++++++ JobMessageReceiveService - BACKEND_TASK_COMPLETED_ACTION - ELSE CONDITION - finished createMessageCorrelation()");
processInstanceId=jobMessageRequest.getProcessInstanceId();
logger.info("++++++ JobMessageReceiveService - BACKEND_TASK_COMPLETED_ACTION - ELSE CONDITION - set processInstanceId");
}
} else {
logger.info("++++++ JobMessageReceiveService - NONE OF THE ABOVE");
if(jobMessageRequest.getAction()!=null) {
logger.info("++++++ Unidentified Action in jobMessageRequest =" + jobMessageRequest.getAction());
} else {
logger.info("++++++ jobMessageRequest is NULL");
}
}
logger.info("++++++ Returning processInstanceId =" + processInstanceId);
return processInstanceId;
}
/**
* @param jobMessageRequest
* @return
*/
private int waitForActiviyIdsInProcess(JobMessageRequest jobMessageRequest /*String processInstanceId*/) {
List<String> activeActivityIds = new ArrayList<>();
int counter = 0;
//wait for the task available
while (activeActivityIds==null || activeActivityIds.isEmpty() || !activeActivityIds.contains(jobMessageRequest.getTaskId()/*.toUpperCase()*/+"_OUT"/*ProcessConstants.CONSTANT_RECEIVE_RPA_MESSAGE_ACTIVITY*/ )) {
activeActivityIds = runtimeService.getActiveActivityIds(jobMessageRequest.getProcessInstanceId().toUpperCase());
logger.info("++++++ START ACTIVITY IDS");
for(String val:activeActivityIds) {
logger.info("++++++ activityId =" + val);
}
logger.info("++++++ END ACTIVITY IDS");
try {
Thread.sleep(500);
} catch (InterruptedException ie) {
logger.info(ie.getMessage());
}
counter++;
if (counter>20) {
logger.info("++++++Problems with processInstanceId =" + jobMessageRequest.getProcessInstanceId());
break;
}
}
return counter;
}
Added async after for my receive tasks and removed my one minute timer but still same optimictic lock error
*i get error at this point e[30m2023-08-16 13:12:04,388e[0;39m e[34mINFO e[0;39m [e[34morg.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1e[0;39m] e[33mca.bell.bbm.orchestrator.service.JobMessageReceiveServicee[0;39m: ++++++ START ACTIVITY IDS
e[30m2023-08-16 13:12:04,388e[0;39m e[34mINFO e[0;39m [e[34morg.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1e[0;39m] e[33mca.bell.bbm.orchestrator.service.JobMessageReceiveServicee[0;39m: ++++++ activityId =BIB_IN_UPDATE_BATCH_OUT
e[30m2023-08-16 13:12:04,388e[0;39m e[34mINFO e[0;39m [e[34morg.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1e[0;39m] e[33mca.bell.bbm.orchestrator.service.JobMessageReceiveServicee[0;39m: ++++++ END ACTIVITY IDS
e[30m2023-08-16 13:12:04,891e[0;39m e[34mINFO e[0;39m [e[34morg.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1e[0;39m] e[33mca.bell.bbm.orchestrator.service.JobMessageReceiveServicee[0;39m: ++++++Backend Task Completed counter=1
e[30m2023-08-16 13:12:04,891e[0;39m e[34mINFO e[0;39m [e[34morg.springframework.amqp.rabbit.RabbitListenerEndpointContainer#0-1e[0;39m] e[33mca.bell.bbm.orchestrator.service.JobMessageReceiveServicee[0;39m: ++++++ JobMessageReceiveService - BACKEND_TASK_COMPLETED_ACTION - ELSE CONDITION - going to call createMessageCorrelation()
org.camunda.bpm.engine.OptimisticLockingException: ENGINE-03005 Execution of âINSERT VariableInstanceEntity[05ea8c3d-3c58-11ee-bc14-16d424da2f2f]â failed. Entity was updated by another transaction concurrently.
No promises, but try setting Async Before as well as Async After on your message receive.
I think your delegate is updating the variables in a parallel transaction, which leads to your OLE.
tried this but still same error
All the tasks, not only receive tasks. 
sorry i shouldâve used more accurate description, and to answer your questions:
1- by Message Emit, i mean the token is stuck on Message Intermediate Throw Event, FYI i was running a script to start over 10K process instances, and just 1 of them got stuck on that Throw Event.
2- If im not mistaken Incident only happen if i had the Event on Async before or after no? i dont have any async on it, so no incidents are recorded.
Firing the event again is not an issue, but since I probably lost the logs (since it happened a while ago and millions of tasks have been executed since I doubt I can find it) I was interested if there is a way for me to record these exeptions in my own DB if camunda DB doesnât have them recorded outside of Incident
@Hadi_Hoteit1 No problem brother !
An incident would be generated anyway, even without the async before / after⌠what would change would be the position where the token of your instance would be showing the incident in the cockpit, as it would return to the last point in which there was a transaction in the camunda database⌠that is, the last point at which the instance stopped (user task / receive tasks / intermediate event / etcâŚ) or at the last point at which it had an asynchronous continuation.
If you want to share your BPM flow to check how your modeling is and show us at which stage your instance stopped, that would be great!
If the token in the camunda cockpit did not turn red, indicating an incident, it is likely that no incident has been generated, and in fact some other type of problem may have occurred, such as an optimisticLockingException or a mismatchingCorrelationException, and this will not be manifested through the cockpit, only in the logs.
William Robert Alves
Thatâs exactly what I meant The token in Camunda cockpit never turned Red, so as you said itâs probably another type of problem like the Exceptions you mentioned.
But is there a way for me to Record these exceptions, using Camunda components?
Iâve never done something like that so not sure, but would it be possible for example to put the entire process flow inside a subprocess, and add an error catch event on it, and somehow send that catched exception to an external system?
Also, another question, i was just checking my Camunda DB and I noticed my incident tables are completely empty, and I know I played a bit with the history level, could that be related to the history level changing? or is there a way to turn off incident recording that I did by accident maybe?
Hi William,
i have marked all the tasks as recceive task but i am still getting optimistic lock error
Hello my friend @Hadi_Hoteit1 !
If you are referring to the ACT_RU_INCIDENT table, it will only have records if there is an incident (red token) in the camunda cockpit, as these are runtime incidents.
If you are referring to the ACT_HI_INCIDENT table, this is indeed related to your change in the âhistory levelâ⌠however, I believe that if you have never done or configured a âhistory cleanupâ, after setting your history level, this table should still have any incidents, if any have occurred since the deployment of your Camunda application.
Now letâs talk about what can be done to catch your mistakesâŚ
In this case of optimisticLocking or mismatchingCorrelation, Camunda will most likely not generate an incident, as this âtheoreticallyâ is a safeguard to avoid unwanted concurrent transactions in the optimistic case, and the mismatch is telling you that the message you want to correlate is not it was possible, because Camunda was not waiting for her.
But as for capturing errors in general, you can wrap everything in a subprocess, capturing generic errors, or even with a subprocess of events.
An alternative that could be done for these cases in which you have an instance stopped for a long time, and without incidents generated, would be to understand the time it takes for an instance to finish the entire flow⌠and wrap your entire flow in a subprocess, adding an interruptive timer that shifts your instance to a treatment flow.
For example, the instance normally takes 24 hours to complete the entire flow⌠so put a timer for 48 hours⌠because every instance that stays there for 48 hours will be moved to a treatment flow and verification of possible errors.
OBS.: I suggest that when making the call for message correlation with camunda using the âMessageâ REST API, you put a Logger in your code to capture the Camunda API response and be able to have this record that is of great importance for check possible invisible problems in the Camunda Cockpit.
William Robert Alves