Hi,
I have been developing critical flows which has limited time to complete.
When I troubleshoot the logs, I see it loses some time during the db operations.
I use PostgreSql and this is only test database with no data, and I let spring to create schema so its created by default indexes, etc.
If you check below logs, it waits around 450 milliseconds for only a DELETE operation.
I see it loses around 1 second to reach only the start event every time its gets into a subflow (Call Activity).
This time passes in the process engine with db operations.
When I start my flow, I see camunda runs 20 db queries in a second for just a single flow. (History and metric is even not enabled)
Thinking that I would expect a high traffic on the prod, this could result in 1000 tps which makes me worry about it.
I really hope to hear some advises to minimize the db transaction numbers.
For instance, can I disable incidents which I don’t think I would have any use cases ?
Can I override org.camunda.bpm.engine.impl.persistence.entity.ProcessDefinitionManager.java to do extra caching because I don’t expect to deploy flows on the fly ? And I start camunda as springboot starter so I should have more options to customize how things work.
I finally found the issue after spending 3 days, and actually it was mentioned in the docs or blogs that
too many save points can result in performance issues. Save points increase the database interactions.
It was a long time ago and I just forgot that I added asyncBeforeTrue for process definitions in bpmnparselistener.
import lombok.extern.slf4j.Slf4j;
import org.camunda.bpm.engine.impl.bpmn.parser.AbstractBpmnParseListener;
import org.camunda.bpm.engine.impl.bpmn.parser.BpmnParseListener;
import org.camunda.bpm.engine.impl.pvm.process.ActivityImpl;
import org.camunda.bpm.engine.impl.pvm.process.ScopeImpl;
import org.camunda.bpm.engine.impl.util.xml.Element;
import org.camunda.bpm.engine.repository.ProcessDefinition;
import org.springframework.stereotype.Service;
@Slf4j
@Service
public class CustomBpmnParseListener extends AbstractBpmnParseListener implements BpmnParseListener {
@Override
public void parseStartEvent(Element startEventElement, ScopeImpl scope, ActivityImpl startEvent) {
if (scope instanceof ProcessDefinition) {
log.trace("Adding save point before Start Event '{}", startEvent.getId());
startEvent.setAsyncBefore(true);
}
}
}
The main purpose adding this at the first place was to allow the start process method to return the process id immediately.
ProcessInstance p = runtimeService.startProcessInstanceByKey(processDefinitionKey, businessKey, parametersMap);
But this small change led camunda to interact with database so often because my flows has too many call activities which means too many processes start and end in one single go.
So basically, I removed startEvent.setAsyncBefore(true); to resolve the issue.
Hi @guvenc
I’m really happy that you managed to solve the problem and wanted to say thanks a lot for posting your solution, i’m sure it’ll help other people in the community experiencing the same problem.