Missing events when handling through DbHistoryEventHandler with separate history & runtime engines

Camunda 7 Topics Discussion & Questions
1

Hi everyone.
I’m trying to separate runtime & historical data using following documentation: provide-a-custom-history-backend. Currently I’m just duplicating historical data produced by process engine by sending HistoryEvents to separate camunda engine using kafka. Despite all historical data is consumed by history camunda side without any error - I’ve observed strong inconsistency of historical data stored at runtime & historical services(missing activity instances, variables updates, etc). After days of digging within this issue I have come to conclusion that there is some bug at implementation of DbHistoryEventHandler which leads to loss of data when events are handled one-by-one.

I’ve prepared small demo of this behaviour on my GitHub.
It mainly reads HistoricActivityInstanceEventEntity objects that were produced by camunda process from json file and passes them one-by-one to DbHistoryEventHandler on application startup.
Can someone please check it out and confirm if it’s a bug, or I’m just missing some crucial detail on my implementation? Any help would be highly appreciated. Thanks in advance.

Camunda version: 7.18.0
Spring Boot version: 2.7.3

STR:

  1. Fetch code from GitHub - Negling/camunda-history
  2. Run ‘docker-compose up’ under /docker folder to bootstrap postgres
  3. Run Application.java as spring-boot application
  4. Wait until EventHandlingExample.java complete its postProcessEngineBuild method(by “Finished processing history events!” line at application logs).
  5. Go to ‘localhost:8082/engine-rest/engine/yesCamundaHistoryService/history/activity-instance’ and observe results

ER:
Endpoint ‘localhost:8082/engine-rest/engine/yesCamundaHistoryService/history/activity-instance’ return 7 entries of HistoricActivityInstance containing all unique events from ‘/data/events.json’.

AR:
Endpoint ‘localhost:8082/engine-rest/engine/yesCamundaHistoryService/history/activity-instance’ return 1-6 entries of HistoricActivityInstance, with randomly missing entries, mostly last one with ‘activityName’ = ‘Task completed’.

Snippet of ‘postProcessEngineBuild’ method at EventHandlingExample.java:

public void postProcessEngineBuild(ProcessEngineImpl processEngine) {
        try (InputStream json = getClass().getClassLoader().getResourceAsStream("data/events.json")) {
            // read events from json one-by-one
            for (JsonNode node : objectMapper.readTree(json)) {
                var historyEvent = objectMapper.treeToValue(node, HistoryEvent.class);
                var processEngineConfiguration = processEngine.getProcessEngineConfiguration();

                processEngineConfiguration.getCommandExecutorTxRequired().execute(commandContext -> {
                    processEngineConfiguration.getHistoryEventHandler().handleEvent(historyEvent);
                    LOG.info("Processed history event: {}", historyEvent);

                    return null;
                });
                // simulate delay between events
                Thread.sleep((long) (Math.random() * 500 + 1));
            }
        } finally {
            LOG.info("Finished processing history events!");
        }
    }