When Camunda is down, It is not picking up the messages that came in during down time

Camunda 7 Topics
1

Hi

We have a Camunda application built using Spring Boot, which we use for workflow automation. Our main application stack is in React and Node.js, and the communication between our application and Camunda is managed via Kafka.

Here’s a brief overview of our setup:

  • We have a couple of workflows installed in Camunda.
  • Messages are published to a Kafka topic, which Camunda listens to.
  • Based on the received messages and DMN rules, Camunda starts a process instance for the corresponding workflow.

Problem: When Camunda goes down (e.g., due to maintenance or unexpected crashes), it doesn’t pick up the messages that were sent to the Kafka topic during the downtime. This results in missing process instances that were expected to be initiated based on those messages.

Potential Solution: We considered setting enable.auto.commit to false in Kafka and implementing manual commits. However, this might introduce another issue—potentially causing duplicate process instances if messages are re-read multiple times.

Question: Is there a recommended solution or workaround for this scenario? We want to ensure that:

  1. No messages are lost when Camunda is temporarily unavailable.
  2. Duplicate process instances are avoided when Camunda comes back online and resumes reading from the Kafka topic.

Any advice or suggestions on how to handle this situation effectively would be greatly appreciated!

Thanks in advance!

2

IMO this is not a camunda but rather a kafka related question – how to write a kafka consumer with the “exactly once” sematics.