Multiple Camunda ProcessInsatnce getting created for single order

Camunda 8 Topics
,
1

Hii Community,
We are using camunda8 version 8.3.1 as self managed in production , recently we saw an issue , we use GCP as cloud platform and we saw that when zeebe resources are used more then requested . for one order more then one camunda process instances have been got created.

Here is the code i am using , i can see that zeebeClient.newCreateInstanceCommand() had been called only once from my orchestrator service but it has created processInstance multiple time at gap of less then one minute.
Can you please suggest why it has happened , am i assuming wrong and how to mitigate it.

Here is the sample code

@Transactional(rollbackFor = Exception.class)
  public void start(@Valid EventData order,
      Map<String, String> attributesMap) {
    try {
      boolean processAlreadyCreated =
          OrderRepository.findByOrderId(Order.orderId()).isPresent();

      if (!processAlreadyCreated) {
        log.info("Starting Order Orchestration: {}", order);
        Order.setOrderCreatedAt(order.orderCreatedAt() == null ?
            OffsetDateTime.now() : order.getOrderCreatedAt());

        OrderOrchestrator orderOrchestrator =
            orderOrchestratorTransactionalService.save(order, attributesMap);
        eventService.registerDataOrchestratorStateChangedEvent(orderOrchestrator, order, EVENT_ACTION_CREATE);

       orderOrchestratorProcessPort.create(order, attributesMap)
            .doOnError(error -> orderOrchestratorTransactionalService
                .setProcessAsError(orderOrchestrator.getId(), error))
            .subscribe(processInfo -> orderOrchestratorTransactionalService
                .setProcessAsStarted(orderOrchestrator.getId(), processInfo));
        eventService.registerDataOrchestratorStateChangedEvent(orderOrchestrator, order, EVENT_ACTION_CREATE);

        log.info("Started Fulfilment Order Orchestration: {}", fulfilmentOrder);
        if (!Objects.isNull(fulfilmentOrderOrchestrator)) {
         publishSomeEvent(###);
        }
      } else {
        log.info("Skipping Already Processed  Order: {}", order);
      }
    } catch (Exception ex) {
      log.error("Error to Start  Order Orchestration: {} {}", order, ex.getStackTrace());
      publishSomeEvent(***);
      throw ex;
    }
  }

  public Mono<ProcessDetail> create(OrderEventData order,
      Map<String, String> eventHeaders) {
   
    processVariables.put("someVaribable", "xyz");
    ProcessInstanceEvent result = zeebeClient.newCreateInstanceCommand()
        .bpmnProcessId(processKey)
        .latestVersion()
        .variables(processVariables)
        .send()
        .join();
    return Mono.just(ProcessUtil.toProcessDetail(result));
  }
2

To address the issue of multiple Camunda process instances being created for a single order, here’s a structured approach to identifying the cause and implementing a robust solution for Camunda 8 self-managed (e.g., with Zeebe):

Problem Recap

  • You are seeing more than one process instance created for a single order, despite the orchestration code only calling zeebeClient.newCreateInstanceCommand() once per order.
  • Code structures an idempotency check—OrderRepository.findByOrderId()—to avoid duplicate orchestration, then starts a process if not already found.

Likely Causes

1. Race Condition / Concurrency

  • If two or more requests for the same order ID are handled nearly simultaneously, and both pass the “not yet created” check before either inserts, they may both proceed to create a new process instance. This is a classic race condition.

2. At-Least-Once Delivery and Retries

  • If the communication with Zeebe (broker) is disrupted (network hiccup, timeout, process crash), your client may retry the instance creation, sometimes resulting in multiple instances if the original request actually succeeded server-side but the client didn’t receive the confirmation.

3. Transactional Boundaries

  • Using @Transactional help, but if the check-insert-then-start logic or the call to Zeebe is outside of a strict serializable/locked transaction, duplicates can still slip through.
  • As Zeebe’s API is external to your DB transaction, the database commit and process instance start are not atomic together.

Solutions & Remediation

A. Enforce Application-Level Idempotency with Database Locks

  • Use an atomic unique constraint (e.g., on order_id) in your order orchestration table. Let the DB enforce one orchestration per order.
  • When trying to create an orchestration, write to the table in such a way that otherwise simultaneous requests fail with a DB exception if a row already exists.
  • Only trigger zeebeClient.newCreateInstanceCommand() if the insert succeeds.

Example (Pseudo-SQL and Java):

try {
  insert into orchestrations (order_id, ...) values (?, ...); // unique constraint on order_id
  zeebeClient.newCreateInstanceCommand()...; // start process only now
} catch (DuplicateKeyException e) {
  // Already started for this order, so skip
}

B. Use External Business Keys for Idempotency

  • Camunda 8 Zeebe does NOT enforce uniqueness of business keys natively, but you can:
    • Model a process start event that only triggers on a unique message (e.g., use a message event with the order ID as the correlation key).
    • Send a “start-process-for-order-123” message instead of a CreateInstanceCommand.
    • Process model can ignore duplicate messages or mark the process as already started for that business key[1].

C. Handle Network Timeouts and Retries Gracefully

  • When you send a command to Zeebe and get a timeout, first check if the process instance for this order already exists before retrying creation[1].
  • Consider logging the Zeebe process instance key returned and persist it with your orchestration record, so you can check on previous execution attempts.

D. Overall Pattern

  1. Unique constraint on DB order orchestration.
  2. Only create process after confirmed insert.
  3. Make process stateless to handle occasional duplicates, or create as above with a “cancel duplicates” mechanism.

Additional Best Practices

  • If you scale out the orchestrator horizontally (more app instances), distributed locks (e.g., DB constraints or reliable distributed locks such as Redlock with Redis) are important.
  • Log all attempts to create a process instance for forensics.
  • Monitor for repeated process instance creation events to alert you to possible regressions.

Reference Patterns & Discussions

  • This pattern and its remedies are well-discussed in the Camunda and broader BPMN/Zeebe developer communities. Official documentation and StackOverflow suggest database-enforced uniqueness and idempotency logic as primary solutions to concurrent requests starting multiple process instances[1][2].
  • Camunda modelers sometimes address this at the BPMN model level, but the most bulletproof solution is at your persistence layer and the orchestration code in your service.

In summary:

  • Use a database unique constraint and only trigger instance creation after a successful insert.
  • Consider a message-based process start for even stronger idempotency, at the cost of a slightly more complex BPMN model.
  • Carefully handle retry logic and transactionality so unexpected network issues do not cause accidental duplication.

This approach will effectively solve the “multiple process instances for one order” problem in most Zeebe/Camunda 8 self-managed environments[1][2].

References:
[1] Is there a way to start only one process instance per business key https://stackoverflow.com/questions/71425593/is-there-a-way-to-start-only-one-process-instance-per-business-key
[2] Multiple Camunda ProcessInsatnce getting created for single order Multiple Camunda ProcessInsatnce getting created for single order
[3] Multiple Camunda ProcessInsatnce getting created for single order Multiple Camunda ProcessInsatnce getting created for single order
[4] Process Instance Modification | docs.camunda.org Process Instance Modification | docs.camunda.org
[5] Multi-instance - Camunda 8 Docs - Console Multi-instance | Camunda 8 Docs
[6] Transactions in Processes | docs.camunda.org Transactions in Processes | docs.camunda.org
[7] Slow processing of large multi instance collection #12946 - GitHub Slow processing of large multi instance collection · Issue #12946 · camunda/camunda · GitHub
[8] Deployment Distribution | Zeebe Chaos Deployment Distribution | Zeebe Chaos
[9] Multi Instance Process in Camunda - N47 Multi Instance Process in Camunda — N47
[10] Camunda Engine behaviour with massive multi-instances processes … https://stackoverflow.com/questions/60390500/camunda-engine-behaviour-with-massive-multi-instances-processes-and-ready-state
[11] Lightweight Processes with Camunda and Zeebe lambdaschmiede GmbH - IT Beratung
[12] What is a Process Instance? - Camunda What is a Process Instance? | Camunda
[13] Camunda 8 Parallel processing with MultiInstance Iteration Camunda 8 Parallel processing with MultiInstance Iteration
[14] Cannot create process instance after moving to camunda … https://stackoverflow.com/questions/68179311/cannot-create-process-instance-after-moving-to-camunda-cloud-lib
[15] Real-world BPMN 2.0 examples and answers to common questions. Real-world BPMN 2.0 examples and answers to common questions. | Camunda
[16] Process instance migration | Camunda 8 Docs Process instance migration | Camunda 8 Docs
[17] When creating an instance using … When creating an instance using createProcessInstanceCommand message start event is triggered · Issue #10536 · camunda/camunda · GitHub
[18] How to Use BPMN Subprocesses and Call Activities in a … - YouTube https://www.youtube.com/watch?v=HgqDqTHj9SI
[19] Dealing with problems and exceptions - Camunda 8 Docs Dealing with problems and exceptions | Camunda 8 Docs
[20] Routing events to processes Routing events to processes | Camunda 8 Docs
[21] Camunda 7 Performance and Scalability Camunda 7 Performance and Scalability | Camunda