Variable Ingestion breaking process instance import into Optimize

Camunda 8 Topics
1

Hi,

I’ve already posted this very issue under Variable Ingestion breaking process instance import into Optimize · Issue #1048 · camunda/issues · GitHub but I feel like it might also be worthwhile to discuss here:

Describe the bug

We ingest variables into Camunda Optimize via the variable ingestion API through a service worker. This happens right after a process instance is started as the first task in the process. The REST call to /api/ingestion/variable succeeds (HTTP 200) and the process succesfully completes. However, the process instance and any future process instances of this process never show up in Optimize (but are present in Operate).

This is likely caused by the variable ingestion happening before the process instance is imported into Optimize. This theory is further supported by the fact that the bug is not 100% reproducible, but will only occur some of the time (potentially meaning that there are cases where import is faster than ingestion and vice versa). Starting multiple process instances will eventually lead to this race condition being observable, just be aware that starting a single instance will work fine some of the time.

This behaviour was first observed on a SaaS-Cluster and later reproduced in a local Docker setup using Camunda 8.9’s docker-compose-full.yml

To Reproduce

  1. Deploy a process containing a service task with Variable Ingestion (calling /api/ingestion/variable) as the first task of the process.

  2. Start process application with job worker.

  3. Start a process instance (in our case through web modeler).

  4. On job execution: call the variable ingestion API with a payload referencing that process instance key from the worker.

  5. Verify the API responds with HTTP 200.

  6. Verify process instance is recorded in Operate and succesfully completed.

Expected behavior

The process instance for which the variable was ingested shows up in Optimize.

Log/Stacktrace

Variable Ingestion (Camunda Client):

2026-04-30T10:54:37.789+02:00  INFO 3510 --- [carbo] [pool-2-thread-1] d.e.c.a.optimize.client.OptimizeClient   : Ingesting variable to Optimize: OptimizeVariableDTO[id=fce95132-4e26-4c35-9882-5e24fa42f9e8, name=executionDuration-doSomething, type=long, value=206, processInstanceId=2251799813687637, processDefinitionKey=Process_Ingestion_with_no_wait]
2026-04-30T10:54:38.117+02:00  INFO 3510 --- [carbo] [pool-2-thread-1] d.e.c.a.optimize.client.OptimizeClient   : Optimize response: <200 OK OK,[Cache-Control:"no-store", Strict-Transport-Security:"max-age=63072000; includeSubDomains", X-Content-Type-Options:"nosniff", Content-Security-Policy:"base-uri 'self'", X-XSS-Protection:"1; mode=block", Content-Length:"0", Date:"Thu, 30 Apr 2026 08:54:38 GMT", Keep-Alive:"timeout=60", Connection:"keep-alive"]>

Known Workaround

If a timer intermediate catch event with a duration of ~60 seconds is inserted before the ingesting service task, the problem no longer occurs.

Environment:

  • OS: WSL (Win11 + Ubuntu 22.04)

Cluster (self-managed):

  • Camunda Version: 8.9:
## Image versions ##
# renovate: datasource=docker depName=camunda/camunda
CAMUNDA_VERSION=8.9.1
# renovate: datasource=docker depName=camunda/connectors-bundle
CAMUNDA_CONNECTORS_VERSION=8.9.1
# renovate: datasource=docker depName=camunda/identity
CAMUNDA_IDENTITY_VERSION=8.9.1
# renovate: datasource=docker depName=camunda/operate
CAMUNDA_OPERATE_VERSION=8.9.1
# renovate: datasource=docker depName=camunda/tasklist
CAMUNDA_TASKLIST_VERSION=8.9.1
# renovate: datasource=docker depName=camunda/optimize
CAMUNDA_OPTIMIZE_VERSION=8.9.1
# renovate: datasource=docker depName=camunda/web-modeler-restapi
CAMUNDA_WEB_MODELER_VERSION=8.9.1
# renovate: datasource=docker depName=camunda/console
CAMUNDA_CONSOLE_VERSION=8.9.26
# renovate: datasource=docker depName=elasticsearch
ELASTIC_VERSION=8.19.11
KEYCLOAK_SERVER_VERSION=26.3.2
# renovate: datasource=docker depName=axllent/mailpit
MAILPIT_VERSION=v1.21.8
POSTGRES_VERSION=15-alpine3.22
  • Configuration: Optimize variable ingestion API enabled

Cluster (SaaS):

  • Camunda Version: Camunda 8.9+gen3

  • Configuration: Optimize variable ingestion API enabled

2

This appears to be a documented race condition where external variable ingestion happens before the process instance is imported into Optimize, preventing proper association. I found the following relevant resources:

Does this help? If not, can anyone from the community jump in? :waving_hand:

:light_bulb: Hints: Use the Ask AI feature in Camunda’s documentation to chat with AI and get fast help. Report bugs and features in Camuda’s GitHub issue tracker. Trust the process. :robot:

3

To clarify:

  • The issue is not only about ingested variables not showing up in Optimize, but more importantly about the ingestion breaking future imports of process instances. This exceeds the descriptions in the docs.
  • Although the optimize logs are (naturally) very verbose at DEBUG, I couldn’t find any clear indication of the variable not being ingested or correlated properly. Together with the non-descript 200 response of the rest call, I believe this issue might benefit from increased visibility in the self-managed system.