Zeebe 8.7.0alpha3 when i launch the process instance its not getting updated in Elastic search

sree.s · January 22, 2025, 2:18pm

when i deploy the process instance in camunda-zeebe-8.7.0-alpha3 the process is getting deployed but i could not see any record in Elastic search index operate-list-view-8.3.0_ but when i run in 8.6.6 i can able to see the values

nathan.loding · January 22, 2025, 6:33pm

@sree.s - are you using a start execution listener by chance?

github.com/camunda/camunda

Process instances with start execution listeners are sometimes not visible in the Operate UI and REST API

opened 05:05PM - 14 Nov 24 UTC

ThorbenLindhauer

kind/bug support component/operate

### Describe the bug If we have a `start` execution listener on the process lev…el (i.e. defined on the `process` element, not the BPMN start event), it is possible that the process instance does not show up in the Operate UI and is not returned by the corresponding API endpoints (e.g. process instance query). The reason is a race condition during import between the process instance record and the job record for the execution listener. Note: This problem does not occur in the new exporter we are planning to deliver with 8.7, due to the race condition not existing there. * Backport to 8.6 * Also verify, test, and if necessary fix in new exporter on main branch * Note: On main we also need to fix it in the 8.6 importer code ### To Reproduce 1. Deploy a process model that has a `start` execution listener on the process level 1. Start a process instance 1. Let the job for the start listener be imported before the process instance * This happens by chance due to an importer race condition * It can be forced via remote debugging and orchestrating the importer threads with breakpoints in https://github.com/camunda/camunda/blob/stable/8.6/operate/importer-8_6/src/main/java/io/camunda/operate/zeebeimport/processors/ImportBulkProcessor.java#L122-L159 (i.e. let job records be imported first) ### Current behavior * The process instance does not become visible in the Operate UI ### Expected behavior * The process instance is displayed correctly * It has no activity badges yet, because at that stage no activity is running yet ### Rootcause * The importer imports different Zeebe value types out of order, so it can happen that jobs are handled before process instances * In https://github.com/camunda/camunda/blob/8.7.0-alpha1/operate/importer-8_7/src/main/java/io/camunda/operate/zeebeimport/processors/ListViewZeebeRecordProcessor.java#L639-L688 we update a flow node instance with job details, whenever a job is created. This has two problems: * `FlowNodeInstanceForListViewEntity` sets the list view's join relation property to `{"name": "activity", "parent": <value>}`. That means, the document is inserted into the list view index as a flow node instance, not a process instance. When the process instance record is handled after that, it upserts this document. The upsert does not overwrite the `joinRelation` value again. So the process instance is wrongfully declared an element instance * The code triggers for process instances which it is not intended for ### Workaround Declare the listener on the start event of the process ### Proposed Solution: * Exclude `ListViewZeebeRecordProcessor#updateFlowNodeInstanceFromJob` from handling process-instance-level records, so that it only updates true flow node instances. For this, we need to decide if the data that this method adds to the list view entity is important/relevant for process instances or not. If not, we can apply this change * Although this problem doesn't apply to the new exporter, to be consistent we should apply the same change there (i.e. not add job details to process instance documents in the list view index) This will lead to a new (but minor) bug, where the "failed but retries left" filter does not work on the process instances with failed EL jobs on process level, because the field does not get set correctly, but additionally also for 8.7 because the search request of the internal API includes a f[ilter for only flowNodeInstances](https://github.com/camunda/camunda/blob/dc945dbabf6ad522e3b55458d72f03ae0503ab36/operate/webapp/src/main/java/io/camunda/operate/webapp/elasticsearch/QueryHelper.java#L333)(aka activities). A new issue is created for this: TODO link issue for this bug. Other proposed solution: Fix the job-based update to also handle process instances. This is the higher complexity fix that may have more unforeseen side effects (as now process instances can start appearing before they are properly imported). Due to our timeline and the rather minor impact that the bug introduced through the easier fix will have, we decided to not proceed with this proposition. ### Links * https://jira.camunda.com/browse/SUPPORT-24288 * https://jira.camunda.com/browse/SUPPORT-24703 * https://app.slack.com/client/T0PM0P1SA/C07UED2BZC2 * https://jira.camunda.com/browse/SUPPORT-25043 ```[tasklist] ### Pull Requests - [ ] main: https://github.com/camunda/camunda/pull/27294 - [ ] 8.6: https://github.com/camunda/camunda/pull/27297 - [ ] 8.7: https://github.com/camunda/camunda/pull/27299 ```

sree.s · January 23, 2025, 5:03am

@nathan.loding no i an launching using the zeebe grpc with start event but its not getting displayed in the operator UI or even in tasklist where i deployed the resources

Zelldon · January 23, 2025, 10:34am

Hey @sree.s

can you share a bit more details on your configuration, how you deploy the platform, etc.?

Thanks and greets
Chris

sree.s · January 23, 2025, 10:56am

I am using camunda8.7.0alpha3 c8 windows run i run the elastic search and zeebe broker elastic version 8.14 and zeebe broker 8.7.0alpha3

nathan.loding · January 23, 2025, 4:17pm

@sree.s - just to clarify, are you launching a separate Elastic instance with a separate exporter? Or using the one bundled with C8 Run?

sree.s · January 24, 2025, 6:22am

Yes i am running the elastic search independently and trying to connect the zeebe broker with it

nathan.loding · January 24, 2025, 5:43pm

@sree.s - I’m not an expert in this area (I haven’t done much work with exporters yet), but I suspect a version mismatch issue - you are running Zeebe 8.7.0-alpha3 (released in Jan 2024), yet your exporter is 8.14, which is from Nov 2022. The exporter is looking for specific events, and that event stream evolves with each release.

You should be able to use C8 Run with a separate exporter, but I would recommend trying to update the exporter to 8.7.0-alpha3 to match your Zeebe instance, then see if the issue continues.