Zeebe Failed to export record to ES

Hi Team

We could see the below error in one of zeebe logs where as this issue does not appear in other pods of zeebe.

Logs:
2024-05-13 03:34:46.807 [Broker-0] [zb-fs-workers-0] [Exporter-2] WARN
io.camunda.zeebe.broker.exporter.elasticsearch - Error on exporting record with key 4503599630758792
io.camunda.zeebe.exporter.ElasticsearchExporterException: Failed to put component template
at io.camunda.zeebe.exporter.ElasticsearchClient.putComponentTemplate(ElasticsearchClient.java:206) ~[zeebe-elasticsearch-exporter-8.3.0.jar:8.3.0]
at io.camunda.zeebe.exporter.ElasticsearchClient.putComponentTemplate(ElasticsearchClient.java:149) ~[zeebe-elasticsearch-exporter-8.3.0.jar:8.3.0]
Caused by: java.net.ConnectException: Connection refused

Other issue we have seen is the below error:

2024-05-13 07:57:05.156 [Broker-0] [zb-actors-1] [InterPartitionCommandReceiverActor-1] WARN
io.camunda.zeebe.broker.transport - Failed to write command MESSAGE_SUBSCRIPTION DELETE from 0 to logstream (error = FULL)

Also this issue is happening in load testing.
Looking for your views for the same.
Thanks

Just trying to brainstorm here - have you checked if the network connectivity between the erroneous zeebe node and elasticsearch cluster dropped for few seconds since the error states connection refused?

1 Like

Thanks for giving the clue but issue appears continuously in one zeebe pod.

Did you try networking tools like telnet or something from the failing node to confirm that connectivity is indeed an issue or not?

Sure will check when issue apperas again, Thank you so much

Also one of issue we do observed that sometime camunda zeebe is not activating jobs of a process in production environment.

how to dubug these kind of issue in zeebe side, please help to find the RCA.

Thanks

I am also facing similar issue, we did 8.3.0 version deployment, any more details to fix this issue?