Zeebe Failed to export record to ES

[rahulmca.rns]

Hi Team

We could see the below error in one of zeebe logs where as this issue does not appear in other pods of zeebe.

Logs:
2024-05-13 03:34:46.807 [Broker-0] [zb-fs-workers-0] [Exporter-2] WARN
io.camunda.zeebe.broker.exporter.elasticsearch - Error on exporting record with key 4503599630758792
io.camunda.zeebe.exporter.ElasticsearchExporterException: Failed to put component template
at io.camunda.zeebe.exporter.ElasticsearchClient.putComponentTemplate(ElasticsearchClient.java:206) ~[zeebe-elasticsearch-exporter-8.3.0.jar:8.3.0]
at io.camunda.zeebe.exporter.ElasticsearchClient.putComponentTemplate(ElasticsearchClient.java:149) ~[zeebe-elasticsearch-exporter-8.3.0.jar:8.3.0]
Caused by: java.net.ConnectException: Connection refused

Other issue we have seen is the below error:

2024-05-13 07:57:05.156 [Broker-0] [zb-actors-1] [InterPartitionCommandReceiverActor-1] WARN
io.camunda.zeebe.broker.transport - Failed to write command MESSAGE_SUBSCRIPTION DELETE from 0 to logstream (error = FULL)

Also this issue is happening in load testing.
Looking for your views for the same.
Thanks

[jgeek1]

Just trying to brainstorm here - have you checked if the network connectivity between the erroneous zeebe node and elasticsearch cluster dropped for few seconds since the error states connection refused?

[rahulmca.rns]

Thanks for giving the clue but issue appears continuously in one zeebe pod.

[jgeek1]

Did you try networking tools like telnet or something from the failing node to confirm that connectivity is indeed an issue or not?

[rahulmca.rns]

Sure will check when issue apperas again, Thank you so much

[rahulmca.rns]

Also one of issue we do observed that sometime camunda zeebe is not activating jobs of a process in production environment.

how to dubug these kind of issue in zeebe side, please help to find the RCA.

Thanks

[Gokul_Skumar]

@rahulmca.rns

zeebe

I am also facing similar issue, we did 8.3.0 version deployment, any more details to fix this issue?