Unexpected exception occurred on periodically flushing bulk

[mhais]

Hello dear Camunda folks!

I’ve found this WARN message in zeebe logs in our DEV cluster and wondering if that can produce any problems for us in future and the ways how to fix this message.

We use Camunda 8.2.7 version.

Thanks in advance.

[Broker-0-Exporter-1] [Broker-0-zb-fs-workers-2] WARN 
      io.camunda.zeebe.broker.exporter.elasticsearch - Unexpected exception occurred on periodically flushing bulk, will retry later.
io.camunda.zeebe.exporter.ElasticsearchExporterException: Failed to flush bulk request: [Failed to flush 818 item(s) of bulk request [type: validation_exception, reason: Validation Failed: 1: this action would add [1] shards, but this cluster currently has [2000]/[2000] maximum normal shards open;]]
	at io.camunda.zeebe.exporter.ElasticsearchClient.throwCollectedBulkError(ElasticsearchClient.java:183) ~[zeebe-elasticsearch-exporter-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.exporter.ElasticsearchClient.exportBulk(ElasticsearchClient.java:166) ~[zeebe-elasticsearch-exporter-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.exporter.ElasticsearchClient.flush(ElasticsearchClient.java:107) ~[zeebe-elasticsearch-exporter-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.exporter.ElasticsearchExporter.flush(ElasticsearchExporter.java:166) ~[zeebe-elasticsearch-exporter-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.exporter.ElasticsearchExporter.flushAndReschedule(ElasticsearchExporter.java:152) ~[zeebe-elasticsearch-exporter-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.scheduler.ActorJob.invoke(ActorJob.java:94) [zeebe-scheduler-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.scheduler.ActorJob.execute(ActorJob.java:45) [zeebe-scheduler-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.scheduler.ActorTask.execute(ActorTask.java:119) [zeebe-scheduler-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask(ActorThread.java:106) [zeebe-scheduler-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.scheduler.ActorThread.doWork(ActorThread.java:87) [zeebe-scheduler-8.2.7.jar:8.2.7]
	at io.camunda.zeebe.scheduler.ActorThread.run(ActorThread.java:198) [zeebe-scheduler-8.2.7.jar:8.2.7]

[krishnakumar797]

We also encountering the same issue, any solution for this?

[aravindhrs]

@nathan.loding Do we have any suggestions or findings for this issue?

[nathan.loding]

@mhais, @krishnakumar797, and @aravindhrs - the root cause is in the error message: the maximum number of shards has been reached. The fix is to temporarily increase the shards so the processing can finish, then implement data retention policies. If your data retention needs are greater than the current Elasticsearch capacity, then I believe the recommendation is to add another ES node and begin to further scale ES.

I am not an expert here though. If you have an Enterprise plan with Camunda, I would recommend reaching out to the support team. The specific fix is very likely going to be environment specific.