Configuring Elasticsearch Exporter to Minimize Latency

Hello everyone,
However, I’m encountering an issue where the Elasticsearch indices do not seem to reflect the exported data from Camunda. Here is a summary of my setup:

  • Camunda Configuration: I have configured the Elasticsearch exporter in my application.yml as follows:
exporters:
  elasticsearch:
    className: io.camunda.zeebe.exporter.ElasticsearchExporter

    args:
      url: http://localhost:9200

      bulk:
        delay: 0
        size: 0
        memoryLimit: 0

      retention:
        enabled: true
        minimumAge: 30d
        policyName: zeebe-records-retention-policy

      authentication:
        username: elastic
        password: changeme

      index:
        prefix: test-record
        createTemplate: true

Observation: Despite configuring the exporter and confirming that Camunda is exporting data to Elasticsearch, when I check http://localhost:9200/_cat/indices, I do not see any indices matching the configured prefix (test-record*).

Could someone please advise on what might be causing this issue? Are there additional steps or configurations I should consider to ensure that the Elasticsearch indices are correctly reflecting the data exported from Camunda?

Here are some specific questions we have:

  1. Bulk Size and Delay: What are the recommended settings for bulk.size and bulk.delay to minimize latency while maintaining good performance?
  2. Indexing: Are there any specific indexing configurations or mapping settings we should use in Elasticsearch to improve the indexing speed of Camunda data?
  3. Performance: Are there other performance settings in Elasticsearch or Camunda that we should consider adjusting to reduce latency?
  4. Debug Logs: We have enabled debugging for the Elasticsearch exporter as shown above. What types of log messages should we monitor to diagnose latency issues?
  5. Best Practices: What are the best practices for configuring and maintaining the Elasticsearch exporter with Camunda to ensure low latency and high availability?

We appreciate any help or experiences you can share on this topic. Thank you in advance for your support!

Best regards,