Data retention and retrieval in Camunda

Camunda 8 Topics
1

We are using OpenSearch as the Zeebe engine database in our self-managed set-up of Camunda. We are using another instance of Opensearch (cross-cluster replication) in a Leader -Follower configuration, Active-Passive set-up for retaining, backing up our data for a longer period of time. If the customer wants to retrieve the historic data from the back-up instance of open search and showcase that in the Operate dashboard, what is the best option for us to pull the historic data and populate it in the operate or optimize dashboards?

2

Hi @KiranBee,

Thank you for your question about retrieving historic data from your backup OpenSearch instance and displaying it in Operate/Optimize dashboards.

Based on the current Camunda documentation and architecture, here are the key points regarding your setup:

Current Limitations

Single Data Source Configuration: Operate and Optimize are designed to connect to a single OpenSearch instance at a time. There’s no built-in support for:

  • Federated queries across multiple OpenSearch clusters
  • Multi-source data aggregation
  • Direct integration with cross-cluster replication setups

Recommended Approaches

Option 1: Separate Operate Instance for Historic Data

Deploy a dedicated Operate instance configured to connect to your backup OpenSearch cluster:

camunda.operate:
  database: opensearch
  opensearch:
    url: https://your-backup-opensearch.example.com:9200
    username: your-username
    password: your-password
    clusterName: opensearch

This would give you a separate dashboard specifically for historic data retrieval.

Option 2: Use Official Backup/Restore Process

The officially supported approach for data retention and retrieval is using OpenSearch snapshots:

This method ensures data consistency across all Camunda components.

Option 3: Custom Integration

For your specific cross-cluster replication setup, you might need to develop a custom solution that:

  1. Queries the backup OpenSearch instance directly
  2. Presents the data through a custom interface
  3. Potentially uses Operate’s APIs to supplement the data

Important Considerations

  • Cross-cluster replication for historic data retrieval isn’t covered in the official Camunda documentation
  • The backup OpenSearch instance would need to maintain the same index structure and mappings as the primary instance
  • Data consistency between active and backup instances needs to be carefully managed

Would you like me to elaborate on any of these approaches, or do you have specific requirements that might help narrow down the best solution for your use case?

References: