Setting Data Retention Policy for Camunda Self Managed v8.2.0

Camunda 8 Topics Discussion & Questions
1

Hey Community!

Does camunda 8 self managed version supports Elasticsearch ILM configuration for data retention policy or we have to rely only on the Elasticsearch curator cron job?

Camunda Self Managed version: 8.2.0

2

As per documentation, it supports ILM.

3

@cpbpm The above reference link points to 8.4 version, but I’m using 8.2.0

4

Checked for 8.2 also.

5

@cpbpm Do you know exactly where can we find this application.yaml file for operate & tasklist?

I have read this docs, but unable to find where to apply this settings.

Helm chart shows config for ES curator only.

6

Let me check the helm chart and update you.

7

Found the value in the chart: It’s slightly different compare to the main documentation:

Retention can be used to define the data in Elasticsearch (ILM).

retention:
## @param operate.retention.enabled if true, the ILM Policy is created and applied to the index templates.
enabled: false
## @param operate.retention.minimumAge defines how old the data must be, before the data is deleted as a duration.
minimumAge: 30d

Link: camunda-platform-helm/charts/camunda-platform/values.yaml at 0b7b220ba69d1b949f5a9fbc3706da5550c1ae30 · camunda/camunda-platform-helm · GitHub

8

Hi @cpbpm Thanks for your reply. But this docs describing for version 8.4

9

@aravindhrs - I believe the application.yml mentioned is the application configuration file, not part of the Helm charts. In other words, it inside the deployed container. If you connect to your pod (for instance, with kubectl exec), you can edit the file and save it. I think the application will pick up the changes immediately, but maybe restart the pod just to be safe.

1 Like
10

Hi @nathan.loding , Thank you for helping in identifying the file and also it brought up the question that is:

If we add some configuration properties (ex: data retention configs) in the application.yaml file in operate and after sometime if we restart the operate service do we lose those configurations which we added or it will be preserved?

Also if kubernetes decided to bring up new pod for operate, then how does the updated data retention configs will be carried over to the default application.yaml file of operate?

11

@aravindhrs - just restarting the pod should not result in a loss of data, as I understand it. Kubernetes should retain the volume. However, the volumes aren’t persistent so if you tear it down (which happens during upgrades I believe) you will lose it, and new pods won’t have the configuration if you scale up.

I admit, I don’t know the best way to push a custom file into a pod during deployment. I will ask around and do some research and let you know what I find!

1 Like
12

@aravindhrs - here’s the answer from our SRE team:

The best way is to create a ConfigMap, and then make use of extraVolumes and extraVolumeMounts in each component in values.yaml to bind it in. I’ll give you an example:

optimize:
  extraVolumeMounts:
    - mountPath: /optimize/config/environment-config.yaml
      subPath: environment-config.yaml
      name: environmentConfig
  extraVolumes:
    - name: environmentConfig
      configMap:
        name: optimize-environment-config
        defaultMode: 0400

metadata:
  name: optimize-environment-config
apiVersion: v1
data:
  environment-config.yaml: |
     CONTENTS_OF_FILE
1 Like