Camunda History Cleanup internal working

I’ve enabled camunda auto history cleanup, but it doesn’t removes all of the historic data whose removal time is more than current date and time. Below are my configuration for history cleanup.

application.properties config:

camunda.bpm.generic-properties.properties.historyCleanupBatchWindowStartTime=22:00
camunda.bpm.generic-properties.properties.historyCleanupBatchWindowEndTime=23:59
camunda.bpm.generic-properties.properties.historyCleanupStrategy= endTimeBased

BPMN config:

camunda:historyTimeToLive=“3”

So my questions are as follows.

  1. There total 18 history tables in camunda, so which tables are actually cleaned up when auto cleanup runs.

  2. What is the mechanism or internal working of auto history clean up.

  3. When history clean up runs it just deletes data from table but do not free disk space, is this the default behaviour of camunda? Is there any way we can reclaim the disk space as well?

It will be great if all or any of the questions asked is answered.

Thanks for your assistance

Hi @rohit74 to Answer your questions.

  1. There total 18 history tables in camunda, so which tables are actually cleaned up when auto cleanup runs.
    Except Case history all are cleaned up, as per official documentation. You can verify all tables having the “removal_time_” column populated in db and if that time is less that current time then your job executor should clean up that data. “removal_time_” Applicable only for Removal Time Based cleanup Strategy.

  2. What is the mechanism or internal working of auto history clean up.
    There are 2 cleanup strategies.
    Removal Time based & End Time based.

Default is Removal Time Based if you dont specify “historyCleanupStrategy”.

For endTimeBased the data to remove is calculated in run time when the Job executor runs based on endTime_+TTL.
endTimeBased strategy will only pick 4 history table as only 4 of them have this endTime_ column. Rest of it will be automatically picked up based on Join queries which are internal to Camunda product. Hence usually endTimeBased strategy is more resource intensive.

  1. When history clean up runs it just deletes data from table but do not free disk space, is this the default behaviour of camunda? Is there any way we can reclaim the disk space as well?
    You have to schedule periodic Vaccum for the database for which you want to cleanup diskspace. By default it will not reclaim diskspace.

If your data is not cleaned up you can check the camunda logs and there you will find the error if any during the Job Execution.

Hi @Sanjay_Vyas,

Thanks for responding,

  1. As per your response on point number 2, I should be using Removal time based clean up strategy because my concern is that history data should be remove after removal time but in my cleanup strategy I’ve mentioned end time based strategy due to which history data is not getting cleaned up!! Right?

  2. Can you please confirm how can I check internal logs of camunda as history cleanup runs automatically and no trace of logs is found under application logs.

  3. What is periodic vacuum, is there any way to configure this in camunda or do I have to write my own sql procedures to achieve the same?

Thanks,
Rohit Sinha

Answer to my problem is to use removalTimeBased cleanup strategy and configure history time to live either in the bpmn file itself or by using camunda’s API. Time to live configuration is mandatory for removalTimeBased strategy to work.

camunda.bpm.generic-properties.properties.historyCleanupStrategy=removalTimeBased

Configure time to live API

curl --location --request PUT ‘http://localhost:8080/engine-rest/process-definition/disconnectionSubprocess:1:9c2b6b4c-e686-11ec-8398-0a580a800e10/history-time-to-live
–header ‘Content-Type: application/json’
–data-raw ‘{
“historyTimeToLive”: 3
}’