History Clean up

Hi,

I am looking at options to purge the process instance data using out of the box Camunda Clean up capabilities.

Our Camunda version is 7.14 community version.

I went through the user guide here History and Audit Event Log | docs.camunda.org

I am struck with a few issues here.

  1. The removal time was not getting calculated.

(Check the configuration in processes.xml attached here. I have this configuration in resources/META-INF/).

I am able to get this working by defining an absolute time. (I was just trying if this works, I want the time to be calculated only)

Someone else also ran into the same issue earlier and unfortunately I couldnt find a solution there Cleaning up the history process instances.

  1. While I was going through Batch operations | docs.camunda.org, I can see that we are having to give the process instance id’s for setting the removal time. Currently in the database, we have millions of process instances, and to provide the process instance id’s as mentioned in the link will be challenging. Is there any other way to deal with the situation ?

Thanks,
Hari.

Hello @hari_kiran ,

do you have defined history TTL on process definitions or process engine level? If yes, the calculated time should work.

If you plan to clean up the engine on a regular basis, please make sure that every process definition has a TTL or the engine is configured with a default historyTimeToLive property.

Jonathan

Hi @jonathan.lukas

I have the below properties set.

camunda:
bpm:
admin-user:
id: demo
password: demo
generic-properties:
properties:
historyCleanupBatchWindowStartTime: “13:53”
historyCleanupBatchWindowEndTime: “23:59”
historyCleanupStrategy: removalTimeBased
historyTimeToLive: “P1D”
historyRemovalTimeStrategy: “end”
historyCleanupEnabled: true

And below is the API call that I am using for test purpose as of now. This is not calculating the removal_time_

@PostMapping("/startCleanupBatch")
public void test(){
    HistoricProcessInstanceQuery query = historyService
            .createHistoricProcessInstanceQuery().completed();

    Batch batch = historyService.setRemovalTimeToHistoricProcessInstances()
            .calculatedRemovalTime().byIds("<ProcessinstanceIDsGoHere>")
            .byQuery(query)
            .hierarchical()
            .executeAsync();
}

But, the code when changed to absoluteRemovalTime() instead of calculatedRemovalTime() as below is setting the removal time.

   Batch batch = historyService.setRemovalTimeToHistoricProcessInstances()
            .absoluteRemovalTime(new Date(System.currentTimeMillis()))
            .byQuery(query)
            .hierarchical()
            .executeAsync();

Am I missing something here when using calculatedRemovalTime() ?

Hello @hari_kiran ,

you could also consider setting the history TTL value on each deployed process definition.

https://docs.camunda.org/rest/camunda-bpm-platform/7.19-SNAPSHOT/#tag/Process-Definition/operation/updateHistoryTimeToLiveByProcessDefinitionId

Jonathan

Hi @jonathan.lukas

Thanks for your reply.

There is a lot of data in one of the environment which I plan to purge. And this is how I plan to do it.

  1. To set the TTL to process definitions - This will ensure the future instances will have a removal_time_ set when the process instance ends.
  2. To set removal_time_ by invoking a batch job which does that for existing instances.

So here in my usecase, I am trying to achieve #2 of it where I can set removal_time_ for existing instances.

Now, in my scenario I am dealing with an older version of Camunda i.e 7.14. So I am not sure if I can update the TTL for existing process definitions in that version (I didnt find something similar to what you have shared for 7.19 in 7.14).

Regards,
Srihari.

Hello @hari_kiran ,

for 7.14, the endpoint is documented here: Update history time to live | docs.camunda.org

Jonathan