History cleanup in a more dynamic way

Hi,

Because we had problems with amount of historic data in the past we implemented a mechanism which deletes historic data after a number of days which can be configured.

The idea is similar to camunda history cleanup available since version 7.7 but it has a huge difference.

It is possible to configure different days to live not only per process definition but also per “data type”.

So we can configure to store historic variables only for 30 days so that our support can use them, but for example the historic activities for one year so that we can analyze them and use them for optimization of process model.

I think this flexibility is very important and I would also desire it in Camunda implementation of history cleanup.

Any remarks?

Best regards,

Markus

1 Like

Hi Markus,
this sounds interesting. Do you store this configuration for different “data types” in database? Can they be dynamically configured?

Can you may be describe in more details, how you then combine this time to live values, the one coming from process definition and the other for “data type”? What is of higher priority?

Svetlana

We abandoned Camunda history as too inefficient and use a custom history service sitting on a separate server and database. It’s running on a Talend ESB server. We can make it do whatever we want, though I just use a MySQL event to clear records after a predefined period. It takes all the load off the Camunda back end database. Our Camunda database server went from a constantly hitting I/O constraints to being asleep most of the time.

Hi @sdorokhova

We have simple spring boot properties which configure the different data types.
At the moment we distinguish between

  1. Historic Tasks
  2. Historic Variables
  3. Historic Activities

We do not use “full” as history level but “Audit”.

At the moment the properties apply to every process definition so we do not configure it on a per process level but this would also be easy to implement.

The intention is again that for example historic variables in our case are only important for a few weeks if some support on process instance is neccessary but we do not need them on a strategic level.
Historic Activities on the other side are important to be stored for a long time to analyse process paths and so on.

Best regards,

Markus

1 Like