Handle load balancing for history

Camunda 7 Topics Discussion & Questions
1

Hi,

I am trying to get the history record so i have used historyService that provided by camunda. We have more than 80K records available in database and keeps increasing, if we try to query the history it’s getting hang and not able to fetch the huge records as we need to show the history in our application within 30 seconds.

Can you please help me on this?

2

Hi @Adithyan,

what about using the pagination provided with the API?

For the REST API it would be firstResult and maxResults: https://docs.camunda.org/manual/7.10/reference/rest/history/activity-instance/get-activity-instance-query/.

Hope this helps, Ingo

3

Hi,

Thanks for replying.
we have functionality called export in the application. when the user clicks the export button we are fetching all the history stored in database where we are facing problem. its working fine with pagination.

4

Hi,

As our typical requirement is, get the historicVariables stored in the database. we are using historyService and createHistoricVariableInstanceQuery() where we pass out defitionKey , date filter and pagination. its working fine with date filter and pagination as we define the limit 20 records per page.
our issue is with export function for fetch all the historic variable , it could be nK records as of now but increase further to lakhs together and so on…
how to fetch such records in shorter timespan?

5

Hi @Adithyan,

then you could consider to use an execution listener on the process instance end event (https://docs.camunda.org/manual/7.10/user-guide/process-engine/delegation-code/#execution-listener) to export the data of the completed process instance into your database instead of exporting all at once (and again and again, when the export button is pushed again)

Another option is to hook into the history event stream and export the variables when they are created. You can try an additional custom history backend (https://docs.camunda.org/manual/7.10/user-guide/process-engine/history/#provide-a-custom-history-backend) or a custom history level: https://docs.camunda.org/manual/7.10/user-guide/process-engine/history/#implement-a-custom-history-level.

Both approaches will push the data in small chunks from the engine into your database istead of fetching the increasing amount at once from the process engine.

Hope this helps, Ingo