OutOfMemoryError when deleting deployment

Camunda 7 Topics Discussion & Questions
1

We have a usecase in which we automatically create around 20k of processes. Now we have about 60k of processes running and we would like to clean up a bit.

To do that we wanted to use “Delete deployment” REST API with cascade = true. Sadly, when executed it runs for a while and throws OutOfMemoryError. Current instance runs with 4G of memory and we have limited possibility to give it more.

What would you suggest in this case? We were thinking about deleting them one by one with a script.

Error message:

<!doctype html><html lang="en"><head><title>HTTP Status 500 – Internal Server Error</title><style type="text/css">h1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} h2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} h3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} body {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} b {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} p {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;} a {color:black;} a.name {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 500 – Internal Server Error</h1><hr class="line" /><p><b>Type</b> Exception Report</p><p><b>Message</b> java.lang.OutOfMemoryError: GC overhead limit exceeded</p><p><b>Description</b> The server encountered an unexpected condition that prevented it from fulfilling the request.</p><p><b>Exception</b></p><pre>org.jboss.resteasy.spi.UnhandledException: java.lang.OutOfMemoryError: GC overhead limit exceeded
        org.jboss.resteasy.core.ExceptionHandler.handleApplicationException(ExceptionHandler.java:76)
        org.jboss.resteasy.core.ExceptionHandler.handleException(ExceptionHandler.java:212)
        org.jboss.resteasy.core.SynchronousDispatcher.writeException(SynchronousDispatcher.java:166)
        org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:393)
        org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:200)
        org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:220)
        org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
        org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
        org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        org.camunda.bpm.engine.rest.security.auth.ProcessEngineAuthenticationFilter.doFilter(ProcessEngineAuthenticationFilter.java:151)
        org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
        org.camunda.bpm.engine.rest.filter.EmptyBodyFilter.doFilter(EmptyBodyFilter.java:97)
</pre><p><b>Root Cause</b></p><pre>java.lang.OutOfMemoryError: GC overhead limit exceeded
</pre><p><b>Note</b> The full stack trace of the root cause is available in the server logs.</p><hr class="line" /><h3>Apache Tomcat/9.0.5</h3></body></html>
2

Hi @Wojciech_Pitula,

With cascade, the process engine removes all running and historic process instances along with the deployment, and does so in one transaction. This also requires loading all these instances into main memory which likely causes the exception.

To split this into multiple smaller transactions, I recommend using the batch APIs to delete the process instances before you delete the deployment. Batches split up a large operation into smaller chunks that the job executor performs asnychronously. See https://docs.camunda.org/manual/7.9/user-guide/process-engine/batch-operations for the corresponding Java APIs.

Cheers,
Thorben