We are using Spring Boot Starter version 1.5.10 and Camunda Spring Boot Starter version 2.3.0.
After running our application for a few days, we are experiencing high heap memory usage and eventually our application will hit the out of memory exception.
Looking at the heap dump, we can see that:
Camunda takes up a lot of heap memory and keeps growing. Is this a known issue?
Hi @firedragon852,
There is a known issue before Spring boot 2.0.2 but it seems that this is something else.
Could you collaborate further to share steps to reproduce?
Best regards,
Yana
Hi @Yana
We are able to consistently reproduce this memory leak issue by re-deploying our spring boot/web application in tomcat. Then we can see that the camunda/ibatis classes will eat up the heap memory.
We are not sure if it is because of our tomcat not being able to clean up the jdbc threads. We keep seeing this warning in catalina.log:
11-Jun-2018 18:02:01.419 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [api] appears to have started a thread named [Abandoned connection cleanup thread] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:64)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)We are getting NPE when trying to perform the Spring application context close:
@SpringBootApplication
@EnableProcessApplication
public class TestApplication {
private static ConfigurableApplicationContext m_context = null;
public static void main(String[] args) {
m_context = SpringApplication.run(TestApplication.class, args);
}
@EventListener
public void onPreUndeploy(PreUndeployEvent event) {
System.out.println("context close start");
try {
m_context.close();
}
catch (Exception e) {
e.printStackTrace();
}
System.out.println("context close end");
}
}
java.lang.NullPointerException
at org.abc.TestApplication.onPreUndeploy(TestApplication.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:256)
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:177)
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:140)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:360)
at org.camunda.bpm.spring.boot.starter.SpringBootProcessApplication.onPreUndeploy(SpringBootProcessApplication.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.camunda.bpm.container.impl.deployment.PreUndeployInvocationStep.performOperationStep(PreUndeployInvocationStep.java:63)
This may explain why camunda resources are not getting removed.
Hi @firedragon852,
It seems that you are missing the @Autowired annotation.
Have a look at SimpleApplication.
Best regards,
Yana
Hi @Yana
I autowired the ConfigurableApplicationContext and did not get the NPE, however after undeploying the application in tomcat, I could still see references to the camunda classes:
Have you try to close it as it is shown in the example?
Yes, I did try to close the context and there’s no NPE after I autowired the ConfigurableApplicationContext.
Hi @firedragon852,
Could you please try to switch to Camunda spring boot 3.0.0 and Spring boot 2.0.2 and share if the problem still persist.
Best regards,
Yana
Hello, we are using Spring-Boot 2.1.2 with Camunda Starter 3.2.1 and have the same problem in production. Our microservice goes out of memory usually once a day. After analyzing the heap dump, we figured out, that it might be because of the MyBatis dependency (like on the screenshot of firedragon852), which is included by Camunda. Are there any known solutions how to prevent the Memory Leak?
Kind regards,
Andrej
Hello!
Same issue with Spring Boot 2.3.0 with Camunda Starter 3.4.0. Any updates on the issue?
Hello,
camunda guys we are facing memory leak issue in our micro service as we observed process definition caching storing lot’s of data. which causing our service getting OOM (out of memory) .
can sameone help on this.
spring-boot-2.3.12.RELEASE.jar
camunda-bpm-spring-boot-starter-3.3.1.jar
Camunda version 7.16.0
MYSQL 5.8
Kubernetes cluster environment
regards
Sahu
Hi!
Do You have any updates? Facing similar issue.
