Camunda Sping boot out of memory

gowtham_m · July 29, 2019, 6:32am

Hi ,

we are using below versions:

Spring-Boot: (v1.5.15.RELEASE)
Camunda BPM: (v7.6.0)
Camunda BPM Spring Boot Starter: (v2.1.2)

Exception in thread “JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]” java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

2019-07-25 15:51:36 [http-nio-8020-exec-10] DEBUG org.springframework.boot.web.filter.OrderedRequestContextFilter - Bound request context to thread: org.apache.catalina.connector.RequestFacade@34b611d5

java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-nio-8020-exec-2”

Exception in thread “http-nio-8020-exec-3” Exception in thread “http-nio-8020-Acceptor-0” java.lang.OutOfMemoryError: Java heap space

Exception in thread “RxComputationScheduler-1” java.lang.IllegalStateException: Fatal Exception thrown on Scheduler.Worker thread.

Caused by: java.lang.OutOfMemoryError: Java heap space

2019-07-25 16:23:15 [http-nio-8020-exec-1] DEBUG org.springframework.boot.web.filter.OrderedRequestContextFilter - Bound request context to thread: org.apache.catalina.connector.RequestFacade@42ecccc9

java.lang.OutOfMemoryError: Java heap space

Niall · July 29, 2019, 8:11am

What exactly do you do before seeing this error?

gowtham_m · July 29, 2019, 8:24am

this is production logs… i see continously below

2019-07-29 06:53:05 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14012 Job acquisition thread woke up

2019-07-29 06:53:05 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14011 Job acquisition thread sleeping for 60000 millis

2019-07-29 06:54:05 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14012 Job acquisition thread woke up

gowtham_m · July 29, 2019, 8:25am

2019-07-25 15:38:32 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14011 Job acquisition thread sleeping for 60000 millis

2019-07-25 15:41:59 [JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]] DEBUG org.camunda.bpm.engine.jobexecutor - ENGINE-14012 Job acquisition thread woke up

Exception in thread “http-nio-8020-ClientPoller-1”

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-nio-8020-ClientPoller-0”

Exception in thread “http-nio-8020-exec-5” Exception in thread “http-nio-8020-exec-2”

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread “http-nio-8020-exec-4”

java.lang.OutOfMemoryError: Java heap space

at java.util.HashMap$KeySet.iterator(HashMap.java:917)

at java.util.HashSet.iterator(HashSet.java:173)

at java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1039)

at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1038)

at org.apache.tomcat.util.net.NioEndpoint$Poller.timeout(NioEndpoint.java:1018)

at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:838)

at java.lang.Thread.run(Thread.java:748)

Exception in thread “JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]” java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

2019-07-25 15:51:36 [http-nio-8020-exec-10] DEBUG org.springframework.boot.web.filter.OrderedRequestContextFilter - Bound request context to thread:

gowtham_m · July 29, 2019, 1:31pm

	SpringProcessEngineConfiguration config = new SpringProcessEngineConfiguration();
	config.setDataSource(dataSource());
	config.setTransactionManager(transactionManager());
	config.setIdGenerator(getIdGenerator());
	config.setDatabaseSchemaUpdate(ProcessEngineConfiguration.DB_SCHEMA_UPDATE_TRUE);
	config.setHistory("full");
	config.setJobExecutorActivate(false);
	config.setMetricsEnabled(false);

	List<ProcessEnginePlugin> processEnginePlugins = new ArrayList<ProcessEnginePlugin>();
	processEnginePlugins.add(getSpinPlugin());
	config.setProcessEnginePlugins(processEnginePlugins);
	return config;

Though i have set jobExecutor as false, still it is working how it works… is that causing out of memory

aravindhrs · July 29, 2019, 2:00pm

It’s not job executor issue. Might be you configured some task need to be executed by job executor, so that task seems to be utilized the memory.

Can you check the server RAM configuration, DB storage and any resources been used and didn’t closed properly?

gowtham_m · July 29, 2019, 2:03pm

Hi, In my process i have service task and user task, few tasks are pending at manual task

aravindhrs · July 29, 2019, 2:05pm

What’s the storage space and RAM size for your servers and database?

How many active process instance running?

gowtham_m · July 29, 2019, 2:08pm

active process as now 30,

remaining info will update

gowtham_m · July 29, 2019, 2:11pm

some thread dumps

Locked ownable synchronizers:
- <0x00000000e696f068> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

“PollingServerListUpdater-0” #67 daemon prio=5 os_prio=0 tid=0x00007f5ccc3a8000 nid=0x4e waiting for monitor entry [0x00007f5ce9bf8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
at java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
at java.util.concurrent.ConcurrentHashMap.(ConcurrentHashMap.java:852)
at com.netflix.loadbalancer.LoadBalancerStats.updateZoneServerMapping(LoadBalancerStats.java:232)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.setServerListForZones(DynamicServerListLoadBalancer.java:180)
at com.netflix.loadbalancer.ZoneAwareLoadBalancer.setServerListForZones(ZoneAwareLoadBalancer.java:91)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.setServersList(DynamicServerListLoadBalancer.java:174)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.updateAllServerList(DynamicServerListLoadBalancer.java:259)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer.updateListOfServers(DynamicServerListLoadBalancer.java:242)
at com.netflix.loadbalancer.DynamicServerListLoadBalancer$1.doUpdate(DynamicServerListLoadBalancer.java:62)
at com.netflix.loadbalancer.PollingServerListUpdater$1.run(PollingServerListUpdater.java:116)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- <0x00000000e6976128> (a java.util.concurrent.ThreadPoolExecutor$Worker)

“DestroyJavaVM” #62 prio=5 os_prio=0 tid=0x00007f5d4c010800 nid=0xc waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“JobExecutor[org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor]” #61 prio=5 os_prio=0 tid=0x00007f5d4db08800 nid=0x4a runnable [0x00007f5ce9cf9000]
java.lang.Thread.State: RUNNABLE

“dd-weak-ref-cleaner” #6 daemon prio=1 os_prio=0 tid=0x00007f5d4c2b9800 nid=0x15 runnable [0x00007f5d384a6000]
java.lang.Thread.State: RUNNABLE
at com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap.getKey(WeakConcurrentMap.java:122)
at com.blogspot.mydailyjava.weaklockfree.WeakConcurrentMap.get(WeakConcurrentMap.java:98)
at datadog.trace.agent.tooling.WeakMapSuppliers$WeakConcurrent$Adapter.get(WeakMapSuppliers.java:166)
at datadog.trace.bootstrap.instrumentation.context.FieldBackedProvider$ContextStore$java$util$concurrent$Callable$datadog$trace$bootstrap$instrumentation$java$concurrent$State.mapGet(FieldBackedProvider.java:834)
at datadog.trace.bootstrap.instrumentation.context.FieldBackedProvider$ContextStore$java$util$concurrent$Callable$datadog$trace$bootstrap$instrumentation$java$concurrent$State.realGet(FieldBackedProvider.java)
at datadog.trace.bootstrap.instrumentation.context.FieldBackedProvider$ContextStore$java$util$concurrent$Callable$datadog$trace$bootstrap$instrumentation$java$concurrent$State.get(FieldBackedProvider.java:778)
at datadog.trace.instrumentation.java.concurrent.AdviceUtils.startTaskScope(AdviceUtils.java:20)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:
- <0x00000000e0f23fe8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

“Signal Dispatcher” #4 daemon prio=9 os_prio=0 tid=0x00007f5d4c1a0000 nid=0x14 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:
- None

“Finalizer” #3 daemon prio=8 os_prio=0 tid=0x00007f5d4c16a800 nid=0x13 in Object.wait() [0x00007f5d38905000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000000e0f26428> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

Locked ownable synchronizers:
- None

“Reference Handler” #2 daemon prio=10 os_prio=0 tid=0x00007f5d4c167800 nid=0x12 in Object.wait() [0x00007f5d38a06000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000e0e1df78> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Locked ownable synchronizers:
- None

“VM Thread” os_prio=0 tid=0x00007f5d4c15d000 nid=0x11 runnable

“GC task thread#0 (ParallelGC)” os_prio=0 tid=0x00007f5d4c033800 nid=0xd runnable

“GC task thread#1 (ParallelGC)” os_prio=0 tid=0x00007f5d4c036000 nid=0xe runnable

“GC task thread#2 (ParallelGC)” os_prio=0 tid=0x00007f5d4c037800 nid=0xf runnable

“GC task thread#3 (ParallelGC)” os_prio=0 tid=0x00007f5d4c039800 nid=0x10 runnable

“VM Periodic Task Thread” os_prio=0 tid=0x00007f5d4c83a800 nid=0x1b waiting on condition

JNI global references: 261

gowtham_m · July 29, 2019, 2:15pm

here u can find bpmn file

gowtham_m · July 31, 2019, 5:53am

Hi Niall, can you help me with this

navptr · October 8, 2019, 8:56am

Hello!
We also have a problem in the docker. After some time (~ 2-3 days), the service killed from memory or cpu. With bpmn and dmn (!).
In version 7.5, a similar error was already corrected (https://app.camunda.com/jira/browse/CAM-5073)… but, apparently, not everywhere.
We have 7.9

Need help

satish8784 · January 25, 2021, 9:09am

Hi @navptr were you able to resolve this issue?

Thanks.

system · January 30, 2024, 11:51am