Can not Display or Start Processes in Cockpit

Stephen_Bucholtz · November 28, 2016, 5:32pm

We are currently unable to display running process instances in the Cockpit. Also, we are unable to start new process instances in the Cockpit.

We are running Camunda 7.5.0, with Apache Tomcat, using the Docker distribution, in a stand-alone deployment.

We observe the following behavior:

When showing the main Cockpit Dashboard screen, the process definitions field never populates and always has the in-progress spinner. Eventually, we get a 500 response from the server.
When attempting to show the Cockpit Processes page, we always get "No process definitions deployed’.
When attempting to start a process we always get the message “The process could not be started: Internal Server Error”.
We frequently observe the following errors in the catalina.log: “java.lang.OutOfMemoryError: GC overhead limit exceeded”

Here are Java Heap settings:

root@4048178d3a8c:/camunda/logs# java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|PermSize|ThreadStackSize'
     intx CompilerThreadStackSize                   = 0                                   {pd product}
    uintx ErgoHeapSizeLimit                         = 0                                   {product}
    uintx HeapSizePerGCThread                       = 87241520                            {product}
    uintx InitialHeapSize                          := 65011712                            {product}
    uintx LargePageHeapSizeThreshold                = 134217728                           {product}
    uintx MaxHeapSize                              := 1038090240                          {product}
     intx ThreadStackSize                           = 1024                                {pd product}
     intx VMThreadStackSize                         = 1024                                {pd product}
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

I have attached some screen shots at the end of this post.

Several error log examples are shown below:

22-Nov-2016 21:21:03.853 WARNING [http-nio-8080-exec-4] org.camunda.bpm.engine.rest.exception.ProcessEngineExceptionHandler.toResponse org.camunda.bpm.engine.ProcessEngineException: Process engine persistence exception
	at org.camunda.bpm.engine.impl.interceptor.CommandInvocationContext.rethrow(CommandInvocationContext.java:60)
	at org.camunda.bpm.engine.impl.interceptor.CommandContext.close(CommandContext.java:291)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:113)
	at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
	at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
	at org.camunda.bpm.engine.impl.AbstractQuery.list(AbstractQuery.java:139)
	at org.camunda.bpm.engine.rest.impl.ProcessInstanceRestServiceImpl.queryProcessInstances(ProcessInstanceRestServiceImpl.java:59)
	at org.camunda.bpm.engine.rest.impl.ProcessInstanceRestServiceImpl.getProcessInstances(ProcessInstanceRestServiceImpl.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
	at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
	at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
	at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:159)
	at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:92)
	at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
	at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
	at org.jboss.resteasy.core.SynchronousDispatcher.invokePropagateNotFound(SynchronousDispatcher.java:169)
	at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:212)
	at org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.doFilter(FilterDispatcher.java:59)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ibatis.exceptions.PersistenceException: 
### Error querying database.  Cause: org.apache.ibatis.reflection.ReflectionException: Could not set property 'cachedEntityState' of 'class org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity' with value '0' Cause: java.lang.OutOfMemoryError: GC overhead limit exceeded
### The error may exist in org/camunda/bpm/engine/impl/mapping/entity/Execution.xml
### The error may involve org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity.selectProcessInstanceByQueryCriteria
### The error occurred while handling results
### SQL: select distinct RES.*                                from ACT_RU_EXECUTION RES                 inner join ACT_RE_PROCDEF P on RES.PROC_DEF_ID_ = P.ID_                  WHERE RES.PARENT_ID_ is null           order by RES.ID_ asc      LIMIT ? OFFSET ?
### Cause: org.apache.ibatis.reflection.ReflectionException: Could not set property 'cachedEntityState' of 'class org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity' with value '0' Cause: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:111)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:102)
	at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectList(DbSqlSession.java:87)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectListWithRawParameter(DbEntityManager.java:171)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:163)
	at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:155)
	at org.camunda.bpm.engine.impl.persistence.entity.ExecutionManager.findProcessInstancesByQueryCriteria(ExecutionManager.java:129)
	at org.camunda.bpm.engine.impl.ProcessInstanceQueryImpl.executeList(ProcessInstanceQueryImpl.java:231)
	at org.camunda.bpm.engine.impl.AbstractQuery.evaluateExpressionsAndExecuteList(AbstractQuery.java:186)
	at org.camunda.bpm.engine.impl.AbstractQuery.execute(AbstractQuery.java:165)
	at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
	at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:104)
	... 40 more
Caused by: org.apache.ibatis.reflection.ReflectionException: Could not set property 'cachedEntityState' of 'class org.camunda.bpm.engine.impl.persistence.entity.ExecutionEntity' with value '0' Cause: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at org.apache.ibatis.reflection.wrapper.BeanWrapper.setBeanProperty(BeanWrapper.java:175)
	at org.apache.ibatis.reflection.wrapper.BeanWrapper.set(BeanWrapper.java:57)
	at org.apache.ibatis.reflection.MetaObject.setValue(MetaObject.java:133)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.applyPropertyMappings(DefaultResultSetHandler.java:371)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.getRowValue(DefaultResultSetHandler.java:341)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValuesForSimpleResultMap(DefaultResultSetHandler.java:294)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleRowValues(DefaultResultSetHandler.java:269)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSet(DefaultResultSetHandler.java:239)
	at org.apache.ibatis.executor.resultset.DefaultResultSetHandler.handleResultSets(DefaultResultSetHandler.java:153)
	at org.apache.ibatis.executor.statement.PreparedStatementHandler.query(PreparedStatementHandler.java:60)
	at org.apache.ibatis.executor.statement.RoutingStatementHandler.query(RoutingStatementHandler.java:73)
	at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:60)
	at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267)
	at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:137)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:96)
	at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:77)
	at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:108)
	... 51 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded

22-Nov-2016 21:18:16.587 SEVERE [http-nio-8080-ClientPoller-1] org.apache.tomcat.util.net.NioEndpoint$Poller.run 
 java.lang.OutOfMemoryError: GC overhead limit exceeded

When attempting to get process instances via the REST API (GET engine-rest/process-instance), sometimes it works, sometimes we get the following:

Response from API endpoint in PostMan
<!DOCTYPE html>
<html>
    <head>
        <title>Apache Tomcat/8.0.24 - Error report</title>
        <style type="text/css">H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;} H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;} H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;} BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;} B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;} P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A {color : black;}A.name {color : black;}.line {height: 1px; background-color: #525D76; border: none;}</style>
    </head>
    <body>
        <h1>HTTP Status 500 - java.lang.OutOfMemoryError: GC overhead limit exceeded</h1>
        <div class="line"></div>
        <p>
            <b>type</b> Exception report
        </p>
        <p>
            <b>message</b>
            <u>java.lang.OutOfMemoryError: GC overhead limit exceeded</u>
        </p>
        <p>
            <b>description</b>
            <u>The server encountered an internal error that prevented it from fulfilling this request.</u>
        </p>
        <p>
            <b>exception</b>
        </p>
        <pre>org.jboss.resteasy.spi.UnhandledException: java.lang.OutOfMemoryError: GC overhead limit exceeded
	org.jboss.resteasy.core.SynchronousDispatcher.handleApplicationException(SynchronousDispatcher.java:365)
	org.jboss.resteasy.core.SynchronousDispatcher.handleException(SynchronousDispatcher.java:233)
	org.jboss.resteasy.core.SynchronousDispatcher.handleInvokerException(SynchronousDispatcher.java:209)
	org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:557)
	org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
	org.jboss.resteasy.core.SynchronousDispatcher.invokePropagateNotFound(SynchronousDispatcher.java:169)
	org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:212)
	org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.doFilter(FilterDispatcher.java:59)
	org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
</pre>
        <p>
            <b>root cause</b>
        </p>
        <pre>java.lang.OutOfMemoryError: GC overhead limit exceeded
</pre>
        <p>
            <b>note</b>
            <u>The full stack trace of the root cause is available in the Apache Tomcat/8.0.24 logs.</u>
        </p>
        <hr class="line">
        <h3>Apache Tomcat/8.0.24</h3>
    </body>
</html>
```<img src="/uploads/default/original/1X/04e93225217596afc57d92e15b82475dad2e19f3.png" width="690" height="207"><img src="/uploads/default/original/1X/92b5dc8637de407b64b98f41b73f8cd42d14a687.png" width="690" height="145"><img src="/uploads/default/original/1X/c41d1736d30cb2fec964780cee4907fe55bab406.png" width="690" height="373"><img src="/uploads/default/original/1X/4452ec354c3934fa4a1090ec24009ad6613f9a83.png" width="690" height="205">

JoHeinem · November 29, 2016, 8:23am

Hi @Stephen_Bucholtz,

your problem seems related to a known issue. All the process definitions/instances are stored in a deployment cache in order to reduce the latency of the access of those objects. Unfortunately, there is no upper bound for cache. When the max heap size is reached, you get OutOfMemoryError.

The good new is, you can set a maximum cache capacity from Camunda BPM 7.6.0-alpha5. All the details are explained here. Maybe you could try out the alpha6 version or wait until Wednesday for Camunda 7.6 and check if that solves your problem?

Best,
Johannes

Stephen_Bucholtz · November 29, 2016, 2:50pm

Thank you for your response. I certainly hope that the changes in Camunda 7.6 solve this issue. This has been killing us for a while now.

system · January 30, 2024, 10:43am