Camunda Spring Boot - Out of Memory Error

sandesh.dsouza · July 15, 2022, 3:20am

We are using following Spring boot and Camunda version. Following are the details of runtime.

JDK - redhat-openjdk-18/openjdk18-openshift
Containerized in Docker version - 20.10.11, build 761974f
OS - Red Hat Enterprise Linux Server release 7.6 (Maipo)
Total no of BPMN files deployed - 34 - There are mostly service tasks and no user task is used. 4 of the BPMN are batch processes which runs mostly at night 11 PM
JAVA_OPTS="-Xms512m -Xmx2048m -XX:+UseStringDeduplication -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/heapdump"

Earlier we had just 2 nodes with 2 CPU configured in each node. We are been getting OOM since some time. So we updated to have the app run on 3 nodes . In terms of CPU settings, the app can take 8 CPU’s in each node now. In spite of that we are still getting OOM. We can’t simulate this in lower environment.

Can someone please provide some guidance in terms of how to debug this further and what I can do to make this better?

Here is the image of heap dump analysis from Heap Analyzer

H2 Application.properties

#-----Datasource properties--------#
spring.datasource.driver-clas=oracle.jdbc.OracleDriver
spring.jpa.show-sql=false
spring.jpa.hibernate.ddl-auto=none
spring.jpa.open-in-view=false
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.Oracle12cDialect
spring.datasource.validationQuery=SELECT 1 FROM DUAL
spring.jpa.hibernate.use-new-id-generator-mappings=true
spring.jpa.properties.hibernate.default_schema=WORKFLOW

camunda.bpm.generic-properties.properties.failed-job-retry-time-cycle=R0/PT5M
camunda.bpm.history-level=NONE

H2 Pom.xml

	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.7.0</version>
		<relativePath />
	</parent>

	<dependency>
			<groupId>org.camunda.bpm.springboot</groupId>
			<artifactId>camunda-bpm-spring-boot-starter-rest</artifactId>
			<version>3.4.4</version>
		</dependency>

		<dependency>
			<groupId>org.camunda.bpm.springboot</groupId>
			<artifactId>camunda-bpm-spring-boot-starter-webapp</artifactId>
			<version>3.4.4</version>
		</dependency>

		<dependency>
			<groupId>org.camunda.connect</groupId>
			<artifactId>camunda-connect-http-client</artifactId>
			<version>1.3.0</version>
		</dependency>

		<dependency>
			<groupId>org.camunda.bpm</groupId>
			<artifactId>camunda-engine-plugin-spin</artifactId>
			<version>7.12.0</version>
		</dependency>
		<dependency>
			<groupId>org.camunda.spin</groupId>
			<artifactId>camunda-spin-dataformat-json-jackson</artifactId>
			<version>1.10.1</version>
		</dependency>
		<dependency>
			<groupId>com.fasterxml.jackson.datatype</groupId>
			<artifactId>jackson-datatype-jdk8</artifactId>
			<version>2.12.3</version>
		</dependency>

		<dependency>
			<groupId>org.camunda.bpm</groupId>
			<artifactId>camunda-engine-plugin-connect</artifactId>
			<version>7.12.0</version>
		</dependency>

		<!-- JDK 8 Compatible. As soon we move to JDK 11 or higher we need to update this -->
		<dependency>
			<groupId>com.oracle.database.jdbc</groupId>
			<artifactId>ojdbc8</artifactId>
			<version>19.14.0.0</version>
		</dependency>

H2 Log

e[2m2022-07-14 09:37:25.759e[0;39m e[32m INFOe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[nio-8080-exec-3]e[0;39m e[36mo.apache.coyote.http11.Http11Processor e[0;39m e[2m:e[0;39m Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: Invalid character found in the request target [/ ]. The valid characters are defined in RFC 7230 and RFC 3986
…

e[2m2022-07-14 09:41:08.546e[0;39m e[33m WARNe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[l-1 housekeeper]e[0;39m e[36mcom.zaxxer.hikari.pool.HikariPool e[0;39m e[2m:e[0;39m HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=58s933ms383?s398ns).
e[2m2022-07-14 09:42:25.468e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[etrics Reporter]e[0;39m e[36mo.h.engine.jdbc.spi.SqlExceptionHelper e[0;39m e[2m:e[0;39m HikariPool-1 - Connection is not available, request timed out after 30519ms.
e…
e[2m2022-07-14 09:44:48.762e[0;39m e[33m WARNe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[etrics Reporter]e[0;39m e[36morg.camunda.bpm.engine.metrics e[0;39m e[2m:e[0;39m ENGINE-18002 Could not collect and log metrics

…
e[2m2022-07-14 09:44:48.762e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[ingJobExecutor]]e[0;39m e[36morg.camunda.bpm.engine.jobexecutor e[0;39m e[2m:e[0;39m ENGINE-14019 Exception during job acquisition Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection

Exception in thread “http-nio-8080-exec-58” java.lang.OutOfMemoryError: Java heap space
e[2m2022-07-14 11:30:48.735e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-41]e[0;39m e[36mo.a.c.c.C.[.[.[.[dispatcherServlet] e[0;39m e[2m:e[0;39m Servlet.service() for servlet [dispatcherServlet] in context with path [/workflow] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection] with root cause

e[2m2022-07-14 11:42:26.753e[0;39m e[33m WARNe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-62]e[0;39m e[36mo.h.engine.jdbc.spi.SqlExceptionHelper e[0;39m e[2m:e[0;39m SQL Error: 17002, SQLState: 08006
e[2m2022-07-14 11:42:26.753e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-62]e[0;39m e[36mo.h.engine.jdbc.spi.SqlExceptionHelper e[0;39m e[2m:e[0;39m IO Error: Connection reset by peer, Authentication lapse 0 ms.
e[2m2022-07-14 11:42:26.753e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-71]e[0;39m e[36mo.h.engine.jdbc.spi.SqlExceptionHelper e[0;39m e[2m:e[0;39m IO Error: Connection reset by peer, Authentication lapse 0 ms.
e[2m2022-07-14 11:42:28.198e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-76]e[0;39m e[36mo.h.engine.jdbc.spi.SqlExceptionHelper e[0;39m e[2m:e[0;39m IO Error: Connection reset by peer, Authentication lapse 0 ms.
e[2m2022-07-14 11:42:26.017e[0;39m e[33m WARNe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[nio-8080-exec-5]e[0;39m e[36mo.h.engine.jdbc.spi.SqlExceptionHelper e[0;39m e[2m:e[0;39m SQL Error: 0, SQLState: 08006
e[2m2022-07-14 11:50:56.594e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-40]e[0;39m e[36mo.a.c.c.C.[Tomcat].[localhost] e[0;39m e[2m:e[0;39m Exception Processing ErrorPage[errorCode=0, location=/error]

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) ~[tomcat-embed-core-9.0.63.jar!/:na]
at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) ~[na:1.8.0_181]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1187) ~[jackson-core-2.13.3.jar!/:2.13.3]
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1009) ~[jackson-databind-2.13.3.jar!/:2.13.3]
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:456) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:290) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:135) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:711) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:385) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:403) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:249) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:344) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) [tomcat-embed-core-9.0.63.jar!/:na]
e[2m2022-07-14 11:54:53.762e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[nio-8080-exec-1]e[0;39m e[36mo.a.c.c.C.[Tomcat].[localhost] e[0;39m e[2m:e[0;39m Exception Processing ErrorPage[errorCode=0, location=/error]

org.apache.catalina.connector.ClientAbortException: java.io.IOException: Connection reset by peer
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:310) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:273) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:118) ~[tomcat-embed-core-9.0.63.jar!/:na]
at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) ~[na:1.8.0_181]
at com.fasterxml.jackson.core.json.UTF8JsonGenerator.flush(UTF8JsonGenerator.java:1187) ~[jackson-core-2.13.3.jar!/:2.13.3]
at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:1009) ~[jackson-databind-2.13.3.jar!/:2.13.3]
at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:456) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:104) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:290) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.HttpEntityMethodProcessor.handleReturnValue(HttpEntityMethodProcessor.java:219) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.method.support.HandlerMethodReturnValueHandlerComposite.handleReturnValue(HandlerMethodReturnValueHandlerComposite.java:78) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:135) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:711) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:385) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:403) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:249) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:344) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.63.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_181]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_181]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_181]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_181]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_181]
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:135) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1384) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:773) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:726) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:716) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.flush(Http11OutputBuffer.java:573) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.http11.filters.ChunkedOutputFilter.flush(ChunkedOutputFilter.java:157) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.http11.Http11OutputBuffer.flush(Http11OutputBuffer.java:221) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.http11.Http11Processor.flush(Http11Processor.java:1255) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:402) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.Response.action(Response.java:209) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:306) ~[tomcat-embed-core-9.0.63.jar!/:na]
… 53 common frames omitted

e[2m2022-07-14 11:54:53.099e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-62]e[0;39m e[36mo.a.c.c.C.[.[.[.[dispatcherServlet] e[0;39m e[2m:e[0;39m Servlet.service() for servlet [dispatcherServlet] in context with path [/workflow] threw exception [Request processing failed; nested exception is org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection] with root cause

java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.8.0_181]
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.8.0_181]
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.8.0_181]
at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.8.0_181]
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471) ~[na:1.8.0_181]
at oracle.net.nt.TimeoutSocketChannel.write(TimeoutSocketChannel.java:211) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.net.ns.NIOPacket.writeToSocketChannel(NIOPacket.java:355) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.net.ns.NIOConnectPacket.writeToSocketChannel(NIOConnectPacket.java:247) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:127) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.net.ns.NSProtocol.connect(NSProtocol.java:347) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1681) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:625) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:813) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:80) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:816) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:620) ~[ojdbc8-19.14.0.0.jar!/:19.14.0.0.0]
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:726) ~[HikariCP-4.0.3.jar!/:na]
at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712) ~[HikariCP-4.0.3.jar!/:na]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

e[2m2022-07-14 11:54:51.751e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[aTaskExecutor-3]e[0;39m e[36mo.h.engine.jdbc.spi.SqlExceptionHelper e[0;39m e[2m:e[0;39m HikariPool-1 - Connection is not available, request timed out after 32077ms.
e[2m2022-07-14 12:31:24.891e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-71]e[0;39m e[36mo.a.c.c.C.[Tomcat].[localhost] e[0;39m e[2m:e[0;39m Exception Processing ErrorPage[errorCode=0, location=/error]

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: Java heap space
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1082) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:655) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.20.jar!/:5.3.20]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.20.jar!/:5.3.20]
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:711) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:461) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:385) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:313) ~[tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:403) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:249) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:344) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:890) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.63.jar!/:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.63.jar!/:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.lang.OutOfMemoryError: Java heap space

e[2m2022-07-14 12:31:48.117e[0;39m e[31mERRORe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[io-8080-exec-68]e[0;39m e[36mo.a.c.c.C.[Tomcat].[localhost] e[0;39m e[2m:e[0;39m Exception Processing ErrorPage[errorCode=0, location=/error]

sandesh.dsouza · July 15, 2022, 3:28am

Missed out one thing.

These are DB properties which show how many connections are setup for each
-Dspring.datasource.hikari.idle-timeout=15000
-Dspring.datasource.hikari.minimum-idle=5
-Dspring.datasource.hikari.max-lifetime=30000
-Dspring.datasource.hikari.maximum-pool-size=30

mimaom · July 15, 2022, 7:37am

Hi @sandesh.dsouza

A few thoughts.

1:
Is 2048 M of memory actually enough heap? Maybe one (or more) of your processes triggers some memory intensive operations?

2:
I don’t know the IBM HeapAnalyzer, so maybe this is nothing.
It reports a leak suspect of jdk.nashorn - that is the Javascript enigne. Are you using javascripts in any of your process definitions?

3:
It’s hard to read the stacktrace, so it’s unclear to me if this is a single stacktrace… But I do notice this:

HikariPool-1 - Connection is not available, request timed out after 32077ms.

It’s unclear if this timeout happens because of the OOM exception. I would look into this also - do you see anything unusual in the Oracle database logs?

4:
You might have to narrow the problem down to one (or more) process definitions which off course can be tricky. Can you add debug log statements to pinpoint which process def is triggering the error. Fx log statements on entry to every service task code.

BR
Michael

sandesh.dsouza · July 15, 2022, 8:10am

Thanks for the response Michael.

I was wondering what is the recommended values we need to put for -Xms and -Xmx given we have 34 processes and we need to put more processes soon.
Yes, inside the bpmn processes, we use javascript extensively. We were thinking of moving away to groovy scripting or or GraalVM if javascripts are causing this issue. Is there any recommendation on which one to go considering memory / speed of operation mainly?
We have spoken to our DBA’s and they have said DB is alright.I couldn’t attach the entire log as it was exhausing the total size. But, I did see lot of below message which to me made me think that the thread is not getting priority to claim the connection back.

e[2m2022-07-14 06:31:09.201e[0;39m e[33m WARNe[0;39m e[35m1e[0;39m e[2m—e[0;39m e[2m[l-1 housekeeper]e[0;39m e[36mcom.zaxxer.hikari.pool.HikariPool e[0;39m e[2m:e[0;39m HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=47s518ms166?s703ns).

Considering this issue only comes up in Production, it has become bit challenging to narrow it down based on the processes. But, I’ll add the debug statements and try to find out more about this.

mimaom · July 15, 2022, 10:08am

1: It’s difficult to answer that in a general way. It depends very much on what your processes are doing. Fx scripts that work with large XML docs…

2: I know this is easier said that done, but you might start considering a change to your process applications and start looking into using external tasks.

This would allow you to move your task logic into real applications (written in any programming language you prefer) - in stead of just raw script files. I think Camunda is now recommending using the external task pattern when creating process applications. Also if you plan to eventually migrate to Camunda 8 - here script tasks behave just like service tasks, so you will not be able to include a script the same way you do now.

Another advantage is that you move your work outside the Camunda server and you will be able to scale / group your worker applications much better (and it could be easier to identify problems like the OOM) .

I know it’s difficult to do in the middle of a production problem - but is worth thinking about it

3: Hmm the thread starvation could be because of a lack a resources (OOM)

4: Could you try to lower the memory of a staging / test environment to try to make it easier to trigger the error outside the prod environment?

Do you know what is triggering the error? Call to Camunda API (message etc) The engine firing a timer etc ? Do you know if it’s the same condition that is triggering the error?

BR
Michael

system · January 31, 2024, 10:21am