Why Tomcat need to restart every day?

Hello all,

As I have developed a small application and deployed it on Server (Tomcat Distribution 7.4.0).

Problem faced: Yesterday I have deployed the application and its running fine.

  1. When I check in application its shows Failure: Loading the filters finished with failures. Try to refresh the page to try again and Failure: Loading the list of tasks finished with failures. Try to refresh the page to try again. Even after refresh, same error message will pop up.

  2. Logs as follows [1].

  3. Camunda engine-rest are working fine.

When I restart Tomcat, it works fine.

Why should I need to restart Tomcat?

Kindly help me.

Thanks & Regards
Harshal Temkar

[1]
22-Jul-2016 05:53:40.769 SEVERE [http-nio-8080-exec-5] org.camunda.commons.logging.BaseLogger.logError ENGINE-16004 Exception while closing command context:

Error querying database. Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 7 ms ago.

The error may exist in org/camunda/bpm/engine/impl/mapping/entity/Filter.xml

The error may involve org.camunda.bpm.engine.impl.persistence.entity.FilterEntity.selectFilterByQueryCriteria

The error occurred while executing a query

Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 7 ms ago.
org.apache.ibatis.exceptions.PersistenceException:

Error querying database. Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 7 ms ago.

The error may exist in org/camunda/bpm/engine/impl/mapping/entity/Filter.xml

The error may involve org.camunda.bpm.engine.impl.persistence.entity.FilterEntity.selectFilterByQueryCriteria

The error occurred while executing a query

Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

Last packet sent to the server was 7 ms ago.
at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:111)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:102)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectList(DbSqlSession.java:87)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectListWithRawParameter(DbEntityManager.java:170)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectList(DbEntityManager.java:162)
at org.camunda.bpm.engine.impl.persistence.entity.FilterManager.findFiltersByQueryCriteria(FilterManager.java:85)
at org.camunda.bpm.engine.impl.filter.FilterQueryImpl.executeList(FilterQueryImpl.java:92)
at org.camunda.bpm.engine.impl.AbstractQuery.evaluateExpressionsAndExecuteList(AbstractQuery.java:186)
at org.camunda.bpm.engine.impl.AbstractQuery.execute(AbstractQuery.java:165)
at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:95)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
at org.camunda.bpm.engine.impl.AbstractQuery.list(AbstractQuery.java:139)
at org.camunda.bpm.engine.rest.impl.FilterRestServiceImpl.executeFilterQuery(FilterRestServiceImpl.java:82)
at org.camunda.bpm.engine.rest.impl.FilterRestServiceImpl.getFilters(FilterRestServiceImpl.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167)
at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257)
at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222)
at org.jboss.resteasy.core.ResourceLocator.invokeOnTargetObject(ResourceLocator.java:159)
at org.jboss.resteasy.core.ResourceLocator.invoke(ResourceLocator.java:92)
at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilterSecure(SecurityFilter.java:67)
at org.camunda.bpm.webapp.impl.security.filter.SecurityFilter.doFilter(SecurityFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:59)
at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter$1.execute(AuthenticationFilter.java:56)
at org.camunda.bpm.webapp.impl.security.SecurityActions.runWithAuthentications(SecurityActions.java:38)
at org.camunda.bpm.webapp.impl.security.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:56)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.filters.CorsFilter.handleNonCORS(CorsFilter.java:436)
at org.apache.catalina.filters.CorsFilter.doFilter(CorsFilter.java:177)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

I would suggest your MySQL connections in the tomcat jdbc connection pool have timed out. Checkout the tomcat docs on connection pool and in particular an sql query to test the connection before its checked out of the pool.

Regards
Rob

1 Like

Hi @Harshal_Temkar.

I agree with @Webcyberrob. Make sure that your connection pool serves valid connections to your application.
See especially the ‘testOnBorrow’ and ‘removeAbandoned’ attributes for your connection pool configuration.

Cheers,
Christian

3 Likes

Hello Rob and Christian

Thanks for your suggestions.

I have written a small BPMN program based on timer which hits the Mysql database for every 6 hrs.
But the results are same with errors.

Thanks & Regards
Harshal Temkar

Hi,

can you share your tomcat connection pool configuration.

Cheers,
Sebastian

Hello Sebastian,

Here is the configuration.

Resource name=“jdbc/ProcessEngine”
auth=“Container”
type=“javax.sql.DataSource”
factory=“org.apache.tomcat.jdbc.pool.DataSourceFactory”
uniqueResourceName=“process-engine”
driverClassName=“com.mysql.jdbc.Driver”
url=“jdbc:mysql://XXXXX:3306/harshal_database”
username=“XXXXX”
password=“XXXXX”
maxActive=“20”
minIdle=“5” />

Connector port=“8080” protocol=“HTTP/1.1”
connectionTimeout=“20000”
redirectPort=“8443” />

Thanks & Regards
Harshal Temkar

Hi Harshal,

did you follow the hints from Rob and Christian? Your configuration still misses testOnBorrow and validationQuery. These are the minimum to validate a connection before it is returned from the connection pool. Without them the connection pool will return already closed connections, which lead to the problems you describe.

Please read some articles how to configure the tomcat connection pool for production environments and add the parameters which are useful in your setup. You can start with the parameters described in this blog post.

Cheers,
Sebastian

2 Likes

Hello Sebastian,

Thanks for your reply.

Best Regards
Harshal Temkar

Hello Harshal,

I’m facing the same issue, can you share with as your configuration and how you have solved your issue.

Thank you in advance

Hi @nouha,

could you share your database configuration? Do you set the configuration options testOnBorrow and validationQuery as described in this comment?

Cheers,
Sebastian

2 Likes

Hi menski, the problem is fix with this configuration:

Thank’s

Hi I am also facing same issue
<!-Resource name=“jdbc/ProcessEngine”
auth=“Container”
type=“javax.sql.DataSource”
factory=“org.apache.tomcat.jdbc.pool.DataSourceFactory”
uniqueResourceName=“process-engine”
driverClassName=“com.mysql.jdbc.Driver”
url=“jdbc:mysql://localhost:3306/camunda77?autoReconnect=true”
username=“root”
password=“root”
maxActive=“20”
minIdle=“5” />

The error occurred while executing a query

Cause: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 57,954,969 milliseconds ago. The last packet sent successfully to the server was 57,954,969 milliseconds ago. is longer than the server configured value of ‘wait_timeout’. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property ‘autoReconnect=true’ to avoid this problem.

at org.apache.ibatis.exceptions.ExceptionFactory.wrapException(ExceptionFactory.java:26)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:111)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:102)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectOne(DefaultSqlSession.java:66)
at org.camunda.bpm.engine.impl.db.sql.DbSqlSession.selectById(DbSqlSession.java:101)
at org.camunda.bpm.engine.impl.db.entitymanager.DbEntityManager.selectById(DbEntityManager.java:197)
at org.camunda.bpm.engine.impl.persistence.entity.ProcessDefinitionManager.findLatestProcessDefinitionById(ProcessDefinitionManager.java:97)
at org.camunda.bpm.engine.impl.persistence.entity.ProcessDefinitionManager.findLatestDefinitionById(ProcessDefinitionManager.java:353)
at org.camunda.bpm.engine.impl.persistence.entity.ProcessDefinitionManager.findLatestDefinitionById(ProcessDefinitionManager.java:44)
at org.camunda.bpm.engine.impl.persistence.deploy.cache.ResourceDefinitionCache.findDeployedDefinitionById(ResourceDefinitionCache.java:47)
at org.camunda.bpm.engine.impl.persistence.deploy.cache.DeploymentCache.findDeployedProcessDefinitionById(DeploymentCache.java:73)
at org.camunda.bpm.engine.impl.cmd.GetDeploymentProcessModelCmd.execute(GetDeploymentProcessModelCmd.java:47)
at org.camunda.bpm.engine.impl.cmd.GetDeploymentProcessModelCmd.execute(GetDeploymentProcessModelCmd.java:34)
at org.camunda.bpm.engine.impl.interceptor.CommandExecutorImpl.execute(CommandExecutorImpl.java:24)
at org.camunda.bpm.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:104)
at org.camunda.bpm.engine.impl.interceptor.ProcessApplicationContextInterceptor.execute(ProcessApplicationContextInterceptor.java:66)
at org.camunda.bpm.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:30)
at org.camunda.bpm.engine.impl.RepositoryServiceImpl.getProcessModel(RepositoryServiceImpl.java:257)
at org.camunda.bpm.engine.rest.sub.repository.impl.ProcessDefinitionResourceImpl.getProcessDefinitionBpmn20Xml(ProcessDefinitionResourceImpl.java:246)
at sun.reflect.GeneratedMethodAccessor728.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:137)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:296)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:250)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:140)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:109)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invokeOnTargetObject(ResourceLocatorInvoker.java:135)
at org.jboss.resteasy.core.ResourceLocatorInvoker.invoke(ResourceLocatorInvoker.java:103)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:356)
at org.jboss.resteasy.core.SynchronousDispatcher.invokePropagateNotFound(SynchronousDispatcher.java:217)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:224)
at org.jboss.resteasy.plugins.server.servlet.FilterDispatcher.doFilter(FilterDispatcher.java:62)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.camunda.bpm.engine.rest.filter.CacheControlFilter.doFilter(CacheControlFilter.java:41)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

Caused by: org.apache.ibatis.transaction.TransactionException: Error configuring AutoCommit. Your driver may not support getAutoCommit() or setAutoCommit(). Requested setting: false. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 57,954,969 milliseconds ago. The last packet sent successfully to the server was 57,954,969 milliseconds ago. is longer than the server configured value of ‘wait_timeout’. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property ‘autoReconnect=true’ to avoid this problem.
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.setDesiredAutoCommit(JdbcTransaction.java:105)
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.openConnection(JdbcTransaction.java:138)
at org.apache.ibatis.transaction.jdbc.JdbcTransaction.getConnection(JdbcTransaction.java:61)
at org.apache.ibatis.executor.BaseExecutor.getConnection(BaseExecutor.java:279)
at org.apache.ibatis.executor.SimpleExecutor.prepareStatement(SimpleExecutor.java:72)
at org.apache.ibatis.executor.SimpleExecutor.doQuery(SimpleExecutor.java:59)
at org.apache.ibatis.executor.BaseExecutor.queryFromDatabase(BaseExecutor.java:267)
at org.apache.ibatis.executor.BaseExecutor.query(BaseExecutor.java:137)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:96)
at org.apache.ibatis.executor.CachingExecutor.query(CachingExecutor.java:77)
at org.apache.ibatis.session.defaults.DefaultSqlSession.selectList(DefaultSqlSession.java:108)
… 52 more
Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 57,954,969 milliseconds ago. The last packet sent successfully to the server was 57,954,969

We had several issues with MySQL connection. Our final stable configuration for the resource is given bellow. As already mentioned above we had added testOnBorrow, validationQuery and validationInterval. But the other issue has appeared which has been fixed with jdbcInterceptors.

	<Resource name="jdbc/ProcessEngine"
          auth="Container"
          type="javax.sql.DataSource"
          factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
          uniqueResourceName="process-engine"
          driverClassName="com.mysql.jdbc.Driver"
          url="jdbc:mysql://XXXX:3306/camunda?autoReconnect=true"
          username="XXXX"
          password="XXXX"
          maxActive="20"
          minIdle="10" 
		  testWhileIdle="true"
		  testOnBorrow="true"
		  testOnReturn="false"
		  validationQuery="SELECT 1"
		  validationInterval="30000"
		  timeBetweenEvictionRunsMillis="30000"
		  maxWait="10000"
		  removeAbandonedTimeout="60"
		  removeAbandoned="true"
		  logAbandoned="true"
		  minEvictableIdleTimeMillis="30000"
		  connectionTimeout="300000"
		  jdbcInterceptors="org.apache.tomcat.jdbc.pool.interceptor.ConnectionState;
							org.apache.tomcat.jdbc.pool.interceptor.StatementFinalizer;
							org.apache.tomcat.jdbc.pool.interceptor.ResetAbandonedTimer"
		  />
1 Like