Camunda APIS performing slowee after upgrading from 7.10 to 7.14

Hi,

We upgraded our hosted camunda instance from 7.10 to 7.14 and ran all the database required update scripts, we are seeing APIs are responding slower in some cases more than 60 seconds in general post upgrade.

Below are configuration in our server.xml and web.xml respectively, could we be missing Some setting changes with the upgrade process
Server.xml

<?xml version='1.0' encoding='utf-8'?>
<!-- Additional properties can be set to avoid a common exception
     regarding connection timeout in the connection pool:
      testOnBorrow="true"
      validationQuery="SELECT 1"
-->

<!--https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html-->
<Resource name="jdbc/ProcessEngine"
          auth="Container"
          type="javax.sql.DataSource"
          factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
          uniqueResourceName="process-engine"
          driverClassName="###db_driver_class_name###"
          url="###db_url###"
          defaultTransactionIsolation="READ_COMMITTED"
          username="###db_username###"
          password="###db_password###"
          maxTotal="###db_maxactive###"
          minIdle="###minIdle###"
          maxIdle="###maxIdle###"
          initialSize="###initialSize###"
          testOnConnect="true"
          testWhileIdle="true"
          testOnBorrow="true"
          testOnReturn="false"
          validationQuery="SELECT 1"
          validationInterval="30000"
          timeBetweenEvictionRunsMillis="30000"
          removeAbandonedTimeout="60"
          removeAbandoned="true"
          logAbandoned="true"
          jdbcInterceptors="ResetAbandonedTimer"
          suspectTimeout="50"
          logValidationErrors="true"
          minEvictableIdleTimeMillis="30000"
          jmxEnabled="true"
/>

<Resource name="global/camunda-bpm-platform/process-engine/ProcessEngineService!org.camunda.bpm.ProcessEngineService" auth="Container"
          type="org.camunda.bpm.ProcessEngineService"
          description="camunda BPM platform Process Engine Service"
          factory="org.camunda.bpm.container.impl.jndi.ProcessEngineServiceObjectFactory" />
          
<Resource name="global/camunda-bpm-platform/process-engine/ProcessApplicationService!org.camunda.bpm.ProcessApplicationService" auth="Container"
          type="org.camunda.bpm.ProcessApplicationService"
          description="camunda BPM platform Process Application Service"
          factory="org.camunda.bpm.container.impl.jndi.ProcessApplicationServiceObjectFactory" />
<!--The connectors can use a shared executor, you can define one or more named thread pools-->

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
    maxThreads="500" minSpareThreads="50"/>



<!-- A "Connector" represents an endpoint by which requests are received
     and responses are returned. Documentation at :
     Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)
     Java AJP  Connector: /docs/config/ajp.html
     APR (HTTP/AJP) Connector: /docs/apr.html
     Define a non-SSL HTTP/1.1 Connector on port 8080
-->

<!-- Define a SSL Coyote HTTP/1.1 Connector on port 443 -->
<Connector
       protocol="org.apache.coyote.http11.Http11NioProtocol"
       port="443" maxThreads="500"
       scheme="https" secure="true" SSLEnabled="true"
       keystoreFile="###keystoreFile###" keystorePass="###keystorePass###"
       clientAuth="false" sslProtocol="TLS"/>

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector protocol="AJP/1.3" port="8009"
       redirectPort="8443" />


<!-- An Engine represents the entry point (within Catalina) that processes
     every request.  The Engine implementation for Tomcat stand alone
     analyzes the HTTP headers included with the request, and passes them
     on to the appropriate Host (virtual host).
     Documentation at /docs/config/engine.html -->

<!-- You should set jvmRoute to support load-balancing via AJP ie :
<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">
-->
<Engine name="Catalina" defaultHost="localhost">

  <!--For clustering, please take a look at documentation at:
      /docs/cluster-howto.html  (simple how to)
      /docs/config/cluster.html (reference documentation) -->
  <!--
  <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>
  -->

  <!-- Use the LockOutRealm to prevent attempts to guess user passwords
       via a brute-force attack -->
  <Realm className="org.apache.catalina.realm.LockOutRealm">
    <!-- This Realm uses the UserDatabase configured in the global JNDI
         resources under the key "UserDatabase".  Any edits
         that are performed against this UserDatabase are immediately
         available for use by the Realm.  -->
    <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
           resourceName="UserDatabase"/>
  </Realm>

  <Host name="localhost"  appBase="webapps"
        unpackWARs="true" autoDeploy="true">

    <!-- SingleSignOn valve, share authentication between web applications
         Documentation at: /docs/config/valve.html -->
    <!--
    <Valve className="org.apache.catalina.authenticator.SingleSignOn" />
    -->

    <!-- Access log processes all example.
         Documentation at: /docs/config/valve.html
         Note: The pattern used is equivalent to using pattern="common" -->
    <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
           prefix="localhost_access_log" suffix=".log"
           pattern="%h %l %u %t &quot;%r&quot; %s %b"  fileDateFormat="yyyy-MM-dd" maxDays='60' />

  </Host>
</Engine>

Web.xml

org.camunda.bpm.engine.rest.impl.web.bootstrap.RestContainerBootstrap org.camunda.bpm.engine.rest.impl.FetchAndLockContextListener EmptyBodyFilter org.camunda.bpm.engine.rest.filter.EmptyBodyFilter true EmptyBodyFilter /* CacheControlFilter org.camunda.bpm.engine.rest.filter.CacheControlFilter true CacheControlFilter /* camunda-auth com.fmr.aps.camunda.auth.ConfigurableProcessEngineAuthenticationFilter true authentication-provider com.fmr.aps.camunda.auth.MultiAuthenticationProvider rest-url-pattern-prefix multi-authentication-providers org.camunda.bpm.engine.rest.security.auth.impl.HttpBasicAuthenticationProvider,com.fmr.aps.camunda.auth.BearerTokenAuthenticationProvider bearer-secret ###bearerSecret### camunda-auth /* Resteasy org.jboss.resteasy.plugins.server.servlet.HttpServlet30Dispatcher javax.ws.rs.Application org.camunda.bpm.engine.rest.impl.application.DefaultApplication true Resteasy /*

This had nothing to do with upgrade, this was happening because we had a custom EnginePlugin to handle process and task lifecycle events which had a kinesis integration which was spawning multiple threads causing CPU to Spike. Fix was made to make kinesis object singleton within our plugin code

1 Like