Much higher processor load with enabled SpringJobExecutor

Hi,

I have a problem when I activate a job executor in my web-app. The processer load will raise from about 5% to 30 or a higher percentage, even if the web-application is not really used. I guess that my configuration is wrong or will not be read the correct way.

my environment
I use an embedded camunda 7.5.0 engine that connects to a postgreSQL database. The Engine is configured and loaded in my spring application context. I used an old 7.0.0 version before and I did not recognized this behaivor there.

my configuration

<!-- DataSource for the camunda BPM Engine -->
<bean
	class="org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy"
	id="dataSourceCamundaBpm">
	<property name="targetDataSource">
		<bean class="org.springframework.jdbc.datasource.SimpleDriverDataSource">
			<property name="driverClass" value="${jdbc_driver_class_name}" />
			<property name="url" value="${spring_jdbc_bpm_url}" />
			<property name="username" value="${spring_mgc_db_user}" />
			<property name="password" value="${spring_mgc_db_password}" />
		</bean>
	</property>
</bean>

<!--camunda bpm transaction manager -->
<bean
	class="org.springframework.jdbc.datasource.DataSourceTransactionManager"
	id="transactionManagerCamundaBpm">
	<property name="dataSource" ref="dataSourceCamundaBpm" />
</bean>

<!-- camunda bpm job executor configuration -->
<bean
	class="org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor"
	id="cbpmSpringJobExecutor">
	<property name="taskExecutor">
		<bean
			class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
			<property name="maxPoolSize" value="5" />
			<property name="corePoolSize" value="2" />
		</bean>
	</property>
	<property name="maxJobsPerAcquisition" value="5" />
	<property name="waitTimeInMillis" value="120000" />
	<property name="lockTimeInMillis" value="120001" />
</bean>

<!-- camunda failed job executor -->
<bean
	class="org.camunda.bpm.engine.impl.jobexecutor.FoxFailedJobCommandFactory"
	id="foxFailedJobCommandFactory" />

<!-- ################################################################################ -->
<!-- #### CAMUNDA BPM DEFAULT ENGINE ################################################ -->
<!-- ################################################################################ -->
<!--process engine configuration default -->
<bean class="org.camunda.bpm.engine.spring.SpringProcessEngineConfiguration"
	id="processEngineConfiguration">
	<property name="processEngineName" value="default" />
	<property name="dataSource" ref="dataSourceCamundaBpm" />
	<property name="transactionManager" ref="transactionManagerCamundaBpm" />
	<property name="databaseSchemaUpdate" value="false" />
	<property name="jdbcMaxActiveConnections" value="10" />
	<property name="jdbcMaxIdleConnections" value="5" />
	<property name="jdbcMaxCheckoutTime" value="30000" />
	<property name="jdbcMaxWaitTime" value="25000" />
	<property name="jobExecutorActivate" value="true" />
	<property name="jobExecutor" ref="cbpmSpringJobExecutor" />
	<!-- <property name="customPostBPMNParseListeners"> <list> <bean class="org.camunda.bpm.engine.impl.bpmn.parser.FoxFailedJobParseListener" 
		/> </list> </property> -->
	<!-- <property name="failedJobCommandFactory" ref="foxFailedJobCommandFactory" 
		/> -->
	<property name="dmnEngineConfiguration">
		<bean class="org.camunda.bpm.dmn.engine.impl.DefaultDmnEngineConfiguration">
			<property name="defaultInputExpressionExpressionLanguage"
				value="juel" />
		</bean>
	</property>
	<property name="deploymentResources"
		value="#{camundaDeployService.readBpmDeploymentFiles()}" />
	<!-- from 02.11 on use this way of loading processes-->
		is used instead of <property name="deploymentResources" value="classpath*:bpmn/*/*.bpmn"/> -->
</bean>

<!--Spring Bean to get the engine runtime environment classes -->
<bean class="org.camunda.bpm.engine.spring.ProcessEngineFactoryBean"
	id="processEngine">
	<property name="processEngineConfiguration" ref="processEngineConfiguration" />
</bean>

Even if I change SpringExecutor params it seems to have no effect.

my tries to suspend other reasons for the problem

  • I used other java classes in this webapp, too. But the connection pooling there seems to be okay.
  • I denied other database connections by disabling other IP areas to make sure nothing else has an impact on the database / server

Only if I set the configuration parameter of the job executor of the camunda engine to true the processor load will raise so much. If I disabled it and restart the tomcat with the webapp the processor load ist low as expected.

other (maybe needful) information
When I have a look at the postgres database log I get many queries against the ACT_RU_JOB table per second.

2016-11-14 12:18:49 CET DETAIL:  parameters: $1 = '2016-11-14 12:18:49.588', $2 = '2016-11-14 12:18:49.588', $3 = '8', $4 = '0'
2016-11-14 12:18:49 CET LOG:  duration: 0.000 ms  parse S_2: COMMIT
2016-11-14 12:18:49 CET LOG:  duration: 0.000 ms  bind S_2: COMMIT
2016-11-14 12:18:49 CET LOG:  duration: 0.000 ms  execute S_2: COMMIT
2016-11-14 12:18:49 CET LOG:  duration: 0.000 ms  parse S_1: BEGIN
2016-11-14 12:18:49 CET LOG:  duration: 0.000 ms  bind S_1: BEGIN
2016-11-14 12:18:49 CET LOG:  duration: 0.000 ms  execute S_1: BEGIN
2016-11-14 12:18:49 CET LOG:  duration: 1.000 ms  parse <unnamed>: select
	      RES.* 
	    from ACT_RU_JOB RES
	
	    where (RES.RETRIES_ > 0)
	      and (RES.DUEDATE_ is null or RES.DUEDATE_ <= $1)
	      and (RES.LOCK_OWNER_ is null or RES.LOCK_EXP_TIME_ < $2)
	      and RES.SUSPENSION_STATE_ = 1
	
	       
	     
	    LIMIT $3 OFFSET $4
2016-11-14 12:18:49 CET LOG:  duration: 1.000 ms  bind <unnamed>: select
	      RES.* 
	    from ACT_RU_JOB RES
	
	    where (RES.RETRIES_ > 0)
	      and (RES.DUEDATE_ is null or RES.DUEDATE_ <= $1)
	      and (RES.LOCK_OWNER_ is null or RES.LOCK_EXP_TIME_ < $2)
	      and RES.SUSPENSION_STATE_ = 1
	
	       
	     
	    LIMIT $3 OFFSET $4

Has anybody an idea how I could solve my problem? (if possible without making my app a shared engine in the tomcat configuration).

Thanks for help

Hi @cweb,

could you please try enable debug logging and see if you get messages like this Job acquisition thread sleeping for {} millis? I assume org.camunda.bpm.engine.impl.jobexecutor.JobAcquisitionStrategy#getWaitTime is returning 0 or some other wrong value, could you check that please?

Hope that helps,
Askar

I was facing the same problem, the jobExecutor was querying for new jobs constantly, causing a high CPU load, when it really should take into account the waiIncreaseFactor and keep waiting a little longer, every time it cannot find anything to do. There was hardly any documentation about using SpringJobExecutor so I decided to answer to this blog even though it is already 2 years old, to spare others the pain of debugging the Camunda code, which I ended up doing.

The answer to the problem can be found in the class BackoffJobAcquisitionStrategy and the method initializeMaxLevels

if (baseIdleWaitTime > 0 && maxIdleWaitTime > 0 && idleIncreaseFactor > 0 && maxIdleWaitTime >= baseIdleWaitTime) {
      maxIdleLevel = (int) log(idleIncreaseFactor, maxIdleWaitTime / baseIdleWaitTime) + 1;

      maxIdleLevel += 1;
    }
    else {
      maxIdleLevel = 0;
    }

Quick answer to the problem, the maxIdleWaitTime has to be larger or equal to the baseIdleWaitTime, it is otherwise being ignored.

In your example you did not set the parameter maxWait in which case it was using the default value of 60000 ms, but you did set the maxTimeInMillis to 120000, which is higher and therefore ignored.

You can either set the property maxWait to a value of 120000 or higher or change the parameter waitTimeInMillis to a value lower than 60000.
I’ve decided to change waitTimeInMillis to 5000, which is the default.

<bean id="bpmnJobExecutor" class="org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor">
   <property name="taskExecutor">
      <bean class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor">
         <property name="threadNamePrefix" value="bpmn-" />
         <property name="maxPoolSize" value="10" />
         <property name="corePoolSize" value="2" />
      </bean>
   </property>
   <property name="maxJobsPerAcquisition" value="5" />
   <property name="lockTimeInMillis" value="30000" /> <!-- can be omitted, as it is the default value -->
   <property name="waitTimeInMillis" value="5000" /> <!-- can be omitted, as it is the default value -->
   <property name="waitIncreaseFactor" value="2" />
   <property name="maxWait" value="60000" /> <!-- can be omitted, as it is the default value -->
</bean>