Hi Team, we are facing a challenge in Camunda where all the flows are stuck and every time we are just seeing this message in logs. Job acquisition thread sleeping is always 100/99 milis and only after Camunda restart it starts working but after some time it again hangs. Can you please suggest how to resolve this issue. We have deployed Camunda using docker image.
01-Dec-2020 11:23:58.920 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14011 Job acquisition thread sleeping for 100 millis
01-Dec-2020 11:23:59.021 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14012 Job acquisition thread woke up
01-Dec-2020 11:23:59.021 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14022 Acquired 0 jobs for process engine ‘default’: []
01-Dec-2020 11:23:59.021 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14023 Execute jobs for process engine ‘default’: [ae1149f3-33c3-11eb-a0bd-565d562f3e7d]
01-Dec-2020 11:23:59.021 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14023 Execute jobs for process engine ‘default’: [f377b55f-33c3-11eb-a0bd-565d562f3e7d]
01-Dec-2020 11:23:59.021 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14023 Execute jobs for process engine ‘default’: [8a5e7f79-33c3-11eb-a0bd-565d562f3e7d]
01-Dec-2020 11:23:59.021 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14011 Job acquisition thread sleeping for 100 millis
01-Dec-2020 11:23:59.121 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14012 Job acquisition thread woke up
01-Dec-2020 11:23:59.122 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14022 Acquired 0 jobs for process engine ‘default’: []
01-Dec-2020 11:23:59.122 FINE [Thread-6] org.camunda.commons.logging.BaseLogger.logDebug ENGINE-14023 Execute jobs for process engine ‘default’: [ae1149f3-33c3-11eb-a0bd-565d562f3e7]
Hi Thorben,
According to this link only we enabled logs, made certain changes in job executor as shown below , but still jobs are getting stuck. Please confirm if these changes looks ok? After restart it works for some time like 1 hour and then everything gets stuck at first stage. Also we observed that process gets stuck when there are some jobs locked in RU_JOB table and select query will not be called but as per logs it keeps trying locked jobs. If we instantiate any process it will call insert query, but select part is never called.
Yes I digged down in the table and these jobs are uploading some file to Box. But when we restart camunda pod same jobs continue for some time. But stucks after a while. I mean Job Executor goes into infinite loop and it will not query database. (Select query is not run or executed). Is there a way to release threads or do something without restarting camunda by which Job executor can start picking up job
how long should the upload take? Are they timed out and the work (upload) never completed successfully? Or did they just take too long?
As you nailed it down to a single service task, inspect the implementation of this service task and add some logging statement to the code. If the upload fails, you can throw an exeception to create an incident after all retries are done.
You set your lock time to two minutes. After two minutes the engines assumes that the job died and executes it again.
how long should the upload take? Are they timed out and the work (upload) never completed successfully? Or did they just take too long?
— work completed in back end, like uploading part. But in cockpit it is still shown as running instance.
As you nailed it down to a single service task, inspect the implementation of this service task and add some logging statement to the code. If the upload fails, you can throw an exeception to create an incident after all retries are done.
– Yes logging statements we will add and check further. There are exception handlers to check the response code and put some message.
You set your lock time to two minutes. After two minutes the engines assumes that the job died and executes it again.
— How to set lock time, it is at each task level or process level? please suggest.
What is the content of retries_?
– Retries is 3 always once it is stuck.
How error handlers cause the issue? It is just checking the response code and taking respective actions like setting some parameter. One example as below.
All services are written using Node JS Loopback 3 (which internally calls other java code). In Process flow these services are accessed using node js API endpoints through http_connector.
as you use connectors, your influence of the exection of the service task is limited.
On the long term I suppose to move to external service tasks, where you gain more control about the service execution. The external task claint for Java script will help you here: https://github.com/camunda/camunda-external-task-client-js
@Ashwini_P Any solution you found till now? We face the same issue.
Upon deep dive I am getting
java.util.concurrent.RejectedExecutionException: Task org.camunda.bpm.engine.impl.jobexecutor.ExecuteJobsRunnable@38960577 rejected from java.util.concurrent.ThreadPoolExecutor@605380a6[Running, pool size = 10, active threads = 10, queued tasks = 3, completed tasks = 138] at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065) at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833) at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365) at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:360) at org.camunda.bpm.engine.spring.components.jobexecutor.SpringJobExecutor.executeJobs(SpringJobExecutor.java:59) at org.camunda.bpm.engine.impl.jobexecutor.SequentialJobAcquisitionRunnable.executeJobs(SequentialJobAcquisitionRunnable.java:139) at org.camunda.bpm.engine.impl.jobexecutor.SequentialJobAcquisitionRunnable.run(SequentialJobAcquisitionRunnable.java:81) at java.base/java.lang.Thread.run(Thread.java:833)
Hi @guptaashish327 - welcome to the Camunda forums! This topic is 3 years old, and much has changed since then. Can you start a new topic with your issue, and provide as many details as you can? Thanks!