Hello,
I’m struggling with a task which calls a potentially long running service.
I that service failed, then, after three retries, the process instance would get an incident.
But, in my case, since it does not fail (in rare cases it takes ~10 minutes - depending on parameters) - its lock expires and the executor calls it again.
It takes 10 minutes, too, because the parameters did not change. Its lock expires, and the story repeats itself.
The process instance ends up calling this long running service in an infinite loop, because the job retry counter is not decremented.
What solution would you suggest in this situation?
I know it is possible to change the timeouts and maybe retry counts if using external tasks. Currently I like having everything in same process/transaction so I’d like to avoid adding complexity of external task runtimes. Is it possible to somehow add retry count decrements on lock expirations using configuration or some minimal coding?
I understand you are using java delegates. In this case, you can reconfigure the lock duration used by the internal job executor. You can do this via the property camunda.bpm.job-execution.lock-time-in-millis.
You can read more about it in the following thread:
Thanks, Stephan,
yes, I will probably increase the lock time and also try to optimize that service.
But this still looks more like workaround, not a proper solution.
What If I spot other set of parameters that will make the service run even longer? I think that repeated timeouts should also cause incidents to prevent infinite loops. Does not Camunda allow that?
If the duration of your java delegates is prolonged (i.e., to a new parameter) you’d need to adapt the timeout period again. This can be avoided by using external task workers.
Alternatively, you could add a wrapper around your java delegates that raise an incident under certain conditions: I.e., experiencing a timeout for the same job a certain number of times.
If the duration of your java delegates is prolonged (i.e., to a new parameter) you’d need to adapt the timeout period again.
That sounds really weird - adjusting process configuration depending on service call parameter values.
Alternatively, you could add a wrapper around your java delegates that raise an incident under certain conditions: I.e., experiencing a timeout for the same job a certain number of times.
This makes sense. I just hoped Camunda could be such a wrapper. The requirement “don’t retry this job infinitely cause it’s getting stuck” seemed very workflow-ish to me.