we have an application running in production on Camunda 7 which uses external tasks pattern, so our process in microservice 1 creates some external tasks which are picked up and processed by subscribers created in another microservice 2.
This all was working fine, but suddenly Camunda/subscribers stopped processing any external tasks. rather external tasks kept waiting at that stage and process was not able to move forward. we couldn’t see anything in the logs, however when we restarted microservice 2 where subscribers are - they again started processing external tasks and everything worked just fine.
However we are still not able to understand what caused that behaviour, may be we are missing some configuration like number or threads etc etc.
Please could you advise what could be wrong and what can we do to ensure this doesn’t happen again.
thanks for sharing this, no we haven’t really configured back off but I don’t really understand why would subscribers stop picking tasks in the lack of a back off strategy.
Also I have looked at the other documentation as well but not really sure how we can use and configure some of these properties to optimize behaviour of our system and to prevent these issues.
also is there any way to force sequencing of subscribers to control the order in which they pick up tasks, as in I want my subscrober1 to complete some work and then only I want my subscriber 2 to start picking task for a particular business key otherwise if subscriber 2 picks work before subscriber 1 then they end up updating application status incorrectly.
I am also not sure that the backoff strategy has anything to do with the workers just stopping, but I thought it might be worth a try.
Regarding sequencing of subscribers, the only thing I can think of that might help is to use priorities. Have a look at this:
That said, I would probably try to model this requirement inside the process instead of relying on a correct order amongst the workers. This can become very difficult to manage.