Hi,
Since nobody posted a proposal yet, I’ll write down my thoughts:
For one, a service-worker running out of memory is a technical error. I’d prefer to handle such an error outside the business process, since the business logic should not be convoluted with technical details. You can read more about this here.
You could, instead, use an interrupting timer boundary event, that is triggered after a certain time and continues the flow if the task did not complete.
An alternative would be to wrap your service in another service. If the inner service succeeds, the outer service does nothing but forwarding the result. If the inner service crashes, the outer service can raise a BPMN error, which can be handled in your process. This post might be good as an inspiration.
Does this help?