io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED

We are getting this error constantly for workers.

Caused by: io.grpc.StatusRuntimeException: 
DEADLINE_EXCEEDED: deadline exceeded after 9.856781371s.
[closed=[], open=[[buffered_nanos=604158453, 
buffered_nanos=126419121, remote_addr=*******************]]]

And due to this issue the pod is unresponsive and we are getting 503 Service Temporarily Unavailable response from pod.

Error:
back-off 5m0s restarting failed container=******* pod=************

Debug information:
Reason: CrashLoopBackOff

Can anyone explain in detail or steps on how to resolve this issue?

One option would be increase the timeout settings.

zeebe.client.worker.override.foo.timeout=10000

Replace the worker name with appropriate value as per your environment.