Process Instance Creation and job activation failed in Zeebe

Camunda 8 Topics
1

Hello Team,
any idea about timeout happening during process instance creation in Camunda8.
Zeebe Client Version: 8.5.0
Camunda8 Version: 8.3.0

Code Snippet used to creation process instance:

client.newCreateInstanceCommand().bpmnProcessId(bpmProcessId).latestVersion()
.variables(bpmnProcessVariablesMap)
.send().join(60000, TimeUnit.MILLISECONDS);

Below are the logs :reason: Timed out waiting on client response

Also observing DEADLINE_EXCEEDED error in logs and jobs are not being activated.
Logs :io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: CallOptions deadline exceeded after 3609.999994094s. Name resolution delay 0.000000000 seconds.
Facing this issue in production intermittently.
Could you please advice what is causing issue here.
Any suspect about why these time out we are observing?
Also after restart of Camunda pods and client(Spring boot microservice) pods, then it started working fine.

Thanks
Rahul

2

@rahulmca.rns DEADLINE_EXCEEDED: The deadline expired before the operation could complete. For operations that change the state of the system, this error may be returned even if the operation has completed successfully. For example, a successful response from a server could have been delayed long enough for the deadline to expire.

Deadline expired before server returned status GRPC_STATUS_DEADLINE_EXCEEDED

By default, gRPC does not set a deadline which means it is possible for a client to end up waiting for a response effectively forever. To avoid this you should always explicitly set a realistic deadline in your clients. To determine the appropriate deadline you would ideally start with an educated guess based on what you know about your system (network latency, server processing time, etc.), validated by some load testing.

If a server has gone past the deadline when processing a request, the client will give up and fail the RPC with the DEADLINE_EXCEEDED status

3

Thanks for the reply, so it seems zeebe was having issue while processing requests?

4

@rahulmca.rns It’s not issue with Zeebe. It’s related how you setup the application. You should always explicitly set a realistic deadline in your clients by running some load testing to determine the ideal value.