io.zeebe.client.api.command.ClientStatusException: deadline exceeded after 19999976300ns

prasadkoribilli · September 16, 2019, 3:48pm

Hello Team,

My micro-service going into infinite loop with below error. (Error message down below)

I have set like this:
return zeebe.newWorker()
.jobType(“my.job”)
.handler(ProvisionJob())
.timeout(Duration.ofMinutes(30))
.open();

Seems like 30 minutes setting is getting over written to 19 min from the system configuraiton.

Could you please tell us where we can set a custom value? I want to increase the timeout from 19 to 30 min.

Thanks for the help.

Error:
2019-09-16 15:31:19.839 WARN 1 — [pool-2-thread-1] io.zeebe.client.job.worker : Worker default failed to handle job with key 4503599628584815 of type sp2-provision-saptop, sending fail command to broker

io.zeebe.client.api.command.ClientStatusException: deadline exceeded after 19999976300ns
at io.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:93) ~[zeebe-client-java-0.20.0.jar!/:0.20.0]
at io.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:50) ~[zeebe-client-java-0.20.0.jar!/:0.20.0]
at com.swissre.g69.ietv.provisioninfrastructure.ProvisionFPSL.handle(ProvisionFPSL.java:41) ~[classes!/:na]
at io.zeebe.client.impl.worker.JobRunnableFactory.executeJob(JobRunnableFactory.java:44) [zeebe-client-java-0.20.0.jar!/:0.20.0]
at io.zeebe.client.impl.worker.JobRunnableFactory.lambda$create$0(JobRunnableFactory.java:39) [zeebe-client-java-0.20.0.jar!/:0.20.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_212]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999976300ns
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) ~[na:1.8.0_212]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) ~[na:1.8.0_212]
at io.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:48) ~[zeebe-client-java-0.20.0.jar!/:0.20.0]
… 10 common frames omitted
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 19999976300ns
at io.grpc.Status.asRuntimeException(Status.java:533) ~[grpc-api-1.22.1.jar!/:1.22.1]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:442) ~[grpc-stub-1.22.1.jar!/:1.22.1]
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.22.1.jar!/:1.22.1]
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.22.1.jar!/:1.22.1]
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.22.1.jar!/:1.22.1]
at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:700) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) ~[grpc-api-1.22.1.jar!/:1.22.1]
at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-api-1.22.1.jar!/:1.22.1]
at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-api-1.22.1.jar!/:1.22.1]
at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:399) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:507) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:66) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:627) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:515) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:686) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:675) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.22.1.jar!/:1.22.1]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.22.1.jar!/:1.22.1]
… 3 common frames omitted

Zelldon · September 16, 2019, 11:40pm

Hey @prasadkoribilli

first of all thanks for using Zeebe.

The problem is that you configured the job timeout, which means it is the time how long the job is assigned to that specific worker. But there is also the default long polling configured of 10 seconds and the request timeout of 20 seconds. So it you want to increase that just use .requestTimeout(Duration.ofMinutes(30)).

I hope that helps.

Greets
Chris

prasadkoribilli · September 17, 2019, 7:39am

Hello @Zelldon

Thanks for your reply. I will try that setting out.
But if I read the exception below,

"io.zeebe.client.job.worker : Worker default failed to handle job with key 4503599628584815 of type sp2-provision-saptop, sending fail command to broker
io.zeebe.client.api.command.ClientStatusException: deadline exceeded after 19999976300ns "

worker is not able to send the command to broker. Which is communication from Worker to Broker.
But the parameter that you suggested is more for polling, meaning getting workers from Broker.

Are they not two different things? One is for sending command other is polling a job.

Please enlighten me a bit here.

Thanks.

Zelldon · September 17, 2019, 9:36am

Hey @prasadkoribilli sorry might that I misunderstood your problem. I thought your request times out and with requestTimeout you can configure that (+long polling).

Could you please share your job handler code and the complete log?

Greets
Chris

i.m.superman · November 8, 2019, 8:46am

I got the same exception

BTW
Zeebe 0.21.1
spring-zeebe-starter 0.6.0

When my program invokes the code as below pic, exception occurrs.

i.m.superman · November 26, 2019, 3:12am

@Zelldon It happened again.
Do I need set some configurations?

Zelldon · November 26, 2019, 6:47am

Hey @i.m.superman,

please share your worker/client code and configuration. If possible then also please share the exception plus stack trace. How long will your jobs take? How does your process look like?

Greets
Chris

i.m.superman · November 26, 2019, 6:57am

hi @Zelldon

here is my configuration.

zeebe.worker.name = upgrade-worker
zeebe.client.broker.contactPoint = 192.168.70.168:26500
zeebe.client.worker.poll-interval = 30m
zeebe.processid = dailyUpdate

The job takes 15~20 minutes.

i.m.superman · November 26, 2019, 8:13am

@Zelldon I find this information in log.

2019-11-26 16:04:40.670 WARN 35784 — [pool-8-thread-1] io.zeebe.client.job.worker : Worker ComplieRouteSpec-worker failed to handle job with key 13510798884484916 of type doAffectLinksService, sending fail command to broker.

Is this helpful for analyzing?

i.m.superman · November 26, 2019, 8:34am

Zelldon · November 26, 2019, 1:07pm

CANCELLED: call already cancelled

This exception can be thrown on the gateway if you close the client.

Worker ComplieRouteSpec-worker failed to handle job with key 13510798884484916 of type doAffectLinksService, sending fail command to broker.

It seems you worker fail to handle the job, due to an exception and tries to send an fail command, which also fails.

The job takes 15~20 minutes.

You then need to increase the activation timeout to an higher value.

i.m.superman · November 26, 2019, 1:11pm

Which time out configuration in spring-zeebe-starter should I set ?

i.m.superman · November 26, 2019, 1:13pm

My spring-zeebe-starter version is 0.6.0 and zeebe version is 0.21.1.
Is that conflicting ?

Zelldon · November 26, 2019, 1:15pm

Which time out configuration in spring-zeebe-starter should I set ?

It is probably zeebe.client.job.timeout.

My spring-zeebe-starter version is 0.6.0 and zeebe version is 0.21.1.
Is that conflicting ?

Seems so please always check the release page https://github.com/zeebe-io/spring-zeebe/releases