Read time-out

yadav1990 · July 13, 2023, 6:20am

Hi,
I have zeebe client self-managed with SpringBoot application and I have configured the worker thread in spring properties file “zeebe.client.worker.threads=3” and I configured all the service-task using “connector” so, sometime my connector service task are getting hanged and not getting processes and after sometimes giving read time-out error as like below:

java.lang.RuntimeException: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
	at io.camunda.zeebe.spring.client.jobhandling.SpringConnectorJobHandler.failJob(SpringConnectorJobHandler.java:37)
	at io.camunda.connector.runtime.util.outbound.ConnectorJobHandler.handle(ConnectorJobHandler.java:97)
	at io.camunda.zeebe.spring.client.jobhandling.SpringConnectorJobHandler.handle(SpringConnectorJobHandler.java:30)
	at io.camunda.zeebe.spring.client.jobhandling.JobHandlerInvokingSpringBeans.lambda$handle$0(JobHandlerInvokingSpringBeans.java:77)
	at io.micrometer.core.instrument.AbstractTimer.record(AbstractTimer.java:171)
	at io.camunda.zeebe.spring.client.actuator.MicrometerMetricsRecorder.executeWithTimer(MicrometerMetricsRecorder.java:42)
	at io.camunda.zeebe.spring.client.jobhandling.JobHandlerInvokingSpringBeans.handle(JobHandlerInvokingSpringBeans.java:76)
	at io.camunda.zeebe.client.impl.worker.JobRunnableFactory.executeJob(JobRunnableFactory.java:44)
	at io.camunda.zeebe.client.impl.worker.JobRunnableFactory.lambda$create$0(JobRunnableFactory.java:39)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: java.net.SocketTimeoutException: Read timed out
	at io.camunda.zeebe.spring.client.jobhandling.SpringConnectorJobHandler.failJob(SpringConnectorJobHandler.java:37)
	at io.camunda.connector.runtime.util.outbound.ConnectorJobHandler.lambda$handle$1(ConnectorJobHandler.java:92)
	at java.base/java.util.Optional.ifPresentOrElse(Optional.java:198)
	at io.camunda.connector.runtime.util.outbound.ConnectorJobHandler.handle(ConnectorJobHandler.java:80)
	... 13 more
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:283)
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:309)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:350)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:803)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:976)
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:484)
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
	at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
	at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1465)
	at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1069)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at com.google.api.client.http.apache.v2.ApacheHttpRequest.execute(ApacheHttpRequest.java:73)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012)
	at io.camunda.connector.common.services.HTTPService.executeHttpRequest(HTTPService.java:72)
	at io.camunda.connector.http.HttpService.executeRequestDirectly(HttpService.java:85)
	at io.camunda.connector.http.HttpService.executeConnectorRequest(HttpService.java:55)
	at io.camunda.connector.http.HttpJsonFunction.execute(HttpJsonFunction.java:74)
	at io.camunda.connector.runtime.util.outbound.ConnectorJobHandler.handle(ConnectorJobHandler.java:68)
	... 13 more

TIA.

nathan.loding · July 13, 2023, 7:36pm

Hi @yadav1990 - I want to make sure I understand your configuration. You have a Spring Boot application with the spring-zeebe library, and that application has one (or more) job workers. Correct so far? If so, what type are the job workers listening for? (Example: @JobWorker(type = "foo") is listening for “foo”)

Then, in your model, you configured the service tasks with a connector template? What connector did you configure for those tasks?

yadav1990 · July 14, 2023, 5:16am

Hi @nathan.loding thanks for understanding the issue.
I am using inbuild rest connector with below details:

"jobType": "io.camunda:http-json:1",
"jobRetries": 3,
"jobWorker": "HTTPJSON",

please let me know if need more details to understand the problem statement.

TIA

nathan.loding · July 14, 2023, 2:27pm

Are you running this in a self-managed instance or SaaS? If you’re running it in self-managed instance, do you have the Connector Runtime running, with the REST Connector service configured?

yadav1990 · July 17, 2023, 5:44am

@nathan.loding Yes, I am running in self-managed instance, and I have not configured only one service task. I configured multiple service task with the help of connectors and some of them passed and some of them giving time-out issue.

TIA.

nathan.loding · July 17, 2023, 2:25pm

The logs indicate that your Connector is successfully being invoked and that the endpoint it is trying to call is timing out. Have you checked for errors in the API you are calling? Is there any commonality between endpoints that work and endpoints that always time out?

yadav1990 · July 21, 2023, 10:51am

@nathan.loding yes connector invoked but the API haven’t executed yet all and the service task sill showing one running instance.

nathan.loding · July 21, 2023, 2:17pm

@yadav1990 - the logs seem to indicate that the Connector is attempting to call the API but the request is timing out. In other words, to me it looks like a network issue (firewall between Connector and API, perhaps?) or an issue with the API, not with the Connector or the Runtime.

Are you able to make the same API call with Postman (or another API tool)?