Failed to activate jobs for worker

krysanov.as89 · October 16, 2023, 12:28pm

Hello community. Camunda 8.2.5 has been deployed to the k8s with camunda-platform helm-chart.

zeebe settings:

Zeebe:
  clusterSize: "3"
  partitionCount: "3"
  replicationFactor: "3"

Zeebe Gateway:
  replicaset: 1

Next, a custom image for connectors was created with Infosys custom connector.

FROM  camunda/connectors-bundle:0.22.1
COPY ./connector-jdbc-0.1.0-SNAPSHOT-with-dependencies.jar /opt/custom/
COPY ./connector-jira-0.1.0-SNAPSHOT-with-dependencies.jar /opt/custom/
COPY ./connector-jdbc-0.1.0-SNAPSHOT-with-dependencies.jar /opt/custom/
COPY ./connector-pop3-0.1.0-SNAPSHOT-with-dependencies.jar /opt/custom/
COPY ./connector-smtp-0.1.0-SNAPSHOT-with-dependencies.jar /opt/custom/

Cluster state:

NAME                                  READY   STATUS    RESTARTS   AGE  
elasticsearch-master-0                1/1     Running   0          4d9h 
elasticsearch-master-1                1/1     Running   0          12d  
kibana-69b7c8f8bc-n6vlm               1/1     Running   0          13d  
network-utils                         1/1     Running   0          4d18h
test-connectors-7dc65d5c99-f48vd      1/1     Running   0          41m  
test-identity-69f5798c58-rd4pp        1/1     Running   0          16h  
test-keycloak-0                       1/1     Running   0          17h  
test-keycloak-1                       1/1     Running   0          17h  
test-operate-76458c4c65-jml7k         1/1     Running   0          12d  
test-optimize-5957d8f77-sx9b6         1/1     Running   0          12d  
test-tasklist-84d5f9d67f-rff78        1/1     Running   0          5d   
test-zeebe-0                          1/1     Running   0          6d23h
test-zeebe-1                          1/1     Running   0          34m
test-zeebe-2                          1/1     Running   0          12d
test-zeebe-gateway-7dc4488c68-vf799   1/1     Running   0          41m

Next, a bpm process was created and launched through the zeebe gateway endpoint. Workers are not assigned to custom connectors. At the same time, it is clear that at startup everything starts successfully. Tell me which way to look?

krysanov.as89 · October 17, 2023, 9:19am

Creating a home directory does not solve the problem of assigning workers and executing the bpm process.

2023-10-17T09:15:34.284Z  WARN 1 --- [ult-executor-38] io.camunda.zeebe.client.job.poller       : Failed to activate jobs for worker SMTP and job type com.infosys.camundaconnectors.email:smtp:1

io.grpc.StatusRuntimeException: CANCELLED
        at io.grpc.Status.asRuntimeException(Status.java:539)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.AccessDeniedException: /home/connectors

krysanov.as89 · October 17, 2023, 9:24am

Tasks are stuck and not running
Screenshot 2023-10-17 142330

{
	"flowNodeId": "Send_mail",
	"flowNodeType": "SERVICE_TASK",
	"startDate": "2023-10-06 13:16:52",
	"endDate": null,
	"calledProcessDefinitionName": null,
	"calledDecisionDefinitionName": null,
	"eventId": "225********05707_225*********13",
	"jobType": "com.infosys.camundaconnectors.email:smtp:1",
	"jobRetries": 12,
	"jobWorker": "SMTP",
	"jobDeadline": null,
	"jobCustomHeaders": {
		"resultVariable": "smtp_response"
	},
	"messageName": null,
	"correlationKey": null,
	"incidentErrorType": null,
	"incidentErrorMessage": null,
	"flowNodeInstanceKey": "22***********13",
	"calledProcessInstanceKey": null,
	"calledDecisionInstanceKey": null
}

krysanov.as89 · October 17, 2023, 9:28am

Zeebe gateway debug log:

      io.camunda.zeebe.gateway.interceptors.impl.IdentityInterceptor - Denying call gateway_protocol.Gateway/Topology as the token could not be fully verified. Error message: the token has expired
2023-10-17 09:24:07.518 [] [grpc-default-executor-13] DEBUG
      io.camunda.zeebe.gateway.interceptors.impl.IdentityInterceptor - Denying call gateway_protocol.Gateway/Topology as no token was provided
2023-10-17 09:24:12.521 [] [grpc-default-executor-17] DEBUG
      io.camunda.zeebe.gateway.interceptors.impl.IdentityInterceptor - Denying call gateway_protocol.Gateway/Topology as no token was provided
2023-10-17 09:24:13.776 [] [grpc-default-executor-17] DEBUG

nathan.loding · November 3, 2023, 9:23pm

His @krysanov.as89 - were you able to resolve this issue yet? Unfortunately I’m not quite able to determine what the problem is from the information provided. It appears that perhaps the Connectors are not registering with Zeebe?

krysanov.as89 · November 4, 2023, 8:45am

Hello. Standard connectors work. But when we assemble a custom image with any infosys connector. We see a successful start of everything, but then the processes are not completed. Why might connectors not register with zeebe? Why can It that perhaps the Connectors are not registering with Zeebe? how to debug all this correctly?

nathan.loding · November 6, 2023, 2:57pm

@krysanov.as89 - Connectors are automatically registered with the Connector Runtime if they are in the classpath. I believe the default classpath for the Connectors Docker image is /opt/app and you are adding the Connectors to /opt/custom … I wonder if you rebuild the container to add the jar files to /opt/app if it will work.

krysanov.as89 · November 6, 2023, 4:25pm

I tried it - the result is similar. After all, if we look, we will see both directories in the process.

connectors@test-connectors-7bfddc8c95-bzzxj:/$ ps aux | grep java
connect+       1  0.2  0.9 9711516 320028 ?      Ssl  Oct26  44:02 java -cp /opt/app/*:/opt/custom/* io.camunda.connector.runtime.app.ConnectorRuntimeApplication
connect+   10907  0.0  0.0   6480  2276 pts/0    S+   16:20   0:00 grep java

nathan.loding · November 6, 2023, 7:19pm

Apologies, I misread your second post with the error message. Are you seeing that same error message for all of the Infosys Connectors?

krysanov.as89 · November 7, 2023, 8:32am

I don’t see any errors related to infosys connectors at all. All I see is that the tasks are not being completed. They hang endlessly.

nathan.loding · November 7, 2023, 3:56pm

Just to clarify - you are not seeing this error any longer?

2023-10-17T09:15:34.284Z  WARN 1 --- [ult-executor-38] io.camunda.zeebe.client.job.poller       : Failed to activate jobs for worker SMTP and job type com.infosys.camundaconnectors.email:smtp:1

io.grpc.StatusRuntimeException: CANCELLED
        at io.grpc.Status.asRuntimeException(Status.java:539)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.nio.file.AccessDeniedException: /home/connectors

krysanov.as89 · November 7, 2023, 4:24pm

Yes, this error went away, I had to create a home directory. As you can see below, the data appeared in it.

FROM camunda/connectors-bundle:0.23.2
USER root
RUN mkdir -p /home/connectors & 
RUN chown connectors:connectors /home/connectors
RUN usermod -d /home/connectors connectors
COPY ./connector-jdbc-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/
COPY ./connector-jira-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/
СOPY ./connector-jdbc-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/
COPY ./connector-pop3-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/
USER connectors


connectors@test-connectors-7bfddc8c95-dbsqz:/$ ls -la /home/connectors/.camunda/credentials
-rw-r--r--. 1 connectors connectors 1605 Nov  7 16:04 /home/connectors/.camunda/credentials

nathan.loding · November 9, 2023, 6:06pm

Hi @krysanov.as89 - one thing I just noticed: in your last reply, that Dockerfile doesn’t contain the SMTP Connector. However, your first Dockerfile in your first post does have it. The screenshot of your process shows it looking for the SMTP Connector. Perhaps this is why it isn’t finding it?

I also talked with some of our engineers about what we can look at next. Their suggestion was to try to only include a single custom Connector as a test. It’s possible one of the Connectors is causing something to fail - and perhaps fail silently - so trying them individually will help identify this. If we can identify the faulty Connector, then I can share that with the engineering team who can dive a bit deeper into what is happening.

krysanov.as89 · November 10, 2023, 8:20am

Hi. Sorry my Dockerfile last with 1 custom connector:

USER root
RUN mkdir -p /home/connectors
RUN chown connectors:connectors /home/connectors
RUN usermod -d /home/connectors connectors

#COPY ./connector-jdbc-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/
#COPY ./connector-jira-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/
#COPY ./connector-jdbc-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/
#COPY ./connector-pop3-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/


USER connectors
COPY ./connector-smtp-0.1.0-SNAPSHOT-with-dependencies.jar /opt/app/

Here is the connector start log:

2023-11-10T06:40:45.541Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='com.infosys.camundaconnectors.email:smtp:1', name='SMTP', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, smtpEmailMailBoxName, smtpEmailToRecipients, smtpEmailCcRecipients, smtpEmailBccRecipients, smtpEmailSubject, smtpEmailContentType, smtpEmailContent, smtpEmailAttachments, smtpEmailImportance, smtpEmailReadReceipt, smtpEmailFollowUp, smtpEmailDirectReplyTo, smtpEmailSensitivity], enabled=null, methodInfo=null}

Here is the process that shows the default rest connector and smtp connector infosys.

{
	"flowNodeId": "Send_mail",
	"flowNodeType": "SERVICE_TASK",
	"startDate": "2023-11-10 12:08:30",
	"endDate": null,
	"calledProcessDefinitionName": null,
	"calledDecisionDefinitionName": null,
	"eventId": "2251***********93129_22517********137",
	"jobType": "com.infosys.camundaconnectors.email:smtp:1",
	"jobRetries": 12,
	"jobWorker": "SMTP",
	"jobDeadline": null,
	"jobCustomHeaders": {
		"resultVariable": "smtp_response"
	},
	"messageName": null,
	"correlationKey": null,
	"incidentErrorType": null,
	"incidentErrorMessage": null,
	"flowNodeInstanceKey": "225****************137",
	"calledProcessInstanceKey": null,
	"calledDecisionInstanceKey": null
}

The number of custom connectors does not affect their operation.

{
  "_index": "zeebe-record_job_8.2.5_2023-11-10",
  "_type": "_doc",
  "_id": "1-2421908",
  "_version": 1,
  "_score": 1,
  "_routing": "1",
  "_source": {
    "partitionId": 1,
    "value": {
      "type": "com.infosys.camundaconnectors.email:smtp:1",
      "errorMessage": "",
      "deadline": 1699603432065,
      "processInstanceKey": 225*************29,
      "retries": 12,
      "retryBackoff": 0,
      "recurringTime": -1,
      "processDefinitionVersion": 2,
      "processDefinitionKey": 225**************26,
      "elementInstanceKey": 225*************137,
      "elementId": "Send_mail",
      "customHeaders": {
        "resultVariable": "smtp_response"
      },
      "bpmnProcessId": "C.1.0",
      "variables": {},
      "errorCode": "",
      "worker": "SMTP"
    },
    "key": 2251************157,
    "timestamp": 1699603432170,
    "valueType": "JOB",
    "brokerVersion": "8.2.5",
    "recordType": "EVENT",
    "sourceRecordPosition": 2421907,
    "intent": "TIMED_OUT",
    "rejectionType": "NULL_VAL",
    "rejectionReason": "",
    "position": 2421908,
    "sequence": 2251799813685279
  }

nathan.loding · November 13, 2023, 3:44pm

Hi @krysanov.as89 - have you tried a different custom Connector? I wonder if something might be broken with the SMTP Connector.

krysanov.as89 · November 13, 2023, 7:06pm

I tried different Infosys connectors. For example: jira and pop3 - did not work either. Can you recommend a proven custom connector for Camunda 8?

nathan.loding · November 16, 2023, 7:39pm

Hi @krysanov.as89 - you can look at all the publicly available Connectors in our new Marketplace (https://marketplace.camunda.com/). I have not personally tested most of them so I do not have a recommendation on one to try.

Do you have a specific need for one of these Connectors? Or are you just trying to test the Connector Runtime and learn how it works?

I would also recommend opening an issue in the Infosys GitHub repository for the Connectors and sharing as many details as you can along with a link to this topic. There is an open issue for the Infosys Postgres Connector that is very similar to what you are encountering.

GotnOGuts · November 16, 2023, 8:21pm

I notice that maxJobsActive is null in the log entries reported.

Could that potentially be causing unhandled NPEs that result in no jobs being activated?
Job worker | Camunda 8 Docs implies that maxJobsActive would be a non-null value, but doesn’t suggest what the default would be if it was given a null-value.

krysanov.as89 · November 22, 2023, 1:29pm

Hello, yes, we need an smtp connector. The task is to check the operation of any custom connector. Maybe all our custom settings don’t work at all and the system is configured incorrectly.

nathan.loding · November 22, 2023, 4:48pm

Have you tested the out of the box Camunda Connectors? Maybe use the default Connector image and try the REST connector to call a simple API? I’m very confused about why these Connectors aren’t starting for you, I would expect to see something more in the error logs. Testing the Camunda Connectors, which I have used myself, will be a good test to see if it’s an issue with the Infosys Connectors or something with the image.