Unexpected error occurred in scheduled task when running custom connector on self-managed instance

lukasmeyeruk · December 22, 2023, 2:59pm

Dear all,

i’m trying to run a custom connector (GitHub - Infosys/camunda-connectors: The solution helps users to connect to various Databases & business applications like ERP's from the camunda business) on my local self-managed camunda instance using the docker runtime.

It seems that the connection to zeebe and operate could be established successfully but I’m getting an error scheduling a task.

Does anyone have an idea what could be wrong here?

Here’s the command I’m using to start the connector via docker runtime:

docker run --rm --name=connectors -it -v ./connector-postgresql-0.1.0-SNAPSHOT.jar:/opt/app/connector.jar -e ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS=192.168.88.137:26500 -e ZEEBE_CLIENT_SECURITY_PLAINTEXT=true -e CAMUNDA_OPERATE_CLIENT_URL=http://192.168.88.137:8081 -e CAMUNDA_OPERATE_CLIENT_CLIENT-ID=operate -e CAMUNDA_OPERATE_CLIENT_CLIENT-SECRET=<secret> -e CAMUNDA_OPERATE_CLIENT_KEYCLOAK-REALM=camunda-platform -e CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL=http://192.168.88.137:18080  camunda/connectors-bundle:latest

This is the error I’m getting:

2023-12-22T10:22:41.623Z  INFO 1 --- [           main] z.s.c.c.ZeebeClientProdAutoConfiguration : Creating ZeebeClient using ZeebeClientConfiguration [ZeebeClientConfigurationProperties{broker=Broker{gatewayAddress='192.168.88.137:26500', keepAlive=PT45S}, cloud=io.camunda.zeebe.spring.client.properties.ZeebeClientConfigurationProperties$Cloud@305552d6, worker=Worker{maxJobsActive=32, threads=10, defaultName='zeebe.client.worker.name', defaultType='null', override={}}, message=Message{timeToLive=PT1H}, security=Security{plaintext=true, overrideAuthority='null', certPath='null'}, job=Job{timeout=PT5M, pollInterval=PT0.1S}, interceptors=[], requestTimeout=PT10S, scheduledExecutorService=java.util.concurrent.ScheduledThreadPoolExecutor@568f4faa[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], ownsJobWorkerExecutor=true}]
2023-12-22T10:22:41.950Z  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 3 endpoint(s) beneath base path '/actuator'
2023-12-22T10:22:41.987Z  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2023-12-22T10:22:41.998Z  INFO 1 --- [           main] org.camunda.feel.FeelEngine              : Engine created. [value-mapper: CompositeValueMapper(List(org.camunda.feel.impl.JavaValueMapper@178dc733)), function-provider: io.camunda.connector.feel.FeelConnectorFunctionProvider@4641f66c, clock: SystemClock, configuration: {externalFunctionsEnabled: false}]
2023-12-22T10:22:42.003Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='com.infosys.camundaconnectors.db:postgresql:1', name='PostgreSQL', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[databaseConnection, operation, data], enabled=null, methodInfo=null}
2023-12-22T10:22:42.004Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:aws-dynamodb:1', name='AWS DynamoDB', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, configuration, input], enabled=null, methodInfo=null}
2023-12-22T10:22:42.004Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:aws-eventbridge:1', name='AWS EventBridge', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, configuration, input], enabled=null, methodInfo=null}
2023-12-22T10:22:42.004Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:aws-lambda:1', name='AWS Lambda', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, configuration, awsFunction], enabled=null, methodInfo=null}
2023-12-22T10:22:42.005Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:aws-sns:1', name='AWS SNS Outbound', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, configuration, topic], enabled=null, methodInfo=null}
2023-12-22T10:22:42.005Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:aws-sqs:1', name='AWS SQS Inbound', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, configuration, queue], enabled=null, methodInfo=null}
2023-12-22T10:22:42.005Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:aws:1', name='AWS DynamoDB Deprecated', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, configuration, input], enabled=null, methodInfo=null}
2023-12-22T10:22:42.104Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:connector-automationanywhere:1', name='Automation Anywhere Outbound Connector', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, operation, configuration], enabled=null, methodInfo=null}
2023-12-22T10:22:42.112Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:connector-graphql:1', name='GraphQL', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[graphql, authentication], enabled=null, methodInfo=null}
2023-12-22T10:22:42.122Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:connector-kafka:1', name='Kafka Producer', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, topic, message, additionalProperties, headers, avro], enabled=null, methodInfo=null}
2023-12-22T10:22:42.202Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:connector-microsoft-teams:1', name='MS Teams', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, data], enabled=null, methodInfo=null}
2023-12-22T10:22:42.202Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:connector-rabbitmq:1', name='RabbitMQ Producer', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, routing, message], enabled=null, methodInfo=null}
2023-12-22T10:22:42.203Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:google-drive:1', name='Google Docs', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, resource], enabled=null, methodInfo=null}
2023-12-22T10:22:42.205Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:google-sheets:1', name='Google Spreadsheets', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[authentication, operation], enabled=null, methodInfo=null}
2023-12-22T10:22:42.205Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:http-json:1', name='HTTP REST', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[url, method, authentication, headers, queryParameters, connectionTimeoutInSeconds, body], enabled=null, methodInfo=null}
2023-12-22T10:22:42.205Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:sendgrid:1', name='SendGrid', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[apiKey, from, to, template, content], enabled=null, methodInfo=null}
2023-12-22T10:22:42.237Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='io.camunda:slack:1', name='Slack Outbound', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[token, method, data], enabled=null, methodInfo=null}
2023-12-22T10:22:42.261Z  INFO 1 --- [           main] i.c.c.r.app.ConnectorRuntimeApplication  : Started ConnectorRuntimeApplication in 2.43 seconds (process running for 2.682)
2023-12-22T10:22:42.519Z ERROR 1 --- [   scheduling-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task

java.lang.RuntimeException: io.camunda.operate.exception.OperateException: /v1/process-definitions/search : 401
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionSearch.query(ProcessDefinitionSearch.java:66)
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionImporter.scheduleImport(ProcessDefinitionImporter.java:57)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.runAndReset(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: io.camunda.operate.exception.OperateException: /v1/process-definitions/search : 401
	at io.camunda.operate.CamundaOperateClient.createDetailedException(CamundaOperateClient.java:107)
	at io.camunda.operate.CamundaOperateClient.execute(CamundaOperateClient.java:101)
	at io.camunda.operate.CamundaOperateClient.execute(CamundaOperateClient.java:98)
	at io.camunda.operate.CamundaOperateClient.execute(CamundaOperateClient.java:98)
	at io.camunda.operate.CamundaOperateClient.execute(CamundaOperateClient.java:98)
	at io.camunda.operate.CamundaOperateClient.execute(CamundaOperateClient.java:91)
	at io.camunda.operate.CamundaOperateClient.executeQuery(CamundaOperateClient.java:203)
	at io.camunda.operate.CamundaOperateClient.search(CamundaOperateClient.java:177)
	at io.camunda.connector.runtime.inbound.importer.ProcessDefinitionSearch.query(ProcessDefinitionSearch.java:64)
	... 13 common frames omitted

many thanks in advance,

Lukas

nathan.loding · December 22, 2023, 4:20pm

Hi @lukasmeyeruk - because the error is a 401 Unauthorized error, I wonder if the client secret you are providing is invalid or missing permissions?

lukasmeyeruk · December 28, 2023, 8:37am

Dear nathan.loding,

many thanks for your help! I believe I could now connect with the Zeebe Worker without any further errors:

postgresql-connector  | 2023-12-28T08:04:01.389Z  INFO 1 --- [           main] z.s.c.c.ZeebeClientProdAutoConfiguration : Creating ZeebeClient using ZeebeClientConfiguration [ZeebeClientConfigurationProperties{broker=Broker{gatewayAddress='192.168.88.169:26500', keepAlive=PT45S}, cloud=io.camunda.zeebe.spring.client.properties.ZeebeClientConfigurationProperties$Cloud@1db7157f, worker=Worker{maxJobsActive=32, threads=10, defaultName='zeebe.client.worker.name', defaultType='null', override={}}, message=Message{timeToLive=PT1H}, security=Security{plaintext=true, overrideAuthority='null', certPath='null'}, job=Job{timeout=PT5M, pollInterval=PT0.1S}, interceptors=[], requestTimeout=PT10S, scheduledExecutorService=java.util.concurrent.ScheduledThreadPoolExecutor@4f4c88f9[Running, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0], ownsJobWorkerExecutor=true}]
postgresql-connector  | 2023-12-28T08:04:01.675Z  INFO 1 --- [           main] org.camunda.feel.FeelEngine              : Engine created. [value-mapper: CompositeValueMapper(List(org.camunda.feel.impl.JavaValueMapper@4026461d)), function-provider: io.camunda.connector.feel.FeelConnectorFunctionProvider@191c6e13, clock: SystemClock, configuration: {externalFunctionsEnabled: false}]
postgresql-connector  | 2023-12-28T08:04:01.679Z  INFO 1 --- [           main] i.c.z.s.c.jobhandling.JobWorkerManager   : . Starting Zeebe worker: ZeebeWorkerValue{type='com.infosys.camundaconnectors.db:postgresql:1', name='PostgreSQL', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[databaseConnection, operation, data], enabled=null, methodInfo=null}
postgresql-connector  | 2023-12-28T08:04:01.691Z  INFO 1 --- [           main] i.c.c.r.app.ConnectorRuntimeApplication  : Started ConnectorRuntimeApplication in 1.544 seconds (process running for 1.831)

Unfortunately, If I configure the template in the BPMN-Model, deploy and execute the process I can observe the following:

Process runs through without any error
The expected activity (table creation in postgresql db) didn’t happen
No errors or any information in console (connector runtime, camunda operate)

I somehow can’t figure out the problem because I’m not getting any errors. Here are the steps I took to create the postgresql connector:

Install “postgresql-database-connector.json” in element-templates of Camunda Modeler. The connector can be selected and configured in the modeler successfully.
Built the connector-postgresql.jar from the GitHub Repo via mvn clean install. I could build the connector-postgresql-0.1.0-SNAPSHOT.jar in ./target/connector-postgresql-0.1.0-SNAPSHOT.jar.
Created a new docker image using the camunda/connectors image with the connector-postgresql.jar. The Dockerfile looks as follows:

FROM camunda/connectors:latest

COPY connector-postgresql-0.1.0-SNAPSHOT.jar /opt/app/

I built the docker image using the following command:

docker build -t postgresql-connector:1 .

Created a docker-compose.yaml file to run the docker image and provide required ENV-Variables for successful connection:

services:
  postgresql-connector:
    image: postgresql-connector:1
    container_name: postgresql-connector

    environment:
      - ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS=192.168.88.169:26500
      - ZEEBE_CLIENT_SECURITY_PLAINTEXT=true
      - CAMUNDA_OPERATE_CLIENT_URL=http://192.168.88.169:8081
      - CAMUNDA_OPERATE_CLIENT_CLIENT-ID=connectors
      - CAMUNDA_OPERATE_CLIENT_CLIENT-SECRET=<secret>
      - CAMUNDA_OPERATE_CLIENT_KEYCLOAK-REALM=camunda-platform
      - CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL=http://192.168.88.137:18080
      - CAMUNDA_CONNECTOR_POLLING_ENABLED=false
      - CAMUNDA_CONNECTOR_WEBHOOK_ENABLED=false
      - SPRING_MAIN_WEB-APPLICATION-TYPE=none
      - OPERATE_CLIENT_ENABLED=false
    networks:
      - camunda-platform-828_camunda-platform
networks:
  camunda-platform-828_camunda-platform:

Start everything with docker compose up

Does anyone have any idea what I’m missing here? I would like to run the connector seperately from the camunda instance in a separate docker container.

Many thanks in advance and kind regards, Lukas

nathan.loding · January 3, 2024, 9:48pm

Hi @lukasmeyeruk - it looks like you configured the Connector with the runtime correctly. I haven’t used the Infosys Postgres Connector yet; its a community contributed Connector, so the best next step might be to open an issue in the GitHub repository … or comment on this issue, which appears to be the same thing you are experiencing:

github.com/Infosys/camunda-connectors

connector-postgresql not working with Camunda 8.3.1

opened 01:45PM - 03 Nov 23 UTC

j-lindner

Hi everyone, I tried to run the postgres connector with my local Camunda 8.3.…1 setup via docker-compose. To do so, I mounted the built fatjar of the postgres connector into the classpath like this: ``` volumes: - "./connector-postgresql-0.1.0-SNAPSHOT-with-dependencies.jar:/opt/app/connector-postgresql-0.1.0-SNAPSHOT-with-dependencies.jar" - "./connectors-home:/home/connectors" ``` The connectors container also shows the corresponding Worker has been started: ``` connectors | 2023-11-03T08:27:45.779Z INFO 1 --- [ main] i.c.z.s.c.jobhandling.JobWorkerManager : . Starting Zeebe worker: ZeebeWorkerValue{type='com.infosys.camundaconnectors.db:postgresql:1', name='PostgreSQL', timeout=null, maxJobsActive=null, requestTimeout=null, pollInterval=null, autoComplete=true, fetchVariables=[databaseConnection, operation, data], enabled=null, methodInfo=null} ``` When I start a process with such connector, I can also see a corresponding log message: ``` connectors | 2023-11-03T13:36:52.591Z INFO 1 --- [pool-2-thread-5] i.c.c.r.c.outbound.ConnectorJobHandler : Received job 2251799813685562 ``` However, nothing happens after that. I can even misconfigure the Connector badly (e.g. writing "????" into host and port), and still nothing happens (I would expect some kind of Exception and/or Incident).