Failed to activate jobs for connector worker when trying to run connector with Saas credentials

Praveen_Kumar_Reddy · August 2, 2023, 9:09am

I’m Trying to run a test connector which will just print a message in logs that is given inside template
I’m running that container using an image built on top of camunda/connectors
A part of docker compose file looks like this

  environment:
      - ZEEBE_CLIENT_CLOUD_CLUSTER-ID=xxx
      - ZEEBE_CLIENT_CLOUD_CLIENT-ID=xxx
      - ZEEBE_CLIENT_CLOUD_CLIENT-SECRET=xxx
      - ZEEBE_CLIENT_CLOUD_REGION=xxx
      - CAMUNDA_CONNECTOR_POLLING_ENABLED=false
      - CAMUNDA_CONNECTOR_WEBHOOK_ENABLED=false
      - OPERATE_CLIENT_ENABLED=false

With local docker composed platform it is working fine and when I try to run the connector with Saas platform It is throwing me this error

2023-08-02 14:35:43 2023-08-02T09:05:43.537Z  WARN 1 --- [ault-executor-2] io.camunda.zeebe.client.job.poller       : Failed to activate jobs for worker TEST and job type io.camunda.connector:test:1
2023-08-02 14:35:43 
2023-08-02 14:35:43 io.grpc.StatusRuntimeException: INTERNAL: http2 exception
2023-08-02 14:35:43     at io.grpc.Status.asRuntimeException(Status.java:539)
2023-08-02 14:35:43     at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:491)
2023-08-02 14:35:43     at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
2023-08-02 14:35:43     at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
2023-08-02 14:35:43     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
2023-08-02 14:35:43     at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
2023-08-02 14:35:43     at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
2023-08-02 14:35:43     at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
2023-08-02 14:35:43     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
2023-08-02 14:35:43     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
2023-08-02 14:35:43     at java.base/java.lang.Thread.run(Unknown Source)
2023-08-02 14:35:43 Caused by: io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS. Hex dump for first 5 bytes: 485454502f
2023-08-02 14:35:43     at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109)
2023-08-02 14:35:43     at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.verifyFirstFrameIsSettings(Http2ConnectionHandler.java:353)
2023-08-02 14:35:43     at io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:247)
2023-08-02 14:35:43     at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
2023-08-02 14:35:43     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
2023-08-02 14:35:43     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
2023-08-02 14:35:43     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
2023-08-02 14:35:43     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
2023-08-02 14:35:43     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2023-08-02 14:35:43     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
2023-08-02 14:35:43     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2023-08-02 14:35:43     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
2023-08-02 14:35:43     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2023-08-02 14:35:43     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
2023-08-02 14:35:43     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
2023-08-02 14:35:43     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
2023-08-02 14:35:43     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
2023-08-02 14:35:43     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
2023-08-02 14:35:43     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
2023-08-02 14:35:43     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
2023-08-02 14:35:43     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2023-08-02 14:35:43     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2023-08-02 14:35:43     ... 1 common frames omitted

Thanks in Advance
Praveen

WilliamR.Alves · August 2, 2023, 2:17pm

Hello my friend!

This seems strange to me…
It seems to be something related to dependencies, or credentials.

Are the environment variables set correctly?

Another situation I’ve seen occur, you probably copied and pasted the credentials from somewhere, make sure there isn’t any “blank space” in the values entered.

I hope I helped in some way to find the solution!

William Robert Alves

Praveen_Kumar_Reddy · August 2, 2023, 3:16pm

Hello @WilliamR.Alves
Thanks for your response
I have set the credentials correctly, made sure no blankspace is present
with the same credentials and with SpringBoot Starter camunda connectors its working fine

WilliamR.Alves · August 2, 2023, 3:37pm

huuuumm…

The error you mentioned “io.netty.handler.codec.http2.Http2Exception: First received frame was not SETTINGS” indicates that the connection was not even established…

I can’t see another problem that is not related to one of the topics below:

Some installed dependency incompatibility with your runtime environment.
Some possible Docker compose config…
Any cluster config that may be blocking external connections (Is the port that cunda listens to in the firewall?)

William Robert Alves

GotnOGuts · August 2, 2023, 4:49pm

A few questions:

Do you have a Proxy between your workstation and the SaaS End-point?
Do you have “plaintext=true” in your YAML?

In other posts on the forum, the error “First received frame was not SETTINGS” indicates HTTP2 not being handled correctly on the hop between the worker and the gateway.

Praveen_Kumar_Reddy · August 3, 2023, 5:28am

Hi @GotnOGuts
Thanks for your reply
Answers for your questions

No proxy is available In between them
No I’m not using yaml file in my project, I’m not giving “plaintext=true” I have followed this for my configuration

My project folder structure looks like this
I’m using camunda/connectors image version 0.21.4
My sdk core version is 0.11.4

Any dependency incompatibility or anything wrong I’m doing here

GotnOGuts · August 3, 2023, 4:34pm

Does your network have a proxy between where you are developing and the internet?
Double check with your network team… it might be a “Transparent Proxy”

Benjoyo · August 13, 2023, 7:35pm

Hey, did you solve this? I have the exact same problem.

Some observations / details that may be interesting:

works fine with local cluster
error is gone when I leave out this part that disables Operate client and inbound connectors:

CAMUNDA_CONNECTOR_POLLING_ENABLED=false
CAMUNDA_CONNECTOR_WEBHOOK_ENABLED=false
SPRING_MAIN_WEB-APPLICATION-TYPE=none
OPERATE_CLIENT_ENABLED=false

BUT then it still doesn’t work (does not fetch any jobs), just the error message is gone.

works totally fine with connector images up to camunda/connectors:0.10.0, from 0.2X.X on the error appears. Using same env vars, even same connector-core maven dependency (0.22.1). Just the switch to a newer docker image causes the error

Praveen_Kumar_Reddy · August 14, 2023, 5:18am

Hey @Benjoyo,

I didn’t solve this as my free trial for SaaS is completed

I have understood the error you are facing which is different from the issue that I mentioned in this topic

I came across the same issue you are facing now, You are trying to run an outbound connector in your local cluster with env configuration by disabling the operate and inbound because when you define operate env configuration it is throwing you error

I have faced errors when I try to configure by enabling operate

      - ZEEBE_CLIENT_BROKER_GATEWAY-ADDRESS=zeebe:26500
      - ZEEBE_CLIENT_SECURITY_PLAINTEXT=true
      - CAMUNDA_OPERATE_CLIENT_URL=http://operate:8080
      - CAMUNDA_OPERATE_CLIENT_USERNAME=demo
      - CAMUNDA_OPERATE_CLIENT_PASSWORD=demo
      - OPERATE_CLIENT_ENABLED=true
      - CAMUNDA_OPERATE_CLIENT_KEYCLOAK-URL=http://keycloak:18080
      - CAMUNDA_OPERATE_CLIENT_KEYCLOAK-REALM=camunda-platform
      - CAMUNDA_OPERATE_IDENTITY_CLIENTID=operate
      - CAMUNDA_OPERATE_IDENTITY_CLIENTSECRET=xxx

I have tried the above but unable to fetch and execute jobs and throwing the below error

023-08-14 10:43:37 org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'camundaOperateClient' defined in class path resource [io/camunda/zeebe/spring/client/configuration/OperateClientProdAutoConfiguration.class]: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'camundaOperateClient' threw exception with message: Failed to authenticate with Camunda Operate using Keycloak: please configure client ID and client secret values.
2023-08-14 10:43:37     at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:659)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:647)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973)
2023-08-14 10:43:37     at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)
2023-08-14 10:43:37     at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608)
2023-08-14 10:43:37     at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146)
2023-08-14 10:43:37     at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734)
2023-08-14 10:43:37     at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:436)
2023-08-14 10:43:37     at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
2023-08-14 10:43:37     at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
2023-08-14 10:43:37     at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
2023-08-14 10:43:37     at io.camunda.connector.runtime.app.ConnectorRuntimeApplication.main(ConnectorRuntimeApplication.java:26)
2023-08-14 10:43:37 Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [io.camunda.operate.CamundaOperateClient]: Factory method 'camundaOperateClient' threw exception with message: Failed to authenticate with Camunda Operate using Keycloak: please configure client ID and client secret values.
2023-08-14 10:43:37     at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:171)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:655)
2023-08-14 10:43:37     ... 19 common frames omitted
2023-08-14 10:43:37 Caused by: java.lang.IllegalArgumentException: Failed to authenticate with Camunda Operate using Keycloak: please configure client ID and client secret values.
2023-08-14 10:43:37     at io.camunda.zeebe.spring.client.properties.OperateClientConfigurationProperties.getAuthentication(OperateClientConfigurationProperties.java:109)
2023-08-14 10:43:37     at io.camunda.zeebe.spring.client.configuration.OperateClientProdAutoConfiguration.lambda$camundaOperateClient$0(OperateClientProdAutoConfiguration.java:50)
2023-08-14 10:43:37     at io.github.resilience4j.retry.Retry.lambda$decorateSupplier$4(Retry.java:211)
2023-08-14 10:43:37     at io.github.resilience4j.retry.Retry.executeSupplier(Retry.java:361)
2023-08-14 10:43:37     at io.camunda.zeebe.spring.client.configuration.OperateClientProdAutoConfiguration.camundaOperateClient(OperateClientProdAutoConfiguration.java:46)
2023-08-14 10:43:37     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2023-08-14 10:43:37     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
2023-08-14 10:43:37     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
2023-08-14 10:43:37     at java.base/java.lang.reflect.Method.invoke(Unknown Source)
2023-08-14 10:43:37     at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139)
2023-08-14 10:43:37     ... 20 common frames omitted
2023-08-14 10:43:37

It works fine for me by fetching and executing the jobs, when I disabled the operate and inbound with both 0.10.0 and 0.22.1

Is this the same error you are facing, If not please post your error

More about configuration

Benjoyo · August 15, 2023, 9:38am

I got the same error from your original post when running against the SaaS cluster.

To me it seems that the latest connector runtime images are broken or at least something non-obvious changed.

Running a spring boot project with spring-boot-starter-camunda-connectors and my connectors as a dependency works fine, so I now just dockerize that myself, which works well.

Not sure what is different about the official image.

WilliamR.Alves · August 15, 2023, 12:02pm

Hello my friend!

Did you configure an authentication and authorization part of Keycloak?

If you are using some logic that fetches this sensitive data from somewhere else, for example the AWS SDK to fetch data from the parameter store, it may be that something is wrong and you are not being able to fetch these parameters.

Or maybe your class that fetches this data is not configured as a Springboot Bean.

William Robert Alves

Praveen_Kumar_Reddy · August 17, 2023, 6:55am

Hello my friend @WilliamR.Alves

Apologies for my late reply

Yes, I have configured it as part of compose file to run the camunda platform 8

Not sure why it is throwing the error, am I using wrong environment variables for client Id, secret

WilliamR.Alves · August 18, 2023, 12:38am

Hello my brother @Praveen_Kumar_Reddy !

This is definitely a possibility…