Copying info to a new topic here from a Slack convo for posterity since C8 Slack is going away.
I had set up a stand-alone Connector Runtime to test a process moved entirely by Kafka Connectors. When testing locally, everything worked fine and I could get through a process end to end using ~20 connectors.
Once I pointed my stand-alone Connector Runtime to Camunda Cloud, I saw inconsistent behavior where after ~3 outbound Connectors executed immediately, the next 1 or 2 would take 90 seconds to execute, the rest would time out and never execute.
It turns out that this was due to the jobs being picked up either by my stand-alone Connector Runtime or the provided SaaS / Cloud Connector Runtime. In the cases where the SaaS Connector Runtime picked up the job, it could not connect to the specified Kafka broker so it timed out after 45 seconds. Zeebe would retry the job 3 times. In some cases, my stand-alone Connector Runtime got one of the jobs upon retry (hence the 90 second success for some of the jobs). In other cases, all 3 retries went by and then the job timed out.
At the time of writing, there is no way to disable the provided Connector Runtime in Cloud so the best workaround is (thanks to Pavel Kotelevsky):
use a custom Connector that the Cloud Connector Runtime does not know about to ensure that the stand-alone Connector Runtime executes all jobs. I did this by copying the OOTB Connector files and changing the type in all of the template files as well as in the java files (do a search on the type string to find all instances). In the case of the Kafka OOTB Connector, that type is “io.camunda:connector-kafka:1”.