Kafka Producer Connector Failing

[ajs1]

Hi Everyone,

I’m quite new to Camunda and am trying to setup a very simple process using Camunda 8 Self-managed and Kafka in my local MacOS environment, so that I can build on that knowledge and start creating more complex processes:

I am able to deploy and start the process to Camunda and see it in Operate:

My settings for the Kafka Producer connector (v1) task_a are:

And the error message is:

Camunda services seem to be working well too (I’m using the latest Dockerised version):

Kafka and Zookeeper are also up and running (these are not dockerised) and I can send and receive messages to the topics of interest using two dummy python programs I’ve written:

I really don’t know what I’m doing wrong and would appreciate a helping hint

Best
Anton

[nathan.loding]

Hi @ajs1, welcome to the forums! This is not an issue with Camunda, but rather this has to do with Docker networking. Camunda is running the Connector inside a virtual container and attempting to connect to localhost:9092 from inside that container. In other words, localhost resolves to the container itself, not to your host machine that the containers are running on.

I did a quick Google search and here’s one article that may help:

How to Connect to Localhost from Docker Container

[ajs1]

Thank you Nathan for your answer.

I’ve tried the different options but without success. I’m running the dockerised version on a MacOS M1 machine which has special installation instructions for keycloak so I’m not sure if this is what is causing the issue.

I’m attaching my docker-compose yaml and model files as I’m also not 100% sure if they’re correct:
docker-compose.yaml (16.2 KB)
sequential_automated_v4.bpmn (4.9 KB)

Not really sure where the problem is and I’m quite stuck at the moment, so any help/hint would be greatly appreciated.

[ajs1]

Also a log from Docker where I can see several warnings (I had to rename the file to .yaml to be able to upload it).
docker_log.txt.yaml (1.5 MB)

[nathan.loding]

@ajs1 - this is not a Camunda issue, or an issue with your BPMN model. This is Docker networking, which is heavily dependent on how you’ve configured your environment (how/where is Kafka running and what address+port is it listening on; what security services, like firewalls, might be blocking traffic; etc.). If you notice, the docker-compose configuration you are using is creating a virtual network; nearly all the services have this annotation:

    networks:
      - camunda-platform

This is what allows the services inside the containers to communicate with each other (eg, how Operate talks with Zeebe). Docker does not know how to communicate with services running outside that virtual network. One quick fix is to run Kafka inside a container and join it to the same camunda-platform network.

[ajs1]

Thank you Nathan. That worked quite well. For those of you that are new to Camunda, want to use Kafka and are also having similar issues, the docker-compose yaml file I used is attached. The bootstrap server to use is “kafka:9092”.
docker-compose_camunda_kafka_apps.yaml (17.0 KB)

[nathan.loding]

Thanks for sharing your working docker-compose file!