Remote Camunda Kafka usage

Camunda 7 Topics Discussion & Questions
1

Moving the question from meenarc out of Remote Camunda Kafka usage · Issue #25 · berndruecker/flowing-retail · GitHub into this forum:

Was looking through the use cases here where the examples show how to use the embedded camunda to talk to Kafka

We have a use case where a remote Camunda cluster would be used for service orchestration across multiple use cases.
In such cases we would have a flow where the remote service may send/receive messages to the event bus and remote Camunda would need to process these messages…
Also the same or diff workflow can talk to different kafka topics and send /receive diff events
What are the options in such cases when designing the workflow?

  1. Send task to send messages?
  2. Receive task-but can we receive messages from diff topics within the same workflow?How do we handle kafka topic offset reset in case of retries etc in such cases?
  3. Have a customized service task to handle kafka communication , deployed in the remote engine ? This lets us throw in some JSON customization,error handling etc . This approach is sort of what we are leaning towards now.
  4. Create a custom kafka connector?
2

It is unfortunately hard to give any recommendation yet, as there are still lots of things unclear.

Let’s start the discussion :slight_smile:

I see two basic options you have:

grafik

  1. Green: Camunda is a remote workflow engine, but you basically work like you would with an embedded one. So Camunda never directly talks to Kafka, but your service does.
  2. Orange: You want Camunda to talk to Kafka directly. Therefor you develop a bespoke connector (the orange bubble) that can send/receive records from/to Kafka.

I am not completely sure about what of these scenarios we are talking?

And I also throw in another resource - not really connected to this question: With Zeebe/Camunda Cloud we did a Kafka Connector here: GitHub - camunda-community-hub/kafka-connect-zeebe: Kafka Connector for Zeebe.io. Maybe that can also give you some inspiration?

This might miss your question, but hopefully you can refine it a bit so that i can better answer it!
Thanks
Bernd