A question for fellow Kafka users (and on-prem Camunda): how do you store the topic names?
The easiest way would be to have it hard-coded in the diagram. However, we can have different topic names in Prod and QA, therefore hard-coding doesn’t work.
The second easiest would be to have it in a secret, but I think it scales very badly. As we use Camunda more and more, we’d need to add lots of secrets to the Connectors pod. It seems wrong from a software engineering POV, and likely to cause confusion.
A third option I though of was to have a secret that just says whether we’re in QA or Prod, and have a little FEEL expression that evaluates to the correct topic. Should do the work most of the time, but it’s a bit too much boilerplate in each diagram just to connect to Kafka.
Finally, the last idea we could come up with, was to put a placeholder for the topic, e.g. ${TOPIC}, and run the diagram thru a template engine before deploying it. This is definitely the most powerful and flexible solution, but adds several moving parts to the pipeline.
I just wanted to hear what people do, and what are your opinions about these options.
Hi @albx79 - are your topic names “standard” between the environments? For instance, both are topicA but with a prefix (such as topicA-prod and topicA-qa)?
One thought I had reading your post was to extend the Kafka Connector Template yourself and provide some of that boilerplate “behind the scenes.” Here’s a blog post that talks through extending existing templates (and links to other resources too!):
Yet another solution would be to not use a connector and use a usual worker instead where you have all the power of a decent programming language including the configuration.
I don’t quite like the idea of connectors. They are appealing for the presales presentations but don’t quite fit the idea of BPM IMO. And have some technical deficiencies like the one described above.
Hi @nathan.loding , the topic names are almost standard, but there are actually 2-3 standards and some exceptions. But I could work on 2-3 templates for the few standards, and fix the exceptions. It’s a good idea.
@fml2 I need the kafka message to activate the process. How can I do it with a worker? Thanks!