How to add the Kafka exporter to my zeebe broker once it is deployed on kind environment with helm/charts installion

Ritaa · January 19, 2024, 5:15pm

Hello ,

I am trying to export zeebe events to an external system by implementing an exporter Kafka .

Regarding to the documentation I will need to generate the jar of my exporter implementation . And then add this jar under the zeebe pod lib folder .
how I could update on the zeebe properties to add the new exporter configurations and then how I could restart that pod . When my camunda images were installed in kind environment .

Thanks for your attention.

Ritaa

nathan.loding · January 19, 2024, 7:33pm

Hi @Ritaa - you can follow this guide:

Ritaa · February 22, 2024, 8:16am

Hello @nathan.loding ,

So I will need to add this config on the values.yaml of the helm chart and install again the whole camunda platform as I installed my camunda platform by using the helm charts camunda.

Second thing I don’t want to download the jar of an existing exporter . I want to add my custom exporter jar.

Could you please help on this.

Regards
Ritaa

Ritaa · February 22, 2024, 9:14am

Hello @nathan.loding

as a beginner , I want to start my adding a custom exporter that will write on a file.
what I am missing here is the steps to add my jar under the zeebe broker as I installed my camunda platform with helm chart.

nathan.loding · February 22, 2024, 4:05pm

Hi @Ritaa - the docs page listed before shows what values to add to your Helm charts. Our current recommended approach is to use wget to download the jar file. This is shown on that docs page also; it is downloading the Hazelcast and Kafka exporters, but you can download any file you need. Then you set the environment variables to configure the exporter as shown in that example also.

An alternate method would be to deploy everything without the jar, you can create extraVolumes and extraVolumeMounts, and then use kubectl to copy the file into the pod (ex: kubectl copy path-to-jar pod/pod-name:/path/to/volumeMount).

Ritaa · February 22, 2024, 7:39pm

Hello @nathan.loding

I want first to add the exporter to the zeebe and start by a simple exporter code . So I used the follow code and I generated the jar GitHub - jwulf/zeebe-exporter-demo: A minimal Zeebe exporter

Then , I created my custom docker image container where I copied there my exporter jar . then on the initContainer I could copy the file to a volume mount.
the jar was copied on the right place .

here the values files I used :

global:
  identity:
    auth:
      # Disable the Identity authentication for local development
      # it will fall back to basic-auth: demo/demo as default user
      enabled: false


# Disable identity as part of the Camunda core
identity:
  enabled: false
  
operate:
  enabled: false

    

optimize:
  enabled: false

# Reduce for Zeebe and Gateway the configured replicas and with that the required resources
# to get it running locally
zeebe:
  clusterSize: 1
  partitionCount: 1
  replicationFactor: 1
  pvcSize: 0.1Gi
  resources:
   requests:
     cpu: 200m
     memory: 600Mi
   limits:
     cpu: 200m
     memory: 600Mi
  initContainers:
   - name: add-exporter-jar
     image: docker.io/ritaa991/nginx-custom
     #imagePullPolicy: Never
     command: ["/bin/sh", "-c"]
     args:
      [
        "cp /etc/nginx/zeebe-exporter-demo-1.0-SNAPSHOT.jar /exporters/zeebe-exporter-demo-1.0-SNAPSHOT.jar; ls -al /exporters",

      ]
     volumeMounts:
      - name: exporters
        mountPath: /exporters/
        
#        "cp zeebe-exporter-demo-1.0-SNAPSHOT.jar /exporters/zeebe-exporter-demo-1.0-SNAPSHOT.jar; ls -al /exporters",
  podSecurityContext:
    fsGroup: 2000
    runAsNonRoot: true
    runAsUser: 1000
  pvcAccessModes: 
    - ReadWriteMany
  pvcStorageClassName: "efs-sc"
  env:
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_CLASSNAME
    value: io.zeebe.DemoExporter
  - name: ZEEBE_BROKER_EXPORTERS_KAFKAEXPORTER_JARPATH
    value: /usr/local/zeebe/exporters/zeebe-exporter-demo-1.0-SNAPSHOT.jar
    
  extraVolumes:
  - name: exporters-zeebe
    emptyDir: {}
  extraVolumeMounts:
  - name: exporters-zeebe
    mountPath: /usr/local/zeebe/exporters
zeebe-gateway:
  replicas: 1
 

connectors:
  enabled: false
  inbound:
    mode: disabled

elasticsearch:
  enabled: true
  master:
    replicaCount: 1
    # Request smaller persistent volumes.
    persistence:
      size: 0.1Gi

I had the follow error on the zeebe pod logs, so I think the code exporter is not good . Could you please share the right documentation within we can start the implementation of the exporter code .

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-02-22 19:36:22.652 [] [main] [] ERROR
      org.springframework.boot.SpringApplication - Application run failed
java.lang.NoClassDefFoundError: io/zeebe/exporter/api/Exporter
        at java.base/java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
        at java.base/java.lang.ClassLoader.defineClass(Unknown Source) ~[?:?]
        at java.base/java.security.SecureClassLoader.defineClass(Unknown Source) ~[?:?]
        at java.base/java.net.URLClassLoader.defineClass(Unknown Source) ~[?:?]
        at java.base/java.net.URLClassLoader$1.run(Unknown Source) ~[?:?]
        at java.base/java.net.URLClassLoader$1.run(Unknown Source) ~[?:?]
        at java.base/java.security.AccessController.doPrivileged(Unknown Source) ~[?:?]
        at java.base/java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]
        at io.camunda.zeebe.util.jar.ExternalJarClassLoader.loadClass(ExternalJarClassLoader.java:54) ~[zeebe-util-8.4.3.jar:8.4.3]
        at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.load(ExporterRepository.java:80) ~[zeebe-broker-8.4.3.jar:8.4.3]
        at io.camunda.zeebe.broker.Broker.buildExporterRepository(Broker.java:148) ~[zeebe-broker-8.4.3.jar:8.4.3]
        at io.camunda.zeebe.broker.Broker.<init>(Broker.java:71) ~[zeebe-broker-8.4.3.jar:8.4.3]
        at io.camunda.zeebe.broker.Broker.<init>(Broker.java:49) ~[zeebe-broker-8.4.3.jar:8.4.3]
        at io.camunda.zeebe.broker.StandaloneBroker.run(StandaloneBroker.java:87) ~[camunda-zeebe-8.4.3.jar:8.4.3]
        at org.springframework.boot.SpringApplication.lambda$callRunner$5(SpringApplication.java:790) ~[spring-boot-3.2.2.jar:3.2.2]
        at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83) ~[spring-core-6.1.3.jar:6.1.3]
        at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60) ~[spring-core-6.1.3.jar:6.1.3]
        at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88) ~[spring-core-6.1.3.jar:6.1.3]
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:798) ~[spring-boot-3.2.2.jar:3.2.2]
        at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:789) ~[spring-boot-3.2.2.jar:3.2.2]
        at org.springframework.boot.SpringApplication.lambda$callRunners$3(SpringApplication.java:774) [spring-boot-3.2.2.jar:3.2.2]
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source) ~[?:?]
        at java.base/java.util.stream.SortedOps$SizedRefSortingSink.end(Unknown Source) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:?]
        at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:?]
        at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:?]
        at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:?]
        at java.base/java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:?]
        at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:774) [spring-boot-3.2.2.jar:3.2.2]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:341) [spring-boot-3.2.2.jar:3.2.2]
        at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:78) [camunda-zeebe-8.4.3.jar:8.4.3]
Caused by: java.lang.ClassNotFoundException: io.zeebe.exporter.api.Exporter
        at java.base/java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
        at java.base/java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
        at io.camunda.zeebe.util.jar.ExternalJarClassLoader.loadClass(ExternalJarClassLoader.java:57) ~[zeebe-util-8.4.3.jar:8.4.3]
        ... 32 more

Regards
Ritaa

nathan.loding · February 22, 2024, 7:53pm

Hi @Ritaa - just confirming: does the Docker image you built (ritaa991/nginx-custom) contain the jar file already? If not, this script is not copying it to the mounted volume; the jar must already exist in /etc/nginx in the container for it to work.

We don’t have any guides to building your own exporter, yet. There is the repo you already linked, a couple related blog posts on Camunda Blog that you can review, and this documentation. We recommend modeling your exporter after the official Elasticsearch exporter.

Ritaa · February 23, 2024, 3:15am

Hi Team,

I could resolve it . But when it runs I have an error . I added on the code of my expoter a Logger . It causes for me the follow error , who can help please on this.

Error starting ApplicationContext. To display the condition evaluation report re-run your application with ‘debug’ enabled.
2024-02-23 03:03:33.961 [main] ERROR
org.springframework.boot.SpringApplication - Application run failed
java.lang.LinkageError: loader constraint violation: loader io.camunda.zeebe.util.jar.ExternalJarClassLoader @789d8fd6 wants to load interface org.slf4j.Logger. A different interface with the same name was previously loaded by ‘app’. (org.slf4j.Logger is in unnamed module of loader ‘app’)
at java.base/java.lang.ClassLoader.defineClass1(Native Method) ~[?:?]
at java.base/java.lang.ClassLoader.defineClass(Unknown Source) ~[?:?]
at java.base/java.security.SecureClassLoader.defineClass(Unknown Source) ~[?:?]
at java.base/java.net.URLClassLoader.defineClass(Unknown Source) ~[?:?]
at java.base/java.net.URLClassLoader$1.run(Unknown Source) ~[?:?]
at java.base/java.net.URLClassLoader$1.run(Unknown Source) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(Unknown Source) ~[?:?]
at java.base/java.net.URLClassLoader.findClass(Unknown Source) ~[?:?]
at io.camunda.zeebe.util.jar.ExternalJarClassLoader.loadClass(ExternalJarClassLoader.java:54) ~[zeebe-util-8.4.3.jar:8.4.3]
at io.zeebe.KafkaExporter.configure(KafkaExporter.java:28) ~[?:?]
at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.lambda$validate$0(ExporterRepository.java:96) ~[zeebe-broker-8.4.3.jar:8.4.3]
at io.camunda.zeebe.util.jar.ThreadContextUtil.runCheckedWithClassLoader(ThreadContextUtil.java:58) ~[zeebe-util-8.4.3.jar:8.4.3]
at io.camunda.zeebe.broker.exporter.repo.ExporterRepository.validate(ExporterRepository.java:95) ~[zeebe-broker-8.4.3.jar:8.4.3]

Ritaa · February 23, 2024, 3:52am

Hello,

my issues are resolved . the exporter now is working.

Thanks
Ritaa

nathan.loding · February 23, 2024, 12:29pm

Hi @Ritaa - that’s great to hear! Can you share what the final fixes were?

Ritaa · February 23, 2024, 2:53pm

Hi @nathan.loding ,

1- I create my own docker image where I copied there the jar of the exporter , hereafter example of my Dockerfile:

FROM nginx

COPY --chown=101:101 index.html /usr/share/nginx/html
COPY --chown=101:101 zeebe-exporter-2.7.3-SNAPSHOT.jar /etc/nginx/zeebe-exporter-2.7.3-SNAPSHOT.jar
RUN rm -rf /usr/share/nginx/html/*

RUN chown -R 101:101 /var/cache/nginx && \

chmod -R 755 /usr/share/nginx/html && \

    chmod -R 777 /etc/nginx && \

chmod -R 755 /var/cache/nginx

USER nginx

EXPOSE 8080

ENTRYPOINT [“nginx”, “-g”, “daemon off;”]

////////////

then I pushed it to the docker registry .

then I used the same values.yaml that I shared before.

then I did helm install

and it is published now messages to kafka topic .

hoping that could help other people.

But I want to complete my example to migrate the user tasks to my external database. I still don’t know which combinaison of recordType and valueType that I need to use to handle only the user task data.

Thanks
Rawaa YAHYA

nathan.loding · February 23, 2024, 3:39pm

Thanks for sharing the solution!