Camunda 8.6 self managed - Grafana Dashboard / broker config issue

[Raghavendra_Chari]

Hi there, I am playing with self managed Camunda 8.6 and trying to view the stats via Grafana Dashboard elements.

Installed 8.6 on Mac-mini-os. KIND cluster.
Installed Prometheus and Grafana

I included Zeebe broker configurations as mentioned in the docs.

ZEEBE_BROKER_EXPERIMENTAL_ROCKSDB_ENABLEACCESSMETRICS=fine
prometheusServiceMonitor.enabled="true"

In order to test my env, I created bpmn process that has job worker and error handling with email notification.

I am able to run a dozen instances using web-hook and I can see all of them sending email notifications (via Operate). The emails are delivered as well.

I have two questions w.r.t Grafana Dashboard.

a. My test data throws a BPMN error which is handled in the process def. I was hoping to see these stats reflected on Grafana under “Worker Jobs”.

For some reason, the Grafana Dashboard does not show anything - it shows No data. I ran the exercise couple of times, but no luck.

b. In Grafana Dashboard, some of the stats are populated, such as - Messaging, CPU, IO - but a lot of them are empty. I am trying to get to see stats under elements such as Processing Error Handling Phase, Throughput, Latency. These are unfortunately empty - No data.

I am suspecting that prometheus configuration perhaps is not configured correctly.

When I ran the “helm install …” command, I do see in the stdout log messages below summary. Note prometheus monitor flag is false. I am not able to figure out how to get this flag set to true.

$ helm install camunda camunda/camunda-platform -f values-combined-ingress.yaml -f values-connectors-env.yaml

Installed Services:

• Console:
  • Enabled: true
  • Docker Image used for Console: camunda/console:8.6.44
• Zeebe:
  • Enabled: true
  • Docker Image used for Zeebe: camunda/zeebe:8.6.7
  • Zeebe Cluster Name: “camunda-zeebe”
  • Prometheus ServiceMonitor Enabled: false
• Operate:
  • Enabled: true
  • Docker Image used for Operate: camunda/operate:8.6.7
• Tasklist:
  • Enabled: true
  • Docker Image used for Tasklist: camunda/tasklist:8.6.7
• Optimize:
  • Enabled: true
  • Docker Image used for Optimize: camunda/optimize:8.6.4
• Connectors:
  • Enabled: true
  • Docker Image used for Connectors: camunda/connectors-bundle:8.6.6
• Identity:
  • Enabled: true
  • Docker Image used for Identity: camunda/identity:8.6.7
  • Keycloak: camunda/keycloak:25.0.6
• Web Modeler:
  • Enabled: false
• Elasticsearch:
  • Enabled: true
  • Docker Image used for Elasticsearch: bitnami/elasticsearch:8.15.4

I am attaching the files for your kind review.

values yml
bpmn file + dmn file + form

Here is the job worker application.yml file.

spring:
  application:
    name: tasklist-toolkit

server:
  port: 9010

camunda:
  client:
    mode: self-managed
    zeebe:
      enabled: true
      grpc-address: http://127.0.0.1:26500
      rest-address: http://127.0.0.1:8088
    auth:
      client-id: rope-toolkit-app
      client-secret: cryptic-data
      issuer: https://c86-mini.makelabs.in/auth/realms/camunda-platform/protocol/openid-connect/token

The job worker is in fact processing all the transactions.

Kindly advise.

lead-to-opportunity-with-dmn.bpmn (20.7 KB)
lead-to-opportunity-with-webhook-Form.form (1.9 KB)
new-lead-approval.dmn (3.2 KB)
values-combined-ingress.yaml (5.0 KB)

[nathan.loding]

Hi @Raghavendra_Chari, welcome to the forums! Have you enabled Zeebe in your Prometheus configuration? (Docs reference)

As for what metrics are available, all of the available metrics can be found here: Metrics | Camunda 8 Docs

[Raghavendra_Chari]

Yes - I have enabled Zeebe in Prometheus.

Here is the prometheus /targets view. I can see the status is UP.

https://c86-mini.makelabs.in/targets

[nathan.loding]

Hi @Raghavendra_Chari - it looks like your endpoint is returning data - if you click on the Zeebe actuator endpoint you see all the data. It would seem the issue is on the Prometheus side, but why that might be I’m not sure unfortunately.

You might need to enable ZEEBE_BROKER_EXECUTION_METRICS_EXPORTER_ENABLED instead or in addition to ZEEBE_BROKER_EXPERIMENTAL_ROCKSDB_ENABLEACCESSMETRICS (docs reference)

FYI, that URL is open to the public, not sure if it contains anything sensitive!

[Raghavendra_Chari]

The URL is public intentionally; I was testing with some changes to see if affects loading the Topology in Grafana dashboard.

I will be removing the public URL shortly. I also checked the content to see nothing important is there - but sometimes you never know! Thanks for pointing it.

I have tried to enable execution metrics flag - it is set to “true”. I will review this once again. Thank you for your inputs.

[nathan.loding]

@Raghavendra_Chari - we’ve taken my Prometheus knowledge as far as it can go, but let me know if you are still stuck after the review. I can share this with some engineers and get their perspective too.

[Raghavendra_Chari]

@nathan.loding - I checked the broker node logs and found an entry indicating the GET operation is failing.

GET not found /zeebe/actuator/prometheus

My config was incorrect. For the Zeebe node, the path is “/actuator/prometheus”.

Here is the modified prometheus yaml file. With this version, I am able to see the data populated in the panels.

prometheus.yaml (618 Bytes)

Some sample screenshots attached.

[nathan.loding]

I’m glad it was an easy fix! I do see that we have /actuator/prometheus in the documentation, but perhaps we could make it more clear how to make sure that path is accurate. If you have any suggestions on how to improve the documentation here, I’d love to hear it!