Not all zeebe metrics exposed in the zeebe cluster

I used the helm chart zeebe-cluster-helm to build the zeebe cluster, the cluster itself works fine, I can deploy workflow and create instance inside it.

below is the related svc, pods.


I make the gateway service nodeport type.
so I can visit the metrics endpoint http://gateway:port/metrics and below is the only zeebe metrics I can see.
zeebe_long_polling_queued_current

But actually there should be much more metrics like below.
zeebe_stream_processor_events_total
zeebe_running_workflow_instances_total
zeebe_pending_jobs_total
zeebe_incident_events_total

I also tried to refer to the zeebe/gateway.yaml.template at main · camunda/zeebe · GitHub and add the environment variable ZEEBE_GATEWAY_MONITORING_ENABLED to true in the helm chart and redeploy the chart.
but it only add one more metrics ‘zeebe_gateway_topology_partition_roles

So what I want is how to display all the metrics like zeebe_stream_processor_events_total,zeebe_running_workflow_instances_total,zeebe_pending_jobs_total,zeebe_incident_events_total in the endpoint gateway:port/metrics for the zeebe cluster.

one more thing I want to mention is that if I start a standalone zeebe broker with builtin gateway in the port 9600(without docker or k8s), then I can visit the host:9600/metrics to get all the zeebe metrics.

Hey @hang

this is because the metrics are part of (and reported by) the broker, not by the gateway.

Greets
Chris

thanks @Zelldon but how can I get all zeebe metrics in the gateway endpoint, or in the best practice how shall these metrics exposed since the broker itself is not exposed to outside, right?

You can either configure an ingress to expose them, or else deploy something in the cluster to read them - such as Prometheus / Grafana.

Thanks for you reply.
my zeebe broker service name is myzeebe2-zeebe
and my broker has two replica, one endpoint domain name is: myzeebe2-zeebe-0.myzeebe2-zeebe ,another one is: myzeebe2-zeebe-1.myzeebe2-zeebe
according to my testing, I found only the myzeebe2-zeebe-1.myzeebe2-zeebe:9600/metrics can return all the zeebe metrics like ‘zeebe_running_workflow_instances_total’, myzeebe2-zeebe-0.myzeebe2-zeebe:9600/metrics only return very small amount metrics like 'zeebe_snapshot_file_size_megabytes_bucket’
So I can not use the myzeebe2-zeebe:9600/metrics to get the metrics, I have to directly access the myzeebe2-zeebe-1.myzeebe2-zeebe

So I was wondering how is it designed? I have not found too much information on the official document.

What is your partition and replica configuration for this Zeebe cluster?