Centralised Logging For Zeebe

Camunda 8 Topics
1

We are deploying camunda 8.1.6 on k8s through helm. We need to deploy around 15-20 brokers to achieve our processing goals. I would like to know ways to push all gateway and broker logs to a centralised logging platform which in our case is EFK (elasticsearch, fluentd and kibana). EFK being a popular and standard stack for logging.

Can zeebe be configured to push its logs to it?

Thanks.

2

Hi @jgeek1

Zeebe uses Log4j2 - Logging | Camunda Platform 8 Docs. Is that not something that you can configure in there? Or do you need to add a log appender for ES to the image?

If that’s the case, you would need to create your own container images from the Camunda ones and use those.

Josh

3

Thanks @jwulf for the reply.

Zeebe uses Log4j2 - Logging | Camunda Platform 8 Docs . Is that not something that you can configure in there?

Fluentd is another docker container that runs on all servers. It reads logs logged in json format from other containers and ships them to elasticsearch. For logback logging we use net.logstash.logback.encoder.LogstashEncoder to write the logs in json format. We need to figure out how do we do it for log4j.

Or do you need to add a log appender for ES to the image?
If that’s the case, you would need to create your own container images from the Camunda ones and use those.

That would make zeebe-upgrades difficult for us, right? Every-time a new release comes we will have to sync our copied zeebe code from the github source and generate new images.

I wonder how do other solve this need of looking into zeebe logs when troubleshooting production issues especially when there are multiple gateways and brokers running in a cluster. Connecting to the server and reviewing the console isn’t practical.
Any inputs?

Thanks.

4

Google Stackdriver is the solution used by Camunda SaaS.

Could this be done at the infrastructure level? Capture the console of pods and redirect it, so that the application does not need to be environment-aware?

5

We found an issue in the helm chart that leads to an error in pushing logs from fluentd to elasticsearch. It would be great if someone can look into the issue so that centralised logging starts working for EKF stack.

I have raised a github issue for tracking this. Could we get some attention there?

Thank you.

6

Apparently the issue is with the app label. If we remove it the logs get pushed to elasticsearch. The only problem remains is removing the app label from the values.yaml file.

I tried setting them null both in the values.yaml and in the helm install command but it didn’t help.

global:
  labels:
    app: null

helm install dev camunda/camunda-platform -f camunda-local.yaml --set global.labels.app=null

Any idea why the app label is not getting set to null?

Thanks.