Growing .log files on Zeebe instance

Camunda 8 Topics Discussion & Questions
1

Hello,

We currently have an issue with our Zeebe v1.2.9 (single instance) on a Kubernetes Cluster. The .log files are growing on the disk and are now occupying 3GB of space (/usr/local/zeebe/data/raft-partition/partitions/1).

We have similar clusters which are only using hundreds of megabytes.

Is there a way to automatically purge these logs?
What is the recommended space for Zeebe?

Thanks in advance for your help.

2

Hello Vincent :wave:

Zeebe should automatically compact the log periodically, after a snapshot has been taken. If you have configured any exporters, for example the Elasticsearch exporter, these must be up and running before the log can be compacted.

If you have access to Zeebe metrics, you could check if the Elasticsearch exporter is active and has exported everything so far. Alternatively, I’d recommend you to check the logs to see if there are any errors related to exporters or snapshots.

As for recommended disk space I can refer you to the documentation here: Resource planning | Camunda Cloud Docs, the tldr is that it is depending on you throughput and number of active instances at any point in time.

3

Hello Oleschoenburg,

Thanks a lot for your reply. We suspect that there is an issue with the hazelcast exporter (1.1.0) in our Zeebe (1.2.9) container. This container works perfectly on many other clusters. The main difference that we have here is the fact the pods in this cluster do not have access to the internet. Could that be the reason why?

We tried with version 1.0.1 of Zeebe with Hazelcast exporter 1.0.0 and it works fine.

Best regards,

Vincent

4

Hey @Vincent,

I’m not very familiar with the Hazelcast exporter but network changes might explain why Zeebe can’t export and then compact the log. I’d recommend to check if the exporter is logging anything helpful.

Maybe @philipp.ossler has an idea?

1 Like
5