Zeebe performance issue

Camunda 8 Topics Discussion & Questions
1

We tried a single flow with 10 empty service tasks.

In worker handlers we are only using Complete Job method.
It takes an average of 150 ms to finish a process instance.

How can we decrease it? Because it is too long for us.

We are using:
C# for workers
Docker Desktop for camunda components (including zeebe)

docker-compose file:
-----------------------------------
** zeebe: # Docker | Camunda Platform 8 Docs
** container_name: zeebe**
** image: camunda/zeebe:${CAMUNDA_PLATFORM_VERSION:-latest}**
** restart: on-failure**
** ports:**
** - “26500:26500”**
** - “9600:9600”**
** environment: # https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/environment-variables/**
** - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_CLASSNAME=io.camunda.zeebe.exporter.ElasticsearchExporter**
** - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_URL=http://elasticsearch:9200**
** - ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_BULK_SIZE=1**
** # allow running with low disk space**
** - ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998**
** - ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999**
** - “JAVA_TOOL_OPTIONS=-Xms512m -Xmx512m”**
** volumes:**
** - zeebe:/usr/local/zeebe/data**
** networks:**
** - camunda-platform**
** depends_on:**
** - elasticsearch**
------------------------

image
image737×379 45.7 KB

test.bpmn (9.3 KB)

2

Hey @hmeniz

actually, the numbers sound quite good tbh.

Questions to that:
Are you running on the same machine? Is your setup related to your production setup? If not I would suggest to first setup it close to how it would look like in production otherwise your benchmarks are useless.

In order to improve performance you can always increase resources (like CPU and ram), add more nodes, and increase the partition count to improve parallelism. Furthermore, it makes sense to use a fast disk (like an SSD), since Zeebe is really IO-intense.

Are you using a standalone gateway? An embedded gateway will steal the broker’s resources on high load.

How many workers are you running?

What are your current load and the expected load in production? Do you test with the expected payload? There are many factors/parameters which can influence performance and you should consider all to be close to the production scenario.

Greets
Chris

3

Hi Chris,

  • Are you running on the same machine?

  • Yes

  • In order to improve performance you can always increase resources (like CPU and ram), add more nodes, and increase the partition count to improve parallelism.

  • 64 gb ram and i7-11800h cpu (8 core)

  • Furthermore, it makes sense to use a fast disk (like an SSD), since Zeebe is really IO-intense.

  • We are using m2-SSD

  • Are you using a standalone gateway? An embedded gateway will steal the broker’s resources on high load.

  • No, it is embedded gateway

  • How many workers are you running?

  • Only for this test, 10 worker but they do nothing (only complete method)

  • What are your current load and the expected load in production? Do you test with the expected payload? There are many factors/parameters which can influence performance and you should consider all to be close to the production scenario.

  • Current there is only 1 empty instance with 10 flow-node performance. In production there will be maybe thousands. And flow instances will call api calls.

4

@hmeniz - did you upgrade your setup to deploy on multiple nodes, k8s along with using a standalone gateway?
I am also benchmarking zeebe to figure out the right cluster configuration for our needs. Wondering if we could learn from each other :slight_smile:

5

If you really want to know: New Zeebe Performance Benchmarking Tool

In the video in that post, Falko shows you how deep the rabbit hole goes.

L2 cache configuration, CPU architecture - it’s the ultimate performance-tuning guide.