Slow performance when starting/creating Workflow Instances

Niklaus · October 21, 2020, 3:03am

Hey guys,
I got into trouble with performance when starting/creating Workflow Instances. Can anyone help me with this? Here are some details: Speed of creating workflow instances drops significantly with higher Zeebe versions · Issue #5639 · camunda/zeebe · GitHub

Zelldon · October 21, 2020, 7:13am

Hey @Niklaus,

I answered in the github issue.

github.com/camunda/zeebe

Speed of creating workflow instances drops significantly with higher Zeebe versions

opened 11:28AM - 20 Oct 20 UTC

closed 07:11AM - 21 Oct 20 UTC

PinoNTK

kind/flake

Hi, we are currently using Zeebe version 0.22.5 after upgrading from Zeebe versi…on 0.21.0-alpha3 and trying to run benchmarks on it but the results are not what we expected. With each version, we run the following test command: ``` /home/debian/ghz/cmd/ghz/ghz --insecure --proto ./gateway-protocol/src/main/proto/gateway.proto --call gateway_protocol.Gateway.CreateWorkflowInstance -d '{"workflowKey": 2251799813685252}' -n 1000 localhost:8500 ``` and get the following results: - __Version 0.21.0-alpha3__ ![Screenshot from 2020-10-20 16-43-35](https://user-images.githubusercontent.com/26008333/96578650-b8b8a080-12ff-11eb-854e-7378fafebb78.png) __CPU usage__ ![Screenshot from 2020-10-20 16-40-58](https://user-images.githubusercontent.com/26008333/96578679-cb32da00-12ff-11eb-992d-c5cf805ceae2.png) - __Version 0.25.x__ ![Screenshot from 2020-10-20 17-11-24](https://user-images.githubusercontent.com/26008333/96578732-e0a80400-12ff-11eb-8505-f018899e412f.png) __CPU usage__ ![Screenshot from 2020-10-20 17-11-12](https://user-images.githubusercontent.com/26008333/96578764-ef8eb680-12ff-11eb-84ac-d1313d4602de.png) We are running the benchmarks on a server with **16 CPUs, 16G RAM, HDD disk**. It is surprising that the speed drops dramatically with later versions. We tested with various versions from 0.22.x to 0.25.x and it gradually slows down. From the results, It seems that the engine in later versions does not make use of CPUs (the CPU consumption is quite low), which results in a bad performance. Are we missing some configs? Please give us some suggestions and explanations. Here is our config for benchmarking: [application.yaml](https://github.com/PinoNTK/zeebe/blob/main/zeebe_default_origin.yaml)

Greets
Chris

Niklaus · October 21, 2020, 11:48am

@Zelldon Do you have any comment about CPU load/usage?

system · January 31, 2024, 10:10am