Hello,
We faced low Zeebe broker performance even with not high load processing.
For the test we’ve used 2 services (1 node each), the first one (shown in yellow below) has 2 workers, the second one (shown in green below) has 1 worker.
Functionality was tested with 10 concurrent users, each one sending 25 requests (the whole test takes up to 30 sec in JMeter)
Unfortunately, we can’t share details of the business flow, but in general, it’s a few conditions where each one may trigger a few workers. Somethings like
Start
If condition1 == true: do A, B, C
If condition2 == true: do X, Y
Finish
Here is a Zeebe configuration
io.camunda.zeebe.broker.system - Starting broker 0 with configuration {
"network" : {
"host" : "0.0.0.0",
"portOffset" : 0,
"maxMessageSize" : "4MB",
"advertisedHost" : "0.0.0.0",
"commandApi" : {
"host" : "0.0.0.0",
"port" : 26501,
"advertisedHost" : "0.0.0.0",
"advertisedPort" : 26501,
"address" : "0.0.0.0:26501",
"advertisedAddress" : "0.0.0.0:26501"
},
"internalApi" : {
"host" : "0.0.0.0",
"port" : 26502,
"advertisedHost" : "0.0.0.0",
"advertisedPort" : 26502,
"address" : "0.0.0.0:26502",
"advertisedAddress" : "0.0.0.0:26502"
},
"security" : {
"enabled" : false,
"certificateChainPath" : null,
"privateKeyPath" : null
},
"maxMessageSizeInBytes" : 4194304
},
"cluster" : {
"initialContactPoints" : [ "zeebe-0.zeebe.dev.svc.cluster.local:26502" ],
"partitionIds" : [ 1 ],
"nodeId" : 0,
"partitionsCount" : 1,
"replicationFactor" : 1,
"clusterSize" : 1,
"clusterName" : "zeebe-cluster",
"heartbeatInterval" : "PT0.25S",
"electionTimeout" : "PT2.5S",
"membership" : {
"broadcastUpdates" : false,
"broadcastDisputes" : true,
"notifySuspect" : false,
"gossipInterval" : "PT0.25S",
"gossipFanout" : 2,
"probeInterval" : "PT1S",
"probeTimeout" : "PT0.1S",
"suspectProbes" : 3,
"failureTimeout" : "PT10S",
"syncInterval" : "PT10S"
},
"raft" : {
"enablePriorityElection" : true
},
"messageCompression" : "NONE"
},
"threads" : {
"cpuThreadCount" : 2,
"ioThreadCount" : 2
},
"data" : {
"directory" : "/usr/local/zeebe/data",
"logSegmentSize" : "512MB",
"snapshotPeriod" : "PT15M",
"logIndexDensity" : 100,
"diskUsageMonitoringEnabled" : true,
"diskUsageReplicationWatermark" : 0.99,
"diskUsageCommandWatermark" : 0.97,
"diskUsageMonitoringInterval" : "PT1S",
"freeDiskSpaceReplicationWatermark" : 41604219,
"logSegmentSizeInBytes" : 536870912,
"freeDiskSpaceCommandWatermark" : 124812657
},
"exporters" : {
"hazelcast" : {
"jarPath" : "/tmp/zeebe-hazelcast-exporter-1.2.1-jar-with-dependencies.jar",
"className" : "io.zeebe.hazelcast.exporter.HazelcastExporter",
"args" : null,
"external" : true
}
},
"gateway" : {
"network" : {
"host" : "0.0.0.0",
"port" : 26500,
"minKeepAliveInterval" : "PT30S"
},
"cluster" : {
"contactPoint" : "0.0.0.0:26502",
"requestTimeout" : "PT15S",
"clusterName" : "zeebe-cluster",
"memberId" : "gateway",
"host" : "0.0.0.0",
"port" : 26502,
"membership" : {
"broadcastUpdates" : false,
"broadcastDisputes" : true,
"notifySuspect" : false,
"gossipInterval" : "PT0.25S",
"gossipFanout" : 2,
"probeInterval" : "PT1S",
"probeTimeout" : "PT0.1S",
"suspectProbes" : 3,
"failureTimeout" : "PT10S",
"syncInterval" : "PT10S"
},
"security" : {
"enabled" : false,
"certificateChainPath" : null,
"privateKeyPath" : null
},
"messageCompression" : "NONE"
},
"threads" : {
"managementThreads" : 1
},
"security" : {
"enabled" : false,
"certificateChainPath" : null,
"privateKeyPath" : null
},
"longPolling" : {
"enabled" : true
},
"interceptors" : [ ],
"initialized" : true,
"enable" : true
},
"backpressure" : {
"enabled" : true,
"algorithm" : "FIXED",
"aimd" : {
"requestTimeout" : "PT1S",
"initialLimit" : 100,
"minLimit" : 1,
"maxLimit" : 1000,
"backoffRatio" : 0.9
},
"fixed" : {
"limit" : 20
},
"vegas" : {
"alpha" : 3,
"beta" : 6,
"initialLimit" : 20
},
"gradient" : {
"minLimit" : 10,
"initialLimit" : 20,
"rttTolerance" : 2.0
},
"gradient2" : {
"minLimit" : 10,
"initialLimit" : 20,
"rttTolerance" : 2.0,
"longWindow" : 600
}
},
"experimental" : {
"maxAppendsPerFollower" : 2,
"maxAppendBatchSize" : "32KB",
"disableExplicitRaftFlush" : false,
"rocksdb" : {
"columnFamilyOptions" : { },
"enableStatistics" : false,
"memoryLimit" : "512MB",
"maxOpenFiles" : -1,
"maxWriteBufferNumber" : 6,
"minWriteBufferNumberToMerge" : 3,
"ioRateBytesPerSecond" : 0,
"disableWal" : false
},
"raft" : {
"requestTimeout" : "PT5S",
"maxQuorumResponseTimeout" : "PT0S",
"minStepDownFailureCount" : 3,
"preferSnapshotReplicationThreshold" : 100,
"preallocateSegmentFiles" : true
},
"partitioning" : {
"scheme" : "ROUND_ROBIN",
"fixed" : [ ]
},
"queryApi" : {
"enabled" : false
},
"consistencyChecks" : {
"enablePreconditions" : false,
"enableForeignKeyChecks" : false,
"settings" : {
"enablePreconditions" : false,
"enableForeignKeyChecks" : false
}
},
"features" : {
"enableYieldingDueDateChecker" : false
},
"maxAppendBatchSizeInBytes" : 32768
},
"executionMetricsExporterEnabled" : false
}
The problem is that although each worker is relatively fast the process, in general, takes quite a lot of time
Picture 1 illustrates time distributions within the processed instance.
The orange bar at the top starts when we send a request to zeebe to create a new process instance and ends with the response from zeebe (we use synchronous calls).
Three bars below depict the time consumed by 3 workers that were used in this instance. Each worker takes 58 to 115 ms, or ~265ms in total which is ~10-15% of the entire time.
This is just one example, but it’s pretty typical.
To dive deeper we’ve tried to use Simple Zeebe monitor and see what’s going on inside the process (the process is different from the screenshot above because http tracing doesn’t show relations to zeebe process but the timing is always pretty much the same).
To simplify understanding please see an attached plot that shows that transition to each new state takes approximately the same amount of time and although each transition is not critically slow (usually up to 100ms) with ~100 steps it makes the overall process slow.
Picture 1 1.png - Google Drive
Picture 2 2.png - Google Drive
So the question is how we can decrease the time of executing these steps needed to change every state? Is it possible to run them in parallel or are there any configurations to improve the performance in this part?
Thanks in advance.
