What are the recommendations around variable sizes and active running process instances in Zeebe/Camunda Cloud?

Matt Arnold: Hello, Can someone point me to any docs related to Limits and Quotas for Zeebe/Camunda Cloud?

Specifically, I’m looking for:
• max data that can be stored as variables in a single process instance
• Any hard limit on maximum concurrent active processes (Google Workflows is limited at 1,000)

korthout: Hi @Matt Arnold :wave:

If you have questions related to specific https://camunda.com/products/cloud#compare|plans, please reach out to sales. I can try to provide a bit of technical info:

> • max data that can be stored as variables in a single process instance
AFAIK we do not have this documented at this time. It depends on your https://camunda.com/products/cloud#compare|plan. Technically it is controlled by the https://github.com/camunda-cloud/zeebe/blob/main/dist/src/main/config/broker.standalone.yaml.template#L135-L137|maxMessageSize. Each partition in Zeebe has a log with records whose size cannot exceed this limit. The limit for the Professional Plan is set to the default of 4MB, I assume that this is negotiable for the Enterprise Plan.

Note that variable records will always also contain some metadata that eats away from this limit. So you’ll not be able to use the full 4MB for a single variable.

Note that a process instance can have many individual variables each of which is limited by this value.

When activating jobs with variables, the entire batch of activated jobs is also a single record limited by this constraint. So if you have 3 jobs available and the process instances that these jobs belong to each have a large variable of multiple MBs then when you request the jobs the provided number of jobs will be limited to what fits in a single record (the record will contain each individual job and their variables). You can control which variables are fetched along with the job using https://docs.camunda.io/docs/apis-clients/grpc/#activatejobs-rpc|fetchVariable.

Note that we generally recommend not to use Zeebe as a data store, but rather store the data in an external system and reference it with ids/keys. This greatly improves the throughput of the system and reduces complications that may arise when single variables are near this limit.

> • Any hard limit on maximum concurrent active processes (Google Workflows is limited at 1,000)
This is not defined, but of course, there are physical limits to the disk size. Zeebe needs to keep track of each process instance’s state which is kept on disk. To put it in perspective, we’ve played around with millions of process instances (albeit that these were very small processes with little state to keep track of). I recommend trying it out, and to keep an eye on the Zeebe disk metrics in the Metrics Tab of Console while you grow the number of process instances for your processes.

Matt Arnold: Awesome, this is very helpful, Thank you!

Note: This post was generated by Slack Archivist from a conversation in the Zeebe Slack, a source of valuable discussions on Zeebe (get an invite). Someone in the Slack thought this was worth sharing!

If this post answered a question for you, hit the Like button - we use that to assess which posts to put into docs.

1 Like