ResourceExhausted message from Zeebe Cluster during deployment

eapen_eapen · September 8, 2020, 3:18pm

Hi ,

we are getting ResourceExhausted error from Zeebe during deployment application from a worker instance. This we are executing during the initial startup phase of the .net application and configured 7 instance for the particular type.

please can you help us to debug the issue?

Thanks,
Eapen T John

Full error

jwulf · September 9, 2020, 1:51pm

Yes, two things:

What code is executing to cause this?
What does the broker log show?

eapen_eapen · September 9, 2020, 2:10pm

We are executing the workflow deployment command using .net zb-client nuget package
i dont see new messages on any of the zeebe brokers . but just seeing a lot of snapshot warnings on logs

This is an intermittent error comes from application , most of the times it is business as usual
Just something came to my thinking , do we need to use any unregister action against broker during application shutdown.

Thanks,
Eapen T John

jwulf · September 11, 2020, 2:09pm

It sounds like it could be this?

github.com/camunda/zeebe

ResourceExhausted constantly appears during workflow deployment

opened 12:12PM - 14 Nov 19 UTC

closed 08:40AM - 15 Jun 21 UTC

IcyEagle

kind/bug area/performance blocker/info severity/mid

Hi there! I use one of the latest version of Zeebe under the hook (took a sna…pshot after 0.21.1). Since this morning I have had an error **ResourceExhausted** when I try to upload an updated workflow. It's reproduced stably. It's strange since I have no load on my 5-nodes cluster. I have some workers connected but there are no active instances. Here is my config: ``` # For more information about this configuration visit: [threads] cpuThreadCount = 40 [gateway.monitoring] enabled = true [network] host = "0.0.0.0" [[exporters]] id = "log" className = "org.projectx.LogExporter" [exporters.args] enabledValueTypes = "JOB,WORKFLOW_INSTANCE,DEPLOYMENT,INCIDENT,TIMER,VARIABLE,MESSAGE,MESSAGE_SUBSCRIPTION,MESSAGE_START_EVENT_SUBSCRIPTION" ``` Here is the command: ```ls -d $dpmn_sources_dir/* | xargs /usr/local/zeebe/bin/zbctl --certPath ca-cert --address $ZEEBE_HOST deploy``` and here is the result: ```Error: rpc error: code = ResourceExhausted desc = Reached maximum capacity of requests handled``` How can I deal with it? One more thing about performance: how is it better to monitor Zeebe cluster state in production environment? I have Zeebe metrics active + Prometeus/Grafana connected. Is it enough in general? Is there a guide somewhere maybe? :) Thanks!

Is there any reason you are using 0.23.2?

eapen_eapen · September 21, 2020, 9:06am

its always lot of work to get all the items updated. we will be updating to latest soon

Thanks,
Eapen T John