RESOURCE_EXHAUSTED: Expected to activate jobs of type

Hassoun · May 10, 2022, 8:42am

We are getting the error of RESOURCE_EXHAUSTED: Expected to activate jobs of type in a huge amount, once I add a new worker, which makes our logs full of this error, tried to find a solution, but nothing works, here are my settings from zbctl status --insecure

Cluster size: 3
Partitions count: 1
Replication factor: 3
Gateway version: 1.2.8
Brokers:
  Broker 0 - emil-bp-zeebe-0.emil-bp-zeebe.default.svc.cluster.local:26501
    Version: 1.2.8
    Partition 1 : Follower, Healthy
  Broker 1 - emil-bp-zeebe-1.emil-bp-zeebe.default.svc.cluster.local:26501
    Version: 1.2.8
    Partition 1 : Follower, Healthy
  Broker 2 - emil-bp-zeebe-2.emil-bp-zeebe.default.svc.cluster.local:26501
    Version: 1.2.8
    Partition 1 : Leader, Healthy

I have between 12 to 14 workers, and every time i am adding one more worker, this error becomes stronger and stronger and pushes alarm every millisecond.

08:38:29.464 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:39:19.499 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:39:19.501 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:39:39.510 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:39:39.511 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:39:59.536 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:39:59.538 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:40:09.561 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:40:09.562 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:40:39.590 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:40:39.591 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:40:49.623 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:40:49.626 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:40:59.653 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:40:59.655 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:41:19.675 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:41:19.677 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:41:49.708 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:41:49.714 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:41:59.740 | zeebe |  [account:notify] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:notify', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.
08:41:59.743 | zeebe |  [account:refresh_idempotency_key] ERROR: Grpc Stream Error: 8 RESOURCE_EXHAUSTED: Expected to activate jobs of type 'account:refresh_idempotency_key', but no jobs available and at least one broker returned 'RESOURCE_EXHAUSTED'. Please try again later.

Would really like to know exactly why this issue happening and how we could avoid it in the future.

BerndRuecker · May 11, 2022, 7:08am

Hi Hassoun.

This error typically indicates backpressure (the broker is at capacity and cannot take more load). Did you try to run fewer workers? Then this should probably go away (at least you can verify if it is around backpressure). See also Backpressure | Camunda Platform 8.

Also keep in mind that it is often more beneficial to write your workers in a scalable manner than simply scale them out (see Writing good workers | Camunda Platform 8). During benchmarking (How to Benchmark Your Camunda 8 Cluster | by Bernd Rücker | Apr, 2022 | berndruecker) I could bring down 10-20 workers to a single-threaded Java app by using reactive programming properly But just an additional thought…

Best
Bernd

(For documentation: Cross-posted via node.js - RESOURCE_EXHAUSTED: Expected to activate jobs of type camunda - Stack Overflow)

Hassoun · May 11, 2022, 9:05am

Hi Bernd,

Thank you for your reply, true it is because of Backpressure,
once i decrease the amount of workers, the error will not be exist anymore, i am pretty new to camunda so i dont know if my questions is kind of basics but i need to have more and more workers so wondering if i need to increase the space or adding more broker or should i increase my resources ?
i am using zeebe-node for worker creation which according to the documentation is doing a good job Writing good workers | Camunda Platform 8 with workers creation.

Best regards
Hussein

BerndRuecker · May 11, 2022, 6:07pm

You run Zeebe self-managed or SaaS?
As a next step I would check resource consumption via Prometheus/Grafana on the cluster, normally that gives you a good idea where the bottleneck is. Typically is somewhere around CPU or memory of the broker.

Also I would not run Zeebe with 1 partition, as this will not scale anyway (See Partitions | Camunda Platform 8), you need to increase partitions to be able to utilize more CPU power.

Hassoun · May 11, 2022, 6:50pm

I am running self managed .

korthout · May 12, 2022, 4:56pm

Hi @Hassoun

Like we discussed in person, I recommend trying the following:

it might be that the resource exhausted happens due to available disk space, you could try to increase the pvc’s disk space
complete more process instances to clear-up data
set up the monitoring with Grafana, so you have more insights into the cluster (incl used disk space)

Good luck and it was fun talking to you
Nico