Exception : UNAVAILABLE: Abrupt GOAWAY closed unsent stream. HTTP/2 error code: NO_ERROR

Hi Team,

I’m attempting to create a simple proof of concept and perform performance testing in order to recommend Camunda 8 for all of our business workflow needs.

I tried the Camunda Cloud trial version as well as the customised Camunda 8 Sass self managed on Kubernetes with Cluster Size 3.

When I try to express the load testing, when I execute 1000 requests to create the process instances with async from the spring boot client application, we can see that only 150 - 200 requests are created and the rest are not available.

After a while, when I check the logs, I can see the following logs, but there may be multiple occurrences of the same request in the log message.

Exception : UNAVAILABLE: Abrupt GOAWAY closed unsent stream. HTTP/2 error code: NO_ERROR

Note:

  • Observing the Zeebe gateway, zeebe, Spring boot producer, and Spring boot consumer, there is plenty of CPU and memory left. So far, the maximum memory or CPU crossed is 40-50%.

  • As per the Camunda helm chart documentation IO and CPU Thread count for the Zeebe is 3

Please find the below sample code which am using to producer the request.

public void triggerJob(String businessKey, DataDTO data, Operation operation) {
    log.info("ID : {}, Thread Name : {}", data.id(), Thread.currentThread().getName());
    Map<String, Object> variables = new HashMap<>();
    variables.put(Constants.BUSINESS_KEY, businessKey);
    variables.put(Constants.DATA, data);
    variables.put(Constants.START_TIME, Instant.now());
    variables.put(Constants.OPERATION, operation);
    zeebeClient
        .newCreateInstanceCommand()
        .bpmnProcessId("sample-data-creation-id")
        .latestVersion().variables(variables).send().exceptionally(throwable -> {
          log.error("ID : {}, Exception : {}", data.getId(), throwable.getMessage());
          return null;
        });
  }

Can someone suggest the right way to handle this problem.

Thanks & Regards,
Raja Gandharaw.

Hey @GRajaMca

if you create instances async this will cause a high peak on the cluster, which means some are rejected due to back pressure. You could handle the backpressure response in your client code, with exponential retry or you run the starters in a synchronous fashion.

Might make sense for your to also look at some other projects which have been built for that topic/goal. GitHub - camunda-community-hub/camunda-8-benchmark: Helper to create benchmarks for Camunda Platform 8 and Zeebe

Hope that helps.

Greets
Chris

Hi @Zelldon ,

I’m just wondering if there is a way in spring boot to establish backpressure limits at the client application level.
The custom implementation that needs to be built for the backpressure is included in the sample application that you supplied.

For example, if we use a spring kafka, we can specify the backpressure in the application.yaml files, and the consumer library will manage the rest at the client application level. Do we have any Camunda 8 with Spring boot options?