Camunda 7.20, Spring boot, prometheus metrics

Camunda 7 Topics
,
1

Hello, I’m running into a strange issue.

We have upgraded to Camunda 7.20 and using the prometheus spring boot actuator to gather metrics on the server. The metric

http_server_requests_seconds_count tag is getting messed up with certain conditions.

If I hit the prometheus endpoint first, then we do not get the metric for the
engine-rest endpoints.

If I hit the engine-rest endpoints first, then we get the metric with correct tags.

I thought this may have something to do with the servlet not starting so I turned on the application property.

spring.servlet.load-on-startup= 1

Now if I hit the prometheus endpoint first, then the engine-rest endpoints. I see the metric, but it is tagged with uri="UNKNOWN"

2

Hi @Frankwayne,

Welcome to the forum!

From which version did you upgrade?

Side note: we do not have any custom implementation for Prometheus metrics. I would assume that a change in the web applications is affecting the metrics in a way if it was working before out of the box.

Best,
Yana

3

thank you for the welcome :slight_smile:

it was Camunda 7.18 to Camunda 7.20

that is correct the prometheus endpoint comes pre-packaged with spring-boot

The Camunda 7.20 update was also with a spring boot update from 2 to 3 as well

4

@Frankwayne
Hi, I’m facing the same problem, have you found a solution?

5

Sadly not yet, I had to move onto other tasks

6

Anyway, here’s my solution that I could come up with.

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationListener;
import org.springframework.core.env.Environment;
import org.springframework.stereotype.Component;

import java.io.IOException;
import java.net.InetAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

import static java.time.temporal.ChronoUnit.SECONDS;

@Component
public class EndpointRunner implements ApplicationListener<ApplicationReadyEvent> {

    private static final Logger log = LoggerFactory.getLogger(EndpointRunner.class);
    @Autowired
    private ApplicationContext applicationContext;

    @Override
    public void onApplicationEvent(ApplicationReadyEvent event) {
        ExecutorService executorService = Executors.newCachedThreadPool();

        try {
            log.info("Creating a single request for http metrics to work correctly.");
            String host = InetAddress.getLocalHost().getHostAddress();
            int port = applicationContext.getBean(Environment.class).getProperty("server.port", Integer.class, 8080);

            HttpRequest request = HttpRequest.newBuilder()
                    .uri(URI.create("http://" + host + ":" + port + "/engine-rest/incident/count"))
                    .timeout(Duration.of(5, SECONDS))
                    .build();

            HttpClient httpClient = HttpClient.newBuilder()
                    .executor(executorService)
                    .build();
            httpClient.send(request, HttpResponse.BodyHandlers.ofString());

        } catch (IOException | InterruptedException e) {
            e.printStackTrace();
//            throw new RuntimeException(e);
        } finally {
            executorService.shutdown();
        }
    }
}