Hello, I’m running into a strange issue.
We have upgraded to Camunda 7.20 and using the prometheus spring boot actuator to gather metrics on the server. The metric
http_server_requests_seconds_count
tag is getting messed up with certain conditions.
If I hit the prometheus endpoint first, then we do not get the metric for the
engine-rest
endpoints.
If I hit the engine-rest
endpoints first, then we get the metric with correct tags.
I thought this may have something to do with the servlet not starting so I turned on the application property.
spring.servlet.load-on-startup= 1
Now if I hit the prometheus endpoint first, then the engine-rest endpoints. I see the metric, but it is tagged with uri="UNKNOWN"
Yana
January 26, 2024, 5:17pm
2
Hi @Frankwayne ,
Welcome to the forum!
From which version did you upgrade?
Side note: we do not have any custom implementation for Prometheus metrics. I would assume that a change in the web applications is affecting the metrics in a way if it was working before out of the box.
Best,
Yana
thank you for the welcome
it was Camunda 7.18 to Camunda 7.20
that is correct the prometheus endpoint comes pre-packaged with spring-boot
The Camunda 7.20 update was also with a spring boot update from 2 to 3 as well
@Frankwayne
Hi, I’m facing the same problem, have you found a solution?
Sadly not yet, I had to move onto other tasks
Anyway, here’s my solution that I could come up with.
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.ApplicationContext;
import org.springframework.context.ApplicationListener;
import org.springframework.core.env.Environment;
import org.springframework.stereotype.Component;
import java.io.IOException;
import java.net.InetAddress;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import static java.time.temporal.ChronoUnit.SECONDS;
@Component
public class EndpointRunner implements ApplicationListener<ApplicationReadyEvent> {
private static final Logger log = LoggerFactory.getLogger(EndpointRunner.class);
@Autowired
private ApplicationContext applicationContext;
@Override
public void onApplicationEvent(ApplicationReadyEvent event) {
ExecutorService executorService = Executors.newCachedThreadPool();
try {
log.info("Creating a single request for http metrics to work correctly.");
String host = InetAddress.getLocalHost().getHostAddress();
int port = applicationContext.getBean(Environment.class).getProperty("server.port", Integer.class, 8080);
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create("http://" + host + ":" + port + "/engine-rest/incident/count"))
.timeout(Duration.of(5, SECONDS))
.build();
HttpClient httpClient = HttpClient.newBuilder()
.executor(executorService)
.build();
httpClient.send(request, HttpResponse.BodyHandlers.ofString());
} catch (IOException | InterruptedException e) {
e.printStackTrace();
// throw new RuntimeException(e);
} finally {
executorService.shutdown();
}
}
}