Log entry floods console

Since version 8.3.0, the following warning message floods our consoles in various components. We are using an externally hosted keycloak:

[ult-executor-14] i.c.z.c.i.ZeebeCallCredentials: The request’s security level does not guarantee that the credentials will be confidential.

The debug output has some more information:

DEBUG 1 --- [ult-executor-14] s.n.w.p.h.HttpURLConnection              : sun.net.www.MessageHeader@3a9184037 pairs: {POST /auth/realms/camunda/protocol/openid-connect/token HTTP/1.1: null}{Content-Type: application/x-www-form-urlencoded}{Accept: application/json}{User-Agent: zeeb ││ 2023-10-13 09:35:28.428 DEBUG 1 --- [ult-executor-14] s.n.w.p.h.HttpURLConnection: sun.net.www.MessageHeader@30350c914 pairs: {null: HTTP/1.1 200 OK}{Server: nginx}{Date: Fri, 13 Oct 2023 09:35:28 GMT}{Content-Type: application/json}{Content-Length: 1627}{Connection: keep-alive}{Referrer-Policy: n ││ 2023-10-13 09:35:28.428  WARN 1 --- [ult-executor-14] i.c.z.c.i.ZeebeCallCredentials: The request's security level does not guarantee that the credentials will be confidential.

Is there any way to solve this warning?

Thanks in advance

Hi @lukas-beumer, welcome to the forums! I think the times I’ve seen this before it has been related to networking (firewall, proxy, etc.) - here’s a related GitHub issue where it was caused by Zscaler: Error: rpc error: code = Unavailable desc = connection closed before server preface received · Issue #8937 · camunda/zeebe · GitHub

I would recommend double checking your networking first. Let me know how it goes!

Thank you for your response! I think the problem is not related.

I found the warning message in the source code here: https://github.com/camunda/zeebe/blob/eeadadd132ef0b19ece6226863ac272b1b87c5f0/clients/java/src/main/java/io/camunda/zeebe/client/impl/ZeebeCallCredentials.java#L41

Can you give me information about what exactly is checked and when this warning is actually displayed? I have this message only in the connectors-bundle container.

Hi @lukas-beumer,

I’m not 100% sure, but I think it’s a warning that you send credentials like an authentication token with confidential user information in a plain http request.

On the wire, you can see the values of the token (They are usually Base64 encoded).

You should send those values over an encrypted wire like a https connection.

Hope this helps, Ingo

Hi @Ingo_Richtsmeier,
the problem is, that this is basically a internal kubernetes connection between the connectors and the operate component. So it’s possible to add a tls encryption, but anyway it shouldn’t flood the whole log file with a warning every second. Is there a way to disable it?

Hey @lukas-beumer - I’ve been doing some digging on this, and Ingo is definitely correct: messages are being sent over HTTP rather than HTTPS, which means that it’s possible to sniff the token and perform a man-in-the-middle attack. However, as you rightly pointed out, this communication is happening within your cluster, which greatly minimizes that risk.

One immediate workaround is to adjust the log levels. I’ve asked around here at Camunda to see what else I can find, and whether there’s a way to disable just this one warning or not. I’ll let you know what I learn!

Hi @lukas-beumer and @nathan.loding,

looking at the implementation Lukas posted above, the log configuration

logging.level.io.camunda.zeebe.client.impl.ZeebeCallCredentials=ERROR

should have the smallest impact on other log entries.

Hope this helps, Ingo

Thanks @Ingo_Richtsmeier for your fast response!

Unfortunately, this probably does not work as desired. In this case, the flooding of the logs only affects “camunda/connectors” containers. Here, however, the same component is probably used.

I can try it on the container only with an environment variable, which I have adapted to the pattern as follows:

LOGGING_LEVEL_IO_CAMUNDA_ZEEBE_CLIENT_IMPL_ZEEBE_CALL_CREDENTIALS: "ERROR"

I don’t see an error there for now, or do you see anything else that could be causing it?

Hi @lukas-beumer,

I don’t have an environment at hand where I can reproduce the original issue.

So would have done it the same way as you supposed:

  • Set the environment variable for the connector container
  • Check if the warning is suppressed

In my docker compose, environment variables are set with VAR=value.

And for the variable itself you could try LOGGING_LEVEL_IO_CAMUNDA_ZEEBE_CLIENT_IMPL_ZEEBECALLCREDENTIALS

I think camel case is not used for environment variables.

Hope this helps, Ingo

Unfortunately this didn’t work for me; just in case, I tried a few different formats of the variable name and none seemed to affect the output. I’m going to keep looking and asking around! :grin:

LOGGING_LEVEL_IO_CAMUNDA_ZEEBE_CLIENT_IMPL_ZEEBECALLCREDENTIALS=ERROR
LOGGING_LEVEL_IO_CAMUNDA_ZEEBE_CLIENT_IMPL_ZEEBECALLCREDENTIALS="ERROR"
LOGGING_LEVEL_IO_CAMUNDA_ZEEBE_CLIENT_IMPL_ZEEBE_CALL_CREDENTIALS=ERROR
LOGGING_LEVEL_IO_CAMUNDA_ZEEBE_CLIENT_IMPL_ZEEBE_CALL_CREDENTIALS="ERROR"

I spoke too soon - I think I found the correct configuration. @lukas-beumer, try adding this environment variable to your Connector container configuration:

- "JAVA_TOOL_OPTIONS=-Dlogging.level.io.camunda.zeebe.client.impl.ZeebeCallCredentials=ERROR"

Let me know if that works!

Works like a charm! In the connectors bundle container there’s a start.sh loading some values. Seems JAVA_OPTS must be used, but your parameter is working.

connectors@camunda-platform-connectors-5b8bd88f77-5r9fw:/$ cat start.sh
#!/bin/bash

JAVA_OPTS="${JAVA_OPTS}"

# Explicitly set trust store location
if [[ -n "${JAVAX_NET_SSL_TRUSTSTORE}" ]]; then
  JAVA_OPTS="${JAVA_OPTS} -Djavax.net.ssl.trustStore=${JAVAX_NET_SSL_TRUSTSTORE}"
fi

# Explicitly set trust store password
if [[ -n "${JAVAX_NET_SSL_TRUSTSTOREPASSWORD}" ]]; then
  JAVA_OPTS="${JAVA_OPTS} -Djavax.net.ssl.trustStorePassword=${JAVAX_NET_SSL_TRUSTSTOREPASSWORD}"
fi

if [[ -n ${DEBUG_JVM_PRINT_JAVA_OPTS} ]]; then
  echo "Applied JVM options: $JAVA_OPTS"
fi

exec java ${JAVA_OPTS} -cp "/opt/app/*:/opt/custom/*" "io.camunda.connector.runtime.app.ConnectorRuntimeApplication"
connectors@camunda-platform-connectors-5b8bd88f77-5r9fw:/$ ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
connect+       1 21.3  0.8 6030604 279104 ?      Ssl  05:49   0:27 java -Dlogging.level.io.camunda.zeebe.client.impl.ZeebeCallCredentials=ERROR -Djavax.net.ssl.trustStore=/cert/keystore.jks -cp /opt/app/*:/opt/custom/* io.camunda.connector.runtime.app.ConnectorRuntimeApplication
spec:
  containers:
  - env:
    - name: JAVA_OPTS
      value: -Dlogging.level.io.camunda.zeebe.client.impl.ZeebeCallCredentials=ERROR

Thank you @nathan.loding and @Ingo_Richtsmeier

1 Like

Glad to hear it @lukas-beumer! It was working in my local environment when I used JAVA_TOOL_OPTIONS but I agree that it’s confusing. I’ll work with the engineering team to get some better clarity around this option and how best to use it.

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.