Not able to authenticate Optimize with an ES cluster with basic auth

krishnakumar797 · April 6, 2023, 2:03pm

Optimize application is throwing error while starting when it is connected to an ES cluster with xpack basic authentication is configured. It fails to connect with ES. I have deployed Optmize using helm charts and so tried with the below enviroment variables.

OPTIMIZE_ELASTICSEARCH_USERNAME
OPTIMIZE_ELASTICSEARCH_PASSWORD

Still it gives the below error

Suppressed: org.elasticsearch.client.ResponseException: method [GET], host [http://elasticsearch-master:9200], URI [/_cluster/health?master_timeout=30s&level=cluster&timeout=30s], status line [HTTP/1.1 401 Unauthorized]
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_cluster/health?master_timeout=30s&level=cluster&timeout=30s]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_cluster/health?master_timeout=30s&level=cluster&timeout=30s]","header":{"WWW-Authenticate":"Basic realm=\"security\" charset=\"UTF-8\""}},"status":401}

Optimize version: 3.9.4
Camunda version: 8.1.9
ES Version: 7.17.9

jonathan.lukas · April 11, 2023, 8:53am

Hello @krishnakumar797 ,

regarding the Elasticsearch security config, there is a section in the docs:

According to this, you would configure your ES connection auth:

ES_SECURITY_USERNAME=abc
ES_SECURITY_PASSWORD=xyz

I hope this helps

Jonathan

krishnakumar797 · April 11, 2023, 9:59am

Hello Jonathan,

Thanks for the reply, I have tried the same env. variables however still I got the authentication error. Please see below.

09:54:55.995 [main] INFO o.c.o.s.e.OptimizeElasticsearchClientFactory - Initializing Elasticsearch rest client…
09:54:56.725 [main] DEBUG o.c.o.u.e.ElasticsearchHighLevelRestClientBuilder - Elasticsearch username and password not provided, skipping connection credential setup.
09:54:58.126 [main] ERROR o.c.o.s.e.OptimizeElasticsearchClientFactory - Can’t connect to any Elasticsearch node [[host=http://elasticsearch-master:9200]]. Please check the connection!
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=security_exception, reason=missing authentication credentials for REST request [/_cluster/health?master_timeout=30s&level=cluster&timeout=30s]]

krishnakumar797 · April 11, 2023, 3:40pm

Hello Jonathan,

Any other way that we can provide the ElasticSearch username and password to Optimize?. In the debug log we can see that since the username and password not provided the Optimize is skipping the connection credential setup.
09:54:56.725 [main] DEBUG o.c.o.u.e.ElasticsearchHighLevelRestClientBuilder - Elasticsearch username and password not provided, skipping connection credential setup.

So the variables we provided is not accepting.

jonathan.lukas · April 11, 2023, 8:07pm

Hello @krishnakumar797 ,

could you try to set any other variable to prove configuration works in general?

jonathan

krishnakumar797 · April 11, 2023, 8:14pm

Yes @jonathan.lukas , the following env. variable is working which provides the ES host name as below
OPTIMIZE_ELASTICSEARCH_HOST
OPTIMIZE_ELASTICSEARCH_HTTP_PORT
these works well.

Please see this file

github.com

camunda/camunda-platform-helm/blob/main/charts/camunda-platform/charts/optimize/templates/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ include "optimize.fullname" . }}
  labels: {{- include "optimize.labels" . | nindent 4 }}
  annotations: {{- toYaml  .Values.global.annotations | nindent 4 }}
spec:
  replicas: 1
  selector:
    matchLabels:
      {{- include "optimize.matchLabels" . | nindent 6 }}
  template:
    metadata:
      labels: 
        {{- include "optimize.labels" . | nindent 8 }}
        {{- if .Values.podLabels }}
        {{- toYaml .Values.podLabels | nindent 8 }}
        {{- end }}
      {{- if .Values.podAnnotations }}
      annotations:

This file has been truncated. show original

krishnakumar797 · April 13, 2023, 6:12am

Hello @jonathan.lukas , Have you got a chance to look into this?

jonathan.lukas · April 13, 2023, 9:35am

Hello @krishnakumar797 ,

I was wrong… the way you can provide user data to Optimize only via environment-config.yaml which has to be mounted.

Here is the docs about it:

Jonathan

Helene · April 13, 2023, 11:44am

Hi @krishnakumar797,

Additionally to what Jonathan has already pointed out please also note that there are the following environment variable you can use (these are just so far undocumented):

CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_USERNAME and
CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_PASSWORD

krishnakumar797 · April 14, 2023, 12:22pm

Thanks @Helene,

This ENV variables worked and the Optimize was able to connect with ES and created indexes. However now it showing connection refused error to the ZeebeEngine.

12:15:27.198 [PlatformUserIdentityCache-1] ERROR o.c.o.rest.engine.EngineContext - Failed to validate engine camunda-bpm version with error message: Engine didn't respond. Can not verify this engine's version
org.camunda.optimize.service.exceptions.OptimizeRuntimeException: Engine didn't respond. Can not verify this engine's version

krishnakumar797 · April 14, 2023, 12:25pm

thanks @jonathan.lukas , I was able to connect with the ENV. variables provided in the above answer. But however after connecting to the ES, now the Optimize was not able to connect with Zeebe Engine. There is no auth configured for the Zeebe engine. I m getting the below error from Optmize now.

12:15:27.198 [PlatformUserIdentityCache-1] ERROR o.c.o.rest.engine.EngineContext - Failed to validate engine camunda-bpm version with error message: Engine didn't respond. Can not verify this engine's version
org.camunda.optimize.service.exceptions.OptimizeRuntimeException: Engine didn't respond. Can not verify this engine's version

In between I had also tried to provide the Zeebe engine URL as below
"OPTIMIZE_CAMUNDABPM_REST_URL": "http://camunda-zeebe.camunda:26502"
Still not working.

krishnakumar797 · April 14, 2023, 12:50pm

I had also tried the following env. variables

"OPTIMIZE_CAMUNDABPM_REST_URL":"http://camunda-zeebe.camunda:26501/engine-rest"
"OPTIMIZE_CAMUNDABPM_WEBAPPS_URL":"http://camunda-zeebe.camunda:26501/camunda"

Please see the log below

12:41:15.427 [main] INFO o.c.o.s.i.PlatformUserIdentityCache - Initializing platform user identity sync.
12:41:15.500 [main] INFO o.c.o.s.i.PlatformUserIdentityCache - Scheduling platform user identity sync
12:41:17.307 [PlatformUserIdentityCache-1] ERROR o.c.o.s.util.EngineVersionChecker - Engine didn’t respond. Can not verify this engine’s version
javax.ws.rs.ProcessingException: java.net.SocketException: Unexpected end of file from server
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:269)
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297)
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:662)
at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:697)
at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:691)
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
at org.glassfish.jersey.internal.Errors.process(Errors.java:205)
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390)
at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:691)
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:661)
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:413)
at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:313)
at org.camunda.optimize.service.util.EngineVersionChecker.checkEngineVersionSupport(EngineVersionChecker.java:49)
at org.camunda.optimize.rest.engine.EngineContext.getEngineClient(EngineContext.java:104)
at org.camunda.optimize.rest.engine.EngineContext.getAuthorizationsForType(EngineContext.java:681)
at org.camunda.optimize.rest.engine.EngineContext.getAllApplicationAuthorizations(EngineContext.java:502)
at org.camunda.optimize.rest.engine.EngineContext.getApplicationAuthorizedIdentities(EngineContext.java:131)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.populateAllAuthorizedIdentitiesForEngineToCache(PlatformUserIdentityCache.java:55)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.lambda$populateCache$0(PlatformUserIdentityCache.java:50)
at java.base/java.util.HashMap$Values.forEach(HashMap.java:977)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.populateCache(PlatformUserIdentityCache.java:50)
at org.camunda.optimize.service.identity.AbstractIdentityCache.synchronizeIdentities(AbstractIdentityCache.java:148)
at org.camunda.optimize.service.identity.AbstractIdentityCache.syncIdentitiesWithRetry(AbstractIdentityCache.java:114)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.SocketException: Unexpected end of file from server
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:899)
at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:722)
at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:896)
at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:722)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:378)
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:267)
… 30 common frames omitted
12:41:17.307 [PlatformUserIdentityCache-1] ERROR o.c.o.rest.engine.EngineContext - Failed to validate engine camunda-bpm version with error message: Engine didn’t respond. Can not verify this engine’s version
org.camunda.optimize.service.exceptions.OptimizeRuntimeException: Engine didn’t respond. Can not verify this engine’s version
at org.camunda.optimize.service.util.EngineVersionChecker.checkEngineVersionSupport(EngineVersionChecker.java:52)
at org.camunda.optimize.rest.engine.EngineContext.getEngineClient(EngineContext.java:104)
at org.camunda.optimize.rest.engine.EngineContext.getAuthorizationsForType(EngineContext.java:681)
at org.camunda.optimize.rest.engine.EngineContext.getAllApplicationAuthorizations(EngineContext.java:502)
at org.camunda.optimize.rest.engine.EngineContext.getApplicationAuthorizedIdentities(EngineContext.java:131)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.populateAllAuthorizedIdentitiesForEngineToCache(PlatformUserIdentityCache.java:55)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.lambda$populateCache$0(PlatformUserIdentityCache.java:50)
at java.base/java.util.HashMap$Values.forEach(HashMap.java:977)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.populateCache(PlatformUserIdentityCache.java:50)
at org.camunda.optimize.service.identity.AbstractIdentityCache.synchronizeIdentities(AbstractIdentityCache.java:148)
at org.camunda.optimize.service.identity.AbstractIdentityCache.syncIdentitiesWithRetry(AbstractIdentityCache.java:114)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
12:41:17.307 [PlatformUserIdentityCache-1] ERROR o.c.o.rest.engine.EngineContext - Could not fetch application authorizations from the Engine with alias [camunda-bpm] to check the access permissions.
org.camunda.optimize.service.exceptions.OptimizeRuntimeException: Engine didn’t respond. Can not verify this engine’s version
at org.camunda.optimize.service.util.EngineVersionChecker.checkEngineVersionSupport(EngineVersionChecker.java:52)
at org.camunda.optimize.rest.engine.EngineContext.getEngineClient(EngineContext.java:104)
at org.camunda.optimize.rest.engine.EngineContext.getAuthorizationsForType(EngineContext.java:681)
at org.camunda.optimize.rest.engine.EngineContext.getAllApplicationAuthorizations(EngineContext.java:502)
at org.camunda.optimize.rest.engine.EngineContext.getApplicationAuthorizedIdentities(EngineContext.java:131)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.populateAllAuthorizedIdentitiesForEngineToCache(PlatformUserIdentityCache.java:55)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.lambda$populateCache$0(PlatformUserIdentityCache.java:50)
at java.base/java.util.HashMap$Values.forEach(HashMap.java:977)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.populateCache(PlatformUserIdentityCache.java:50)
at org.camunda.optimize.service.identity.AbstractIdentityCache.synchronizeIdentities(AbstractIdentityCache.java:148)
at org.camunda.optimize.service.identity.AbstractIdentityCache.syncIdentitiesWithRetry(AbstractIdentityCache.java:114)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
12:41:17.308 [PlatformUserIdentityCache-1] ERROR o.c.o.s.i.PlatformUserIdentityCache - Could not sync platform user identities. Will stop retrying as next scheduled sync is approaching
org.camunda.optimize.service.exceptions.OptimizeRuntimeException: Could not fetch application authorizations from the Engine with alias [camunda-bpm] to check the access permissions.
at org.camunda.optimize.rest.engine.EngineContext.getAllApplicationAuthorizations(EngineContext.java:509)
at org.camunda.optimize.rest.engine.EngineContext.getApplicationAuthorizedIdentities(EngineContext.java:131)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.populateAllAuthorizedIdentitiesForEngineToCache(PlatformUserIdentityCache.java:55)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.lambda$populateCache$0(PlatformUserIdentityCache.java:50)
at java.base/java.util.HashMap$Values.forEach(HashMap.java:977)
at org.camunda.optimize.service.identity.PlatformUserIdentityCache.populateCache(PlatformUserIdentityCache.java:50)
at org.camunda.optimize.service.identity.AbstractIdentityCache.synchronizeIdentities(AbstractIdentityCache.java:148)
at org.camunda.optimize.service.identity.AbstractIdentityCache.syncIdentitiesWithRetry(AbstractIdentityCache.java:114)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
12:41:18.517 [main] INFO o.c.o.s.i.PlatformUserTaskIdentityCache - Initializing platform assignee/candidateGroup identity sync.
12:41:18.518 [main] INFO o.c.o.s.i.PlatformUserTaskIdentityCache - Scheduling platform assignee/candidateGroup identity sync
12:41:18.814 [PlatformUserTaskIdentityCache-1] INFO o.c.o.s.i.PlatformUserTaskIdentityCache - platform assignee/candidateGroup identity sync complete
12:41:19.817 [main] INFO o.c.o.s.i.i.m.ExternalVariableUpdateEngineImportMediator - Engine ‘optimize’ could not be found in the configuration
12:41:23.723 [main] INFO o.c.o.s.i.ImportSchedulerManagerService - Import was disabled by config for import source IngestedDataSourceDto(super=DataSourceDto(type=ingested, name=null)).
12:41:23.724 [main] INFO o.c.o.s.i.AbstractImportScheduler - Start scheduling import from EngineDataSourceDto(super=DataSourceDto(type=engine, name=camunda-bpm)).
12:41:23.725 [main] INFO o.c.o.s.i.AbstractImportScheduler - Start scheduling import from ZeebeConfigDto(name=zeebe-record, partitionCount=2).
12:41:23.811 [EngineImportScheduler-1] ERROR o.c.o.s.util.EngineVersionChecker - Engine didn’t respond. Can not verify this engine’s version
javax.ws.rs.ProcessingException: java.net.SocketException: Unexpected end of file from server

jonathan.lukas · April 14, 2023, 1:27pm

Hello @krishnakumar797 ,

when connecting Optimize to Zeebe, you need to specify other env variables than for Camunda 7.

As Optimize loads the zeebe data directly from the Elasticsearch export by Zeebe, it needs to know where to search for the data (the assumption is that Zeebe exports to the same Elasticsearch as Optimize):

I hope this helps

Jonathan

chandresh.mishra · November 28, 2023, 1:22pm

Hi krishna,
I am getting the same error. Were you able to solve this ?