Optimize not able to connect to Elasticsearch instance

Hello there,
I am doing a POC with Camunda platform 7.16.0-ee and Optimize version 3.7.2.
I have Camunda-Platform 7 running and have created a docker-compose to start ElasticSearch and Optimize:3.7.2 locally.

docker-compose as follows

services:
    elasticsearch: # https://hub.docker.com/_/elasticsearch
        image: elasticsearch:d38b85f
        #docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION:-7.17.0}
        container_name: elasticsearch
        ports:
          - "9200:9200"
          - "9300:9300"
        environment:
          - bootstrap.memory_lock=true
          - discovery.type=single-node
          # allow running with low disk space
          - cluster.routing.allocation.disk.threshold_enabled=false
          - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        ulimits:
          memlock:
            soft: -1
            hard: -1
        restart: always
        healthcheck:
          test: [ "CMD-SHELL", "curl -f http://localhost:9200/_cat/health | grep -q green" ]
          interval: 30s
          timeout: 5s
          retries: 3
        volumes:
          - elastic:/usr/share/elasticsearch/data
        networks:
          - camunda-platform
          
    optimize:
        image: camunda/optimize:3.7.2
        container_name: optimize
        ports:
          - "8090:8090"
        environment:
           - "OPTIMIZE_JAVA_OPTS=-Xms1024m -Xmx2048m"
           - ABSOLUTE_PATH_ON_HOST_TO_CONFIGURATION_FILE=/data/environment-config.yaml:ro
           - ABSOLUTE_PATH_ON_HOST_TO_LICENSE_FILE=/data/camunda-license.txt:ro
        restart: on-failure   
        volumes:
          - C:/CAMS/optimize:/data
        networks:
          - camunda-platform
        depends_on:
          - elasticsearch             

volumes:
  elastic:

networks:
  # Note there are two bridge networks: One for Camunda Platform and one for Identity.
  # Operate, Tasklist, and Optimize use both
  camunda-platform:

ElasticSearch instance is running fine and I could access the api(curl -k -X GET -u admin:admin https://elasticsearch:9200/_cat/nodes) from Optimize container using curl.
Optimize itself is failing to start saying “Failed getting number of cluster nodes” and Connection Refused exception.
Logs from Optimize container

Starting Camunda Optimize 3.7.2...


21:39:02.672 [main] INFO o.c.o.s.e.OptimizeElasticsearchClientFactory - Initializing Elasticsearch rest client...

21:39:05.117 [main] ERROR o.c.o.s.e.OptimizeElasticsearchClientFactory - Failed getting number of cluster nodes.

java.net.ConnectException: Connection refused

at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:865)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:275)

at org.elasticsearch.client.RestClient.performRequest(RestClient.java:262)

at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1628)

at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1598)

at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1568)

at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:130)

at org.camunda.optimize.service.es.OptimizeElasticsearchClientFactory.getNumberOfClusterNodes(OptimizeElasticsearchClientFactory.java:81)

at org.camunda.optimize.service.es.OptimizeElasticsearchClientFactory.waitForElasticsearch(OptimizeElasticsearchClientFactory.java:59)

at org.camunda.optimize.service.es.OptimizeElasticsearchClientFactory.create(OptimizeElasticsearchClientFactory.java:41)

at org.camunda.optimize.service.es.OptimizeElasticsearchClientConfiguration.createOptimizeElasticsearchClient(OptimizeElasticsearchClientConfiguration.java:35)

at org.camunda.optimize.service.es.OptimizeElasticsearchClientConfiguration.optimizeElasticsearchClient(OptimizeElasticsearchClientConfiguration.java:30)

at org.camunda.optimize.service.es.OptimizeElasticsearchClientConfiguration$$EnhancerBySpringCGLIB$$82aaf979.CGLIB$optimizeElasticsearchClient$0(<generated>)

at org.camunda.optimize.service.es.OptimizeElasticsearchClientConfiguration$$EnhancerBySpringCGLIB$$82aaf979$$FastClassBySpringCGLIB$$ec11f14.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)

at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)

at org.camunda.optimize.service.es.OptimizeElasticsearchClientConfiguration$$EnhancerBySpringCGLIB$$82aaf979.optimizeElasticsearchClient(<generated>)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)

at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)

at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1352)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1195)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)

at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)

at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)

at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)

at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)

at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)

at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)

at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1389)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1309)

at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)

at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)

at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)

at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:953)

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)

at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:401)

at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:292)

at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)

at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:1067)

at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:572)

at org.eclipse.jetty.server.handler.ContextHandler.contextInitialized(ContextHandler.java:996)

at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:746)

at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)

at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:910)

at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)

at org.eclipse.jetty.server.Server.start(Server.java:423)

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110)

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97)

at org.eclipse.jetty.server.Server.doStart(Server.java:387)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)

at org.camunda.optimize.jetty.EmbeddedCamundaOptimize.startOptimize(EmbeddedCamundaOptimize.java:245)

at org.camunda.optimize.Main.main(Main.java:27)

Caused by: java.net.ConnectException: Connection refused

at java.base/sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)

at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:777)

at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174)

at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148)

at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351)

at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221)

at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)

at java.base/java.lang.Thread.run(Thread.java:829)

I have verified my configuration and kind of clueless to proceed further, have attached environment-config.yaml for reference.
I would really appreciate some help.
environment-config.yaml.dmn (13.9 KB)

I don’t think this should matter, but I’d be worried about your volume assignment on the optimize container. C:/CAMS/optimize:/data doesn’t seem right.

But… if that mapping is failing, your config file shouldn’t be able to be found, so opimize should then be using all the default values.

I verified the files are mounted on container and available under /data.
I now enabled trace logs and it seems Optimize is trying to use http://elasticsearch:9200/_cluster/health url and it is not accessible as ElasticSearch is running over https. So the url should have been https://elasticsearch:9200/_cluster/health. I have enabled ssl in Optimize using es.security.ssl.enabled=true but still it is using http port.

I don’t really have much for you then.
@Helene might have more information (based on some of the other threads)

The only other thing I can suggest is the old “Have you tried turning it off and back on again?” … after changing the es.security.ssl.enabled=true, Optimize needs to be restarted…

Thanks for answering. Yes I restarted Optimize few times after enabling ssl. So the issue now is that Optimize is trying to reach ElasticSearch over Http and not Https.

Hi @hrishi_joshi,
ABSOLUTE_PATH_ON_HOST_TO_CONFIGURATION_FILE is not an environment variable, I think this is why Optimize is running with its default config (ssl disabled for ES) instead of using your custom configuration. You can mount your configuration as described in the docs here or use a configmap.
Note that ABSOLUTE_PATH_ON_HOST_TO_CONFIGURATION_FILE should be replaced with the actual path to your config file in the line-v ABSOLUTE_PATH_ON_HOST_TO_CONFIGURATION_FILE:/optimize/config/environment-config.yaml:ro.

Hope that helps!
Helene

Thanks @Helene for reply.
I have now updated my docker-compose to specify environment-config.yaml and license correctly and I am getting errors regarding certificates. I will try to resolve those and keep posted here.

Hi @hrishi_joshi ,
Glad you could mount the config and license files. Can you post the error logs you’re getting now?

I am getting SSL verification exception

Caused by: javax.net.ssl.SSLPeerUnverifiedException: Host name 'elasticsearch' does not match the certificate subject provided by the peer (CN=node-0.example.com, OU=node, O=node, L=test, DC=de)

I had used the certificate generated by Elasticsearch as
certificate_authorities: [’/data/certs/root-ca.pem’].
Can we somehow instruct Optimize not to verify ssl cert?
I know it is not recommended to do that but for time being can we do that and then I will generate valid certs to be used by Optimize and ElasticSearch.

This is not possible, you’d need to fix the certificate to point to the correct host.

Thanks @Helene. I will update the cert and keep you posted. Thanks for your help.

Hi @Helene,
I am able to connect to ElasticSearch instance now. I had to change the host name in docker-compose to match with the CN of client certificate used by ElasticSearch. And update OPTIMIZE_ELASTICSEARCH_HOST=node-0.example.com in Optimize configuration.

elasticsearch: # https://hub.docker.com/_/elasticsearch
        image: dtr.dockeree.oneadr.net:7843/com-nordea-esp/elasticsearch:d38b85f
        #docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION:-7.17.0}
        container_name: **node-0.example.com**
        ports:
          - "9200:9200"
          - "9300:9300"
        environment:
          - bootstrap.memory_lock=true
          - discovery.type=single-node
          - CERTS_DIR=/usr/share/elasticsearch/config/certificates
          # allow running with low disk space
          - cluster.routing.allocation.disk.threshold_enabled=false
          - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
          **#- xpack.security.enabled=true**
          **#- xpack.security.http.ssl.key=$CERTS_DIR/localhost.pem**
          **#- xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/localhost.pem**
        ulimits:
          memlock:
            soft: -1
            hard: -1
        restart: always
        healthcheck:
          test: [ "CMD-SHELL", "curl -f http://localhost:9200/_cat/health | grep -q green" ]
          interval: 30s
          timeout: 5s
          retries: 3
        volumes:
          - elastic:/usr/share/elasticsearch/data
          - ./certs:/usr/share/elasticsearch/config/certificates
        networks:
          - camunda-platform

I was trying to change Elastic certs using xpack variables but ElasticSearch is not recognizing those somehow. I was referring to Camunda 8 docker-compose file and it seems ssl certs are not explicitly configured there so ElasticSearch must be using the certs generated while startup. And ElasticSearch is using hostname as

container_name: elasticsearch

I am wondering why it dosen’t fail there as the certs generated by ElasticSearch dose not contain elasticsearch as CN or SAN.

Hi @hrishi_joshi

So if I am understanding your setup correctly then this does not fail because per default, the Elasticsearch connection does not use ssl.

ElasticSearch does use SSL by default and it generates the certs itself. I had downloaded the root_ca cert from ElasticSearch and configured in Optimize under es configuration.
The issue was that the ElasticSearch client cert has CN=node-0.example.com and I had configured hostname as “elasticsearch” so Optimize was failing to verify the certs from ElasticSearch as the hostname in url was “elasticsearch”(https:// elasticsearch:9200) but SSL cert had CN as node-0.example.com. (Thats why the error javax.net.ssl.SSLPeerUnverifiedException: Host name ‘elasticsearch’ does not match the certificate subject provided by the peer (CN=node-0.example.com, OU=node, O=node, L=test, DC=de))
My question was how does it work for camunda 8 self-managed docker-compose

Hi @hrishi_joshi ,
Are you maybe referring to Elasticsearch 8? The docker compose defaults 7.17 and the default for the connection from Optimize to Elasticsearch is to have ssl disabled, that’s what I was referring to in the previous post. We do not support ES 8 yet.

Hi,
I was not refferring to ElasticSearch 8 but Camunda platform 8 docker-compose.
But I understood that ElasticSearch 7.17 used with Optimize in Camunda 8 runs on HTTP by default and so it dosen’t get SSL errors :slightly_smiling_face:.

I am now getting below error while starting Optimize
ERROR o.c.o.s.i.e.f.instance.TenantFetcher - Error during fetching of entities. Please check the connection with [workflow-engine-qa]!

org.camunda.optimize.service.exceptions.OptimizeRuntimeException: Engine didn't respond. Can not verify this engine's version

I have verified that the connection is there to engine and I could access rest api from container using curl.
I was wondering if I could see the source code of Optimize.

Hi @hrishi_joshi

Apologies for the delayed reply, I was on holiday. I see you’ve already created a second thread for this question, I hope you’ve since been able to resolve the engine connection issue.

Unfortunately, Optimize is not open source so this is not possible.

Hi @Helene,
I think the issue was related to SSL context incorrectly set for Optimize. When I set the keystore and truststore for Optimize using OPTIMIZE_JAVA_OPTS and javax.net.ssl arguments the engine connection worked. It would have been good to have the error logs more clear though. That is why I was asking to check the source code earlier.
Thanks for you help, really appreciate your time.

1 Like

Hi @Helene,

we are facing the same issue when trying to connect to Elastic Cloud, the root CA certificate was added to the truststore and truststore was set by using OPTIMIZE_JAVA_OPTS.
However, still get

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

curl --cacert with the root CA certificate works.

When es.security.ssl.certificate_authorities is set, there is error such as:

Caused by: org.camunda.optimize.service.exceptions.OptimizeRuntimeException: Could not create certificate trustStore for the secured Elasticsearch Connection!
	at org.camunda.optimize.upgrade.es.ElasticsearchHighLevelRestClientBuilder.loadCustomTrustStore(ElasticsearchHighLevelRestClientBuilder.java:192)
	at org.camunda.optimize.upgrade.es.ElasticsearchHighLevelRestClientBuilder.buildHttpsRestClient(ElasticsearchHighLevelRestClientBuilder.java:59)
	... 82 more
Caused by: org.camunda.optimize.service.exceptions.OptimizeConfigurationException: Could not load CA authority certificate for the secured Elasticsearch Connection!
	at org.camunda.optimize.upgrade.es.ElasticsearchHighLevelRestClientBuilder.loadCustomTrustStore(ElasticsearchHighLevelRestClientBuilder.java:185)
	... 83 more
Caused by: java.security.cert.CertificateException: Could not parse certificate: java.io.IOException: Incomplete data
	at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:115)
	at java.base/java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:355)
	at org.camunda.optimize.upgrade.es.ElasticsearchHighLevelRestClientBuilder.loadCertificateFromPath(ElasticsearchHighLevelRestClientBuilder.java:204)
	at org.camunda.optimize.upgrade.es.ElasticsearchHighLevelRestClientBuilder.loadCustomTrustStore(ElasticsearchHighLevelRestClientBuilder.java:180)
	... 83 more
Caused by: java.io.IOException: Incomplete data
	at java.base/sun.security.provider.X509Factory.readOneBlock(X509Factory.java:619)
	at java.base/sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:99)
	... 86 more

How does the ssl certificate processed by Optimize?

Hi @LOuuu ,
This issue would best be addressed with the help of our support team. It actually looks like you already have an ongoing ticket for this but if not please open a new ticket so we can have someone from support look into it.
Thanks,
Helene