Adding Self Signed SSL Certificate to the Java Truststore for the connection between Elasticsearch and the Zeebe Tasklist doesn't work

Hi all!
I’m running Zeebe deployed with a Helm Chart on 3 Linux nodes with following version info for Kubernetes:
version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.9+vmware.1", GitTreeState:"clean", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

I’ve created a Kubernetes secret containing the *.jks file which I created with the needed elasticsearch.crt file. That secret has been mounted to a volume in the deployment of the Tasklist like this:

volumeMounts:
	- name: truststore
	mountPath: "/etc/truststore"
	readOnly: true
volumes:
	- name: truststore
	  secret:
		secretName: elastic-certificates

Additionally the path and password to the truststore have been configured inside the JAVA_OPTS like this:

env:
	- name: JAVA_OPTS
	  value: >
	    -Djavax.net.ssl.trustStore=/etc/truststore/foo.jks
        -Djavax.net.ssl.trustStorePassword=password
        -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2

When I do the deployment the Pod has following error when starting up:

2021-09-14 09:34:34.606  INFO 1 --- [           main] i.c.t.Application                        : Starting Application using Java 11.0.11 on zeebe-tasklist-helm-7dfc5bdbc8-nlzqf with PID 1 (/app/classes started by root in /)
2021-09-14 09:34:34.614  INFO 1 --- [           main] i.c.t.Application                        : The following profiles are active: dev,dev-data,auth
2021-09-14 09:34:39.324  INFO 1 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat initialized with port(s): 8080 (http)
2021-09-14 09:34:39.397  INFO 1 --- [           main] o.a.c.c.StandardService                  : Starting service [Tomcat]
2021-09-14 09:34:39.398  INFO 1 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet engine: [Apache Tomcat/9.0.48]
2021-09-14 09:34:39.692  INFO 1 --- [           main] o.a.c.c.C.[.[.[/tasklist]                : Initializing Spring embedded WebApplicationContext
2021-09-14 09:34:39.693  INFO 1 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4977 ms
2021-09-14 09:34:41.419 ERROR 1 --- [           main] i.c.t.e.ElasticsearchConnector           : Error occurred while connecting to Elasticsearch: clustername [elasticsearch], elasticsearch-master:9200. Will be retried (0/50) ...

javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:876) ~[elasticsearch-rest-client-7.13.2.jar:7.13.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283) ~[elasticsearch-rest-client-7.13.2.jar:7.13.2]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.13.2.jar:7.13.2]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654) ~[elasticsearch-rest-high-level-client-7.13.2.jar:7.13.2]

Interestingly the configuration as described above works for the Zeebe Operate and I can’t seem to figure out where the problem lies for the Tasklist.

Have any of you bumped into a similar issue or have an idea how to try to fix it?

Hi @dav11d,

For Tasklist, it is something like this:

  1. Setup password for elasticsearch
  • set xpack.security.enabled: true in elasticsearch.yml
  1. Change password with ./bin/elasticsearch-setup-passwords interactive https://localhost:9200
  2. Enable ssl and set custom certificate file path in elasticsearch.yml
  • copy the certificate into Elasticsearch config folder certs
  • Enable ssl and set certificate path
xpack.security.http.ssl.enabled: true 
xpack.security.http.ssl.keystore.path: certs/elastic-stack-ca.p12
  1. Start Elasticsearch and test manually with browser
  • There should come up a username/password dialog
  1. Can also be tested by setting an HTTPS URL and username/password Elasticsearch configuration and start the application
  2. Maybe set camunda.tasklist.elasticsearch.ssl.verifyHostname to false

These are the configuration instructions used for testing the feature internally. Hopefully from this you can figure it out.

@felix.mueller is the Product Owner of Tasklist, and he is across this now. We are going to add better configuration instructions to the official documentation. Any feedback you can give on these instructions is much appreciated!

Josh

1 Like

Hi @jwulf,
Thanks for your response! I’ve went through the steps that you’ve described and the elasticsearch SSL integration worked without a problem.

For troubleshooting I forwarded the port 9200 to localhost and tested it manually with a browser. There I was able to see that the SSL certificate is also present. The SSL certificate on the site is also the same certificate that is present in the .jks file which is used for the connection between the Tasklist and elasticsearch.

Sadly, when I try to start the Tasklist, I still get the same error concerning the fact that it can’t find a valid certification. The Operate uses the same .jks file for the connection to elastisearch and there it works without a problem. Operate and Tasklist follow the same setup at the moment but interestingly enough the SSL certificate validation only works for the Operate. Do you have further configuration instructions concerning the connection between the Tasklist and elasticsearch, which focuses on the Java Truststore?

I’ll also add the application.yml of the Tasklist, as perhaps it can provide you further information.

# Tasklist configuration file

camunda.tasklist:
  # Set Tasklist username and password.
  # If user with <username> does not exists it will be created.
  # Default: demo/demo
  #username:
  #password:
  # ELS instance to store Tasklist data
  elasticsearch:
    # Cluster name
    clusterName: elasticsearch
    # Url
    url: https://elasticsearch-master:9200
    # Host
    # host: elasticsearch-master
    # Transport port
    # port: 9200
    ssl:
      verifyHostname: false
  # Zeebe instance
  zeebe:
    # Broker contact point
    brokerContactPoint: blengine-zeebe-gateway:26500
  # ELS instance to export Zeebe data to
  zeebeElasticsearch:
    # Cluster name
    clusterName: elasticsearch
    # Url
    url: https://elasticsearch-master:9200
    # Host
    # host: elasticsearch-master
    # Transport port
    # port: 9200
    # Index prefix, configured in Zeebe Elasticsearch exporter
    prefix: zeebe-record
    ssl:
      verifyHostname: false
#Spring Boot Actuator endpoints to be exposed
management.endpoints.web.exposure.include: health,info,conditions,configprops,prometheus
# Enable or disable metrics
management.metrics.export.prometheus.enabled: true
# Change the root path of the application
server:
  servlet:
    context-path: /tasklist

David

1 Like

Update:
I was able to solve my problem by adjusting my environment variable. For some reason the Truststore can not be found when it’s set like this in the deployment.yaml:

env:
	- name: JAVA_OPTS
	  value: >
	    -Djavax.net.ssl.trustStore=/etc/truststore/foo.jks
        -Djavax.net.ssl.trustStorePassword=password
        -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2

Instead, when setting the JAVA_TOOL_OPTIONS value the Truststore is found. Example:

env:
	- name: JAVA_TOOL_OPTIONS
	  value: >
	    -Djavax.net.ssl.trustStore=/etc/truststore/foo.jks
        -Djavax.net.ssl.trustStorePassword=password
        -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2