Hi all!
I’m running Zeebe deployed with a Helm Chart on 3 Linux nodes with following version info for Kubernetes: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.9+vmware.1", GitTreeState:"clean", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
I’ve created a Kubernetes secret containing the *.jks file which I created with the needed elasticsearch.crt file. That secret has been mounted to a volume in the deployment of the Tasklist like this:
When I do the deployment the Pod has following error when starting up:
2021-09-14 09:34:34.606 INFO 1 --- [ main] i.c.t.Application : Starting Application using Java 11.0.11 on zeebe-tasklist-helm-7dfc5bdbc8-nlzqf with PID 1 (/app/classes started by root in /)
2021-09-14 09:34:34.614 INFO 1 --- [ main] i.c.t.Application : The following profiles are active: dev,dev-data,auth
2021-09-14 09:34:39.324 INFO 1 --- [ main] o.s.b.w.e.t.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2021-09-14 09:34:39.397 INFO 1 --- [ main] o.a.c.c.StandardService : Starting service [Tomcat]
2021-09-14 09:34:39.398 INFO 1 --- [ main] o.a.c.c.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.48]
2021-09-14 09:34:39.692 INFO 1 --- [ main] o.a.c.c.C.[.[.[/tasklist] : Initializing Spring embedded WebApplicationContext
2021-09-14 09:34:39.693 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 4977 ms
2021-09-14 09:34:41.419 ERROR 1 --- [ main] i.c.t.e.ElasticsearchConnector : Error occurred while connecting to Elasticsearch: clustername [elasticsearch], elasticsearch-master:9200. Will be retried (0/50) ...
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:876) ~[elasticsearch-rest-client-7.13.2.jar:7.13.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283) ~[elasticsearch-rest-client-7.13.2.jar:7.13.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270) ~[elasticsearch-rest-client-7.13.2.jar:7.13.2]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1654) ~[elasticsearch-rest-high-level-client-7.13.2.jar:7.13.2]
Interestingly the configuration as described above works for the Zeebe Operate and I can’t seem to figure out where the problem lies for the Tasklist.
Have any of you bumped into a similar issue or have an idea how to try to fix it?
Start Elasticsearch and test manually with browser
There should come up a username/password dialog
Can also be tested by setting an HTTPS URL and username/password Elasticsearch configuration and start the application
Maybe set camunda.tasklist.elasticsearch.ssl.verifyHostname to false
These are the configuration instructions used for testing the feature internally. Hopefully from this you can figure it out.
@felix.mueller is the Product Owner of Tasklist, and he is across this now. We are going to add better configuration instructions to the official documentation. Any feedback you can give on these instructions is much appreciated!
Hi @jwulf,
Thanks for your response! I’ve went through the steps that you’ve described and the elasticsearch SSL integration worked without a problem.
For troubleshooting I forwarded the port 9200 to localhost and tested it manually with a browser. There I was able to see that the SSL certificate is also present. The SSL certificate on the site is also the same certificate that is present in the .jks file which is used for the connection between the Tasklist and elasticsearch.
Sadly, when I try to start the Tasklist, I still get the same error concerning the fact that it can’t find a valid certification. The Operate uses the same .jks file for the connection to elastisearch and there it works without a problem. Operate and Tasklist follow the same setup at the moment but interestingly enough the SSL certificate validation only works for the Operate. Do you have further configuration instructions concerning the connection between the Tasklist and elasticsearch, which focuses on the Java Truststore?
I’ll also add the application.yml of the Tasklist, as perhaps it can provide you further information.
# Tasklist configuration file
camunda.tasklist:
# Set Tasklist username and password.
# If user with <username> does not exists it will be created.
# Default: demo/demo
#username:
#password:
# ELS instance to store Tasklist data
elasticsearch:
# Cluster name
clusterName: elasticsearch
# Url
url: https://elasticsearch-master:9200
# Host
# host: elasticsearch-master
# Transport port
# port: 9200
ssl:
verifyHostname: false
# Zeebe instance
zeebe:
# Broker contact point
brokerContactPoint: blengine-zeebe-gateway:26500
# ELS instance to export Zeebe data to
zeebeElasticsearch:
# Cluster name
clusterName: elasticsearch
# Url
url: https://elasticsearch-master:9200
# Host
# host: elasticsearch-master
# Transport port
# port: 9200
# Index prefix, configured in Zeebe Elasticsearch exporter
prefix: zeebe-record
ssl:
verifyHostname: false
#Spring Boot Actuator endpoints to be exposed
management.endpoints.web.exposure.include: health,info,conditions,configprops,prometheus
# Enable or disable metrics
management.metrics.export.prometheus.enabled: true
# Change the root path of the application
server:
servlet:
context-path: /tasklist
Update:
I was able to solve my problem by adjusting my environment variable. For some reason the Truststore can not be found when it’s set like this in the deployment.yaml: