I am running camunda 8.8 docker on linux, and my docker-compose.yaml is as attached below.
I am using external open id connect for authentication.
the external identity provider requires ssl, added the certificate to custom java trust store and mounted the trust store in the docker.
Docker is starting healthy, but when trying to access the identity component, it produces javax.net.ssl.SSLHandshakeException.
the docker-compose is attached.
docker-compose-camunda-forum.yaml (8.9 KB)
I added the ssl config to all components, but now I am getting a different error in orchestration:
orchestration | io.camunda.operate.util.RetryOperation - Retry Operation Connect to Elasticsearch cluster [elasticsearch] at ``http://localhost:9200`` failed: Connection refused
orchestration | java.net.ConnectException: Connection refused
orchestration | at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:934)
orchestration | at org.elasticsearch.client.RestClient.performRequest(RestClient.java:304)
orchestration | at org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
orchestration | at co.elastic.clients.transport.rest_client.RestClientHttpClient.performRequest(RestClientHttpClient.java:92)
orchestration | at co.elastic.clients.transport.ElasticsearchTransportBase.performRequest(ElasticsearchTransportBase.java:145)
orchestration | at co.elastic.clients.elasticsearch.cluster.ElasticsearchClusterClient.health(ElasticsearchClusterClient.java:439)
orchestration | at io.camunda.operate.connect.ElasticsearchConnector.lambda$checkHealth$2(ElasticsearchConnector.java:157)
orchestration | at io.camunda.operate.util.RetryOperation.retry(RetryOperation.java:59)
orchestration | at io.camunda.operate.connect.ElasticsearchConnector.checkHealth(ElasticsearchConnector.java:162)
orchestration | at io.camunda.operate.connect.ElasticsearchConnector.elasticsearchClient(ElasticsearchConnector.java:131)
orchestration | at io.camunda.operate.connect.ElasticsearchConnector$$SpringCGLIB$$0.CGLIB$elasticsearchClient$1()
orchestration | at io.camunda.operate.connect.ElasticsearchConnector$$SpringCGLIB$$FastClass$$1.invoke()
docker-compose-camunda-forum.yaml (9.4 KB)
Hi @devmsaleh
The CamundaExporter configuration is missing so it defaulted to localhost:9200
In the official Camunda compose file, they rely on the application.yaml
Try add the following
exporters:
elasticsearch:
className: "io.camunda.zeebe.exporter.ElasticsearchExporter"
args:
url: "http://elasticsearch:9200"
# Default is 1000, reduced to 1 for faster local development feedback
bulk:
size: 1
index:
prefix: "zeebe-record"
CamundaExporter:
className: "io.camunda.exporter.CamundaExporter"
args:
connect:
type: elasticsearch
url: "http://elasticsearch:9200"
createSchema: true
I found out the issue was indentation in the configs section in the bottom, after fixing it now I am getting javax.net.ssl.SSLHandshakeException in the connectors, below is my corrected docker file.
docker-compose-camunda-forum.yaml (9.6 KB)
Please ensure that you followed the following recommendation for client ID and username claim configuration
the issue I am facing right now is javax.net.ssl.SSLHandshakeException in the connectors despite the correct configuration are in place
I was having a duplicate connector configuration, after correcting the docker file now I am getting a differenet error in connectors:
Caused by: java.lang.IllegalArgumentException: java.lang.NullPointerException: Expected valid client id but none was provided.
connectors | at io.camunda.client.impl.oauth.OAuthCredentialsProviderBuilder.validate(OAuthCredentialsProviderBuilder.java:515)
connectors | at io.camunda.client.impl.oauth.OAuthCredentialsProviderBuilder.build(OAuthCredentialsProviderBuilder.java:376)
connectors | at io.camunda.client.spring.configuration.CredentialsProviderConfiguration.buildOAuthCredentialsProvider(CredentialsProviderConfiguration.java:104)
connectors | at io.camunda.client.spring.configuration.CredentialsProviderConfiguration.camundaClientCredentialsProvider(CredentialsProviderConfiguration.java:48)
connectors | at io.camunda.client.spring.configuration.CredentialsProviderConfiguration$$SpringCGLIB$$0.CGLIB$camundaClientCredentialsProvider$0()
connectors | at io.camunda.client.spring.configuration.CredentialsProviderConfiguration$$SpringCGLIB$$FastClass$$1.invoke()
connectors | at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258)
connectors | at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:400)
connectors | at io.camunda.client.spring.configuration.CredentialsProviderConfiguration$$SpringCGLIB$$0.camundaClientCredentialsProvider()
connectors | at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(Unknown Source)
connectors | at java.base/java.lang.reflect.Method.invoke(Unknown Source)
connectors | at org.springframework.beans.factory.support.SimpleInstantiationStrategy.lambda$instantiate$0(SimpleInstantiationStrategy.java:172)
connectors | … 153 common frames omitted
docker-compose-camunda-forum.yaml (9.6 KB)
You have both a config mount AND a volume mount
configs:
- source: connectors-config
target: application.yaml
volumes:
- "./.connectors/application.yaml:/app/application.yaml"
So use only one approach and ensure it points to the right configs as you are overriding spring behavior to only look at /app/application.yaml.
environment:
SPRING_CONFIG_LOCATION: /app/application.yaml
docker-compose-camunda-forum.yaml (8.9 KB)
when using the connectors config under environment only, now I get the error:
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
while the custom truststore is added under the connector environment JAVA_TOOL_OPTIONS and I verified that the customTrustStore exists inside the connectos image using:
docker exec -it connectors sh -lc ‘ls -l /opt/certs && ls -l /opt/certs/customTrustStore.jks’
also when Run Java with “show settings” inside the container:
docker exec -it connectors sh -lc ‘java -XshowSettings:properties -version 2>&1 | grep -E “trustStore|trustStorePassword|trustStoreType”’
it prints the correct customTrustStore setted in java opts as follows:
Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=/opt/certs/customTrustStore.jks -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=JKS
javax.net.ssl.trustStore = /opt/certs/customTrustStore.jks
javax.net.ssl.trustStorePassword = changeit
javax.net.ssl.trustStoreType = JKS
Update (attached connectors startup logs).
camunda-connectors-logs.yaml (3.6 KB)
Update: I tried copying the customTrustStore from different server, which is working fine on c8run, but this also gives same error.
I found out that the issue was related to linux file permssions on the customTrustStore.jks file, and the file was exisiting inside the docker image and has the correct certificate, the key to that was executing the following commands to detect the file permissions:
sudo docker exec -it connectors bash
head -c 1 /opt/certs/customTrustStore.jks >/dev/null && echo readable || echo not_readable
my final docker is attached.
services:
orchestration:
image: camunda/camunda:${CAMUNDA_VERSION}
container_name: orchestration
ports:
- "26500:26500"
- "9600:9600"
- "8088:8080"
environment:
CAMUNDA_DATABASE_INDEX_NUMBER_OF_REPLICAS: 0
CAMUNDA_IDENTITY_BASEURL: http://identity:8084
CAMUNDA_SECURITY_AUTHORIZATIONS_ENABLED: true
CAMUNDA_SECURITY_AUTHENTICATION_METHOD: oidc
CAMUNDA_SECURITY_AUTHENTICATION_UNPROTECTEDAPI: false
CAMUNDA_SECURITY_INITIALIZATION_DEFAULTROLES_ADMIN_USERS_0: myAdminUser
CAMUNDA_SECURITY_INITIALIZATION_DEFAULTROLES_ADMIN_GROUPS_0: myAdminGroup
CAMUNDA_SECURITY_INITIALIZATION_DEFAULTROLES_ADMIN_CLIENTS_0: zeepi-api-client
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_CLIENTID: camunda-web-client
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_CLIENTSECRET: camunda-web-client-secret
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_ISSUERURI: https://my-identityserver:8055
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_REDIRECTURI: http://mycamundaexample.com/sso-callback
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_AUDIENCES: camunda-api-audience,zeebe-api-audience,camunda-web-client
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_SCOPE: openid,profile,offline_access,camunda-api-scope
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_USERNAMECLAIM: preferred_username
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_CLIENTIDCLAIM: client_id
CAMUNDA_SECURITY_AUTHENTICATION_OIDC_GROUPSCLAIM: role
JAVA_TOOL_OPTIONS: "-Xms512m -Xmx512m -Djavax.net.ssl.trustStore=/opt/certs/customTrustStore.jks -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=JKS"
env_file:
- path: .env
required: true
restart: unless-stopped
healthcheck:
test:
[
"CMD-SHELL",
"timeout 60s bash -c ':> /dev/tcp/127.0.0.1/9600' || exit 1",
]
interval: 30s
timeout: 60s
retries: 5
start_period: 30s
volumes:
- orchestration:/usr/local/zeebe/data
- "./.orchestration/application.yaml:/usr/local/camunda/config/application.yaml"
- "/app/camunda/customTrustStore.jks:/opt/certs/customTrustStore.jks:ro"
networks:
- camunda-platform
extra_hosts:
- "host.docker.internal:host-gateway"
depends_on:
elasticsearch:
condition: service_healthy
connectors: # https://docs.camunda.io/docs/self-managed/connectors-deployment/connectors-configuration/
image: camunda/connectors-bundle:${CAMUNDA_CONNECTORS_VERSION}
container_name: connectors
ports:
- "8086:8080"
environment:
CAMUNDA_CLIENT_MODE: self-managed
CAMUNDA_CLIENT_RESTADDRESS: http://orchestration:8080
CAMUNDA_CLIENT_GRPCADDRESS: http://orchestration:26500
CAMUNDA_CLIENT_AUTH_METHOD: oidc
CAMUNDA_CLIENT_AUTH_TOKENURL: https://my-identityserver:8055/connect/token
CAMUNDA_CLIENT_AUTH_CLIENTID: zeepi-api-client
CAMUNDA_CLIENT_AUTH_CLIENTSECRET: zeepi-api-client-secret
JAVA_TOOL_OPTIONS: "-Djavax.net.ssl.trustStore=/opt/certs/customTrustStore.jks -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=JKS"
env_file: connector-secrets.txt
restart: unless-stopped
healthcheck:
test:
["CMD", "curl", "-f", "http://localhost:8080/actuator/health/readiness"]
interval: 30s
timeout: 60s
retries: 5
start_period: 30s
volumes:
#- "./.connectors/application.yaml:/app/application.yaml"
- "/app/camunda/customTrustStore.jks:/opt/certs/customTrustStore.jks:ro"
networks:
- camunda-platform
extra_hosts:
- "host.docker.internal:host-gateway"
depends_on:
orchestration:
condition: service_healthy
identity:
container_name: identity
image: camunda/identity:${CAMUNDA_IDENTITY_VERSION}
ports:
- "8084:8084"
environment:
IDENTITY_DATABASE_HOST: postgres
IDENTITY_DATABASE_PORT: 5432
IDENTITY_DATABASE_NAME: ${POSTGRES_DB}
IDENTITY_DATABASE_USERNAME: ${POSTGRES_USER}
IDENTITY_DATABASE_PASSWORD: ${POSTGRES_PASSWORD}
HOST: ${HOST}
management.endpoints.web.exposure.include: health,configprops
management.endpoint.health.probes.enabled: true
MANAGEMENT_ENDPOINT_CONFIGPROPS_SHOW_VALUES: ALWAYS
SPRING_PROFILES_ACTIVE: oidc
CAMUNDA_IDENTITY_TYPE: GENERIC
CAMUNDA_IDENTITY_ISSUER: https://my-identityserver:8055
CAMUNDA_IDENTITY_ISSUERBACKENDURL: https://my-identityserver:8055
CAMUNDA_IDENTITY_CLIENT_ID: zeepi-api-client
CAMUNDA_IDENTITY_CLIENT_SECRET: zeepi-api-client-secret
CAMUNDA_IDENTITY_AUDIENCE: zeebe-api-audience
JAVA_TOOL_OPTIONS: "-Djavax.net.ssl.trustStore=/opt/certs/customTrustStore.jks -Djavax.net.ssl.trustStorePassword=changeit -Djavax.net.ssl.trustStoreType=JKS"
healthcheck:
test:
[
"CMD",
"wget",
"-q",
"--tries=1",
"--spider",
"http://localhost:8082/actuator/health",
]
interval: 5s
timeout: 60s
retries: 30
start_period: 60s
restart: unless-stopped
volumes:
#- "./.identity/application.yaml:/app/application.yaml"
- "/app/camunda/customTrustStore.jks:/opt/certs/customTrustStore.jks:ro"
networks:
- camunda-platform
- identity-network
extra_hosts:
- "host.docker.internal:host-gateway"
postgres:
container_name: postgres
image: postgres:${POSTGRES_VERSION}
ports:
- "5432:5432"
environment:
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
restart: unless-stopped
healthcheck:
test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
interval: 10s
timeout: 60s
retries: 5
start_period: 10s
volumes:
- postgres:/var/lib/postgresql/data
networks:
- identity-network
extra_hosts:
- "host.docker.internal:host-gateway"
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:${ELASTIC_VERSION}
container_name: elasticsearch
ports:
- "9200:9200"
- "9300:9300"
environment:
- bootstrap.memory_lock=true
- discovery.type=single-node
- xpack.security.enabled=false
- cluster.routing.allocation.disk.threshold_enabled=false
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
restart: unless-stopped
healthcheck:
test:
[
"CMD-SHELL",
"curl -s http://localhost:9200/_cluster/health | grep -E '\"status\":\"(yellow|green)\"' || exit 1",
]
interval: 30s
timeout: 60s
retries: 5
start_period: 60s
volumes:
- elastic:/usr/share/elasticsearch/data
networks:
- camunda-platform
extra_hosts:
- "host.docker.internal:host-gateway"
volumes:
orchestration:
elastic:
postgres:
postgres-web:
networks:
camunda-platform:
identity-network:
web-modeler:
docker-compose-camunda-forum.yaml (8.5 KB)