Connecting to zeebe gateway under ingress and tls enabled

[msjo]

This issue is next step to https://forum.camunda.io/t/self-managed-local-cluster-with-ingress-controller-does-not-diaplay-identity/50529

as above chart value mentioned, I’ve setup ingress and tls. see the following snippet.

zeebe-gateway:
  replicas: 1
  ingress:
    enabled: true
    className: nginx
    host: "zeebe.camunda-demo-ingress.committed.co.kr"
    tls:
      enabled: true
      secretName: camunda-demo-ingress-tls

When I test connection with

zbctl \
--address zeebe.camunda-demo-ingress.committed.co.kr \
--authzUrl https://camunda-demo-ingress.committed.co.kr/auth/realms/camunda-platform/protocol/openid-connect/token \
--clientId <client id> \
--clientSecret <client secret> \
--host zeebe.camunda-demo-ingress.committed.co.kr \
--port 443 \
status

It throws deadline exception.

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate signed by unknown authority"

So, I’ve tested certificate using openssl

openssl s_client -connect camunda-demo-ingress.committed.co.kr:443

subject=C = KR, ST = Seoul, L = Guro, O = Committed Korea, OU = Camunda, CN = *.committed.co.kr, emailAddress = minsoo.jo@committed.co.kr

issuer=C = KR, ST = Seoul, L = Guro, O = Committed Korea, OU = camunda, CN = camunda-demo-ingress.committed.co.kr, emailAddress = minsoo.jo@committed.co.kr

above command shows that the server crt configured correctly. but using the following test

openssl s_client -connect zeebe.camunda-demo-ingress.committed.co.kr:443

subject=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate

issuer=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate

It shows that zeebe-gateway host configuration ‘zeebe.camunda-demo-ingress.committed.co.kr’ does not applied.

What did I missed?

[msjo]

My ingress controller was configured as following (zeebe.camunda-demo-ingress.committed.co.kr)

## start server zeebe.camunda-demo-ingress.committed.co.kr
        server {
                server_name zeebe.camunda-demo-ingress.committed.co.kr ;

                listen 80  ;
                listen [::]:80  ;
                listen 443  ssl http2 ;
                listen [::]:443  ssl http2 ;

                set $proxy_upstream_name "-";

                ssl_certificate_by_lua_block {
                        certificate.call()
                }

                location / {

                        set $namespace      "default";
                        set $ingress_name   "camunda-platform-zeebe-gateway";
                        set $service_name   "camunda-platform-zeebe-gateway";
                        set $service_port   "26500";
                        set $location_path  "/";
                        set $global_rate_limit_exceeding n;

                        rewrite_by_lua_block {
                                lua_ingress.rewrite({
                                        force_ssl_redirect = false,
                                        ssl_redirect = false,
                                        force_no_ssl_redirect = false,
                                        preserve_trailing_slash = false,
                                        use_port_in_redirects = false,
                                        global_throttle = { namespace = "", limit = 0, window_size = 0, key = { }, ignored_cidrs = { } },
                                })
                                balancer.rewrite()
                                plugins.run()
                        }

                        # be careful with `access_by_lua_block` and `satisfy any` directives as satisfy any
                        # will always succeed when there's `access_by_lua_block` that does not have any lua code doing `ngx.exit(ngx.DECLINED)`
                        # other authentication method such as basic auth or external auth useless - all requests will be allowed.
                        #access_by_lua_block {
                        #}

                        header_filter_by_lua_block {
                                lua_ingress.header()
                                plugins.run()
                        }

                        body_filter_by_lua_block {
                                plugins.run()
                        }

                        log_by_lua_block {
                                balancer.log()

                                monitor.call()

                                plugins.run()
                        }

                        port_in_redirect off;

                        set $balancer_ewma_score -1;
                        set $proxy_upstream_name "default-camunda-platform-zeebe-gateway-26500";
                        set $proxy_host          $proxy_upstream_name;
                        set $pass_access_scheme  $scheme;

                        set $pass_server_port    $server_port;

                        set $best_http_host      $http_host;
                        set $pass_port           $pass_server_port;

                        set $proxy_alternative_upstream_name "";

                        client_max_body_size                    1m;

                        grpc_set_header Host                   $best_http_host;

                        # Pass the extracted client certificate to the backend

                        # Allow websocket connections
                        grpc_set_header                        Upgrade           $http_upgrade;

                        grpc_set_header                        Connection        $connection_upgrade;

                        grpc_set_header X-Request-ID           $req_id;
                        grpc_set_header X-Real-IP              $remote_addr;

                        grpc_set_header X-Forwarded-For        $remote_addr;

                        grpc_set_header X-Forwarded-Host       $best_http_host;
                        grpc_set_header X-Forwarded-Port       $pass_port;
                        grpc_set_header X-Forwarded-Proto      $pass_access_scheme;
                        grpc_set_header X-Forwarded-Scheme     $pass_access_scheme;

                        grpc_set_header X-Scheme               $pass_access_scheme;

                        # Pass the original X-Forwarded-For
                        grpc_set_header X-Original-Forwarded-For $http_x_forwarded_for;

                        # mitigate HTTPoxy Vulnerability
                        # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
                        grpc_set_header Proxy                  "";

                        # Custom headers to proxied server

                        proxy_connect_timeout                   5s;
                        proxy_send_timeout                      60s;
                        proxy_read_timeout                      60s;

                        proxy_buffering                         off;
                        proxy_buffer_size                       128k;
                        proxy_buffers                           4 128k;

                        proxy_max_temp_file_size                1024m;

                        proxy_request_buffering                 on;
                        proxy_http_version                      1.1;

                        proxy_cookie_domain                     off;
                        proxy_cookie_path                       off;

                        # In case of errors try the next upstream server before returning an error
                        proxy_next_upstream                     error timeout;
                        proxy_next_upstream_timeout             0;
                        proxy_next_upstream_tries               3;

                        grpc_pass grpc://upstream_balancer;

                        proxy_redirect                          off;

                }

        }
        ## end server zeebe.camunda-demo-ingress.committed.co.kr

[msjo]

I just save default nginx controller certificate “subject=O = Acme Co, CN = Kubernetes Ingress Controller Fake Certificate” as file and set the file into certPath parameter like following

zbctl \
--address zeebe.camunda-demo-ingress.committed.co.kr \
--authzUrl https://camunda-demo-ingress.committed.co.kr/auth/realms/camunda-platform/protocol/openid-connect/token \
--certPath /Users/minsoojo/Workspaces/BPMN/certificate-demo/default-nginx.crt \
--clientId <client id> \
--clientSecret <client-secret> \
--host zeebe.camunda-demo-ingress.committed.co.kr \
--port 443 \
status

following error returned

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for ingress.local, not zeebe.camunda-demo-ingress.committed.co.kr"

It tells that the right certificate should be set to the zeebe.camunda-demo-ingress.committed.co.kr server name section.

[msjo]

I guess it is related to my self signed wild card certificate.
With googling, I regenerate my self signed 2 certificates(one for https and the other for zeebe grpc).

First trial, zbctl throws "Error: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: x509: certificate relies on legacy Common Name field, use SANs instead”, And I regenerate with SAN(subject altertive name) added.

After that, another error throws

Error: rpc error: code = Canceled desc = failed to apply token: failed to obtain access token: Post "https://camunda-demo-ingress.committed.co.kr/auth/realms/camunda-platform/protocol/openid-connect/token": x509: “camunda-demo-ingress.committed.co.kr” certificate is not standards compliant

After register camunda-demo-ingress.committed.co.kr certificate into keychain app again, It finally succeeded. I’ll upload my journey

[msjo]

Self Signed Certificate Generation Step is.

# 1. generate rootca.key 
openssl ecparam -out rootca.key -name prime256v1 -genkey

# 2. generate csr(Certificate Signing Request) for self signing root ca 
openssl req -new -sha256 -key rootca.key -out rootca.csr -config rootca-csr.txt

####### rootca-csr.txt
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C=KR
ST=Seoul
L=Guro
O=Committed Korea
OU=Camunda
emailAddress=<mail id here>
CN = committed.co.kr

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = committed.co.kr
########## end of rootca-csr.txt

3. self signing root ca 
openssl x509 -req -sha256 -days 999999 -in rootca.csr -signkey rootca.key -out rootca.crt

4. generate server key for (camunda-demo-ingress.committed.co.kr)
openssl ecparam -out server.key -name prime256v1 -genkey

5. generate csr for server.key 
openssl req -new -sha256 -key server.key -out server.csr -config server-csr.txt
########## server-csr.txt contents
[req]
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn

[ dn ]
C=KR
ST=Seoul
L=Guro
O=Committed Korea
OU=Camunda
emailAddress=<mail id>
CN = camunda-demo-ingress.committed.co.kr

[ req_ext ]
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = camunda-demo-ingress.committed.co.kr
###########

6. generate self signed certificate for the server.key
openssl x509 -req -extfile <(printf "subjectAltName=DNS:camunda-demo-ingress.committed.co.kr") -sha256 -days 999999 -in server.csr -CA rootca.crt -CAkey rootca.key -CAcreateserial -out server.crt

7. verify server certificate( the section for "X509v3 Subject Alternative Name:" should exist!)
openssl x509 -in server.crt -text -noout

8. generate chaine key (rootca.crt +  server.crt)
cat server.crt rootca.crt > server.pem

9. get base64 encoding for key and certificates
cat server.pem|base64
cat server.key|base64

10. repeat 4 thourgh 9 for zeebe (zeebe.camunda-demo-ingress.committed.co.kr)
  10-1. openssl ecparam -out zeebe.key -name prime256v1 -genkey
  10-2. openssl req -new -sha256 -key zeebe.key -out zeebe.csr -config zeebe-csr.txt
  10-3. openssl x509 -req -extfile <(printf "subjectAltName=DNS:zeebe.camunda-demo-ingress.committed.co.kr") -sha256 -days 999999 -in zeebe.csr -CA rootca.crt -CAkey rootca.key -CAcreateserial -out zeebe.crt
  10-4. openssl x509 -in zeebe.crt -text -noout
  10-5. cat zeebe.crt rootca.crt > zeebe.pem
  10-6. cat zeebe.pem|base64
  10-7. cat zeebe.key|base64

before generate camunda platform using helm, the secret should be applied into k8s.
remaining steps in the documentation works well.

Thank you

[msjo]

lastly. register self signed certificates into cacerts in jvm using keytool as following :

keytool -keystore $JAVA_HOME/lib/security/cacerts -importcert -storepass changeit -file server.pem -alias camunda-demo-ingress
keytool -keystore $JAVA_HOME/lib/security/cacerts -importcert -storepass changeit -file zeebe.pem -alias zeebe