Kuberntes zeebe cluster secure

Mihnea_Staub · February 3, 2022, 12:54pm

Hello Zeebe team,

I’m trying to deploy a secure zeebe cluster, I have followed this procedure in order to create the certificates: Secure cluster communication | Camunda Cloud Docs. After I have create the certificates, this error appears in each broker:

“Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER”

Openssl version: OpenSSL 1.1.1f
ChartVersion: 1.3.2

The insecure cluster is working as expected, but the secure is not working. Can you please guide me what I’m doing wrong here? Thank you!

Zelldon · February 3, 2022, 1:27pm

Hey @Mihnea_Staub

can you please share a step by step guide how to reproduce this issue or at least what kind of configuration you have deployed together with your cluster. What kind of deployment method do you use?

Greets
Chris

Mihnea_Staub · February 3, 2022, 1:56pm

Hello @Zelldon,

Thank you for the respond.

I have deployed the zeebe cluster using the the helm-chart. I have three replicas in the cluster.

And I have added a secret with the certificates that where created like in the link above with the openssl commands.

I have added those variables in the StatefulSet:

        - name: ZEEBE_BROKER_NETWORK_SECURITY_ENABLED
          value: "true"
        - name: ZEEBE_BROKER_NETWORK_SECURITY_CERTIFICATECHAINPATH
          value: "/usr/local/zeebe/cert/chainNode.pem"
        - name: ZEEBE_BROKER_NETWORK_SECURITY_PRIVATEKEYPATH
          value: "/usr/local/zeebe/cert/node.key"

I don’t know really, if I need to create a certificate for each replica or I can use one for all replicas. I have tried the both scenario and the same issue appear.

Zelldon · February 3, 2022, 2:18pm

Hey @Mihnea_Staub

thanks for sharing!

Did you added the certs via separate PVC ? Like with the exrtra volume config ? camunda-cloud-helm/values.yaml at main · camunda-community-hub/camunda-cloud-helm · GitHub

Yes one cert for all is enough.

Mihnea_Staub · February 3, 2022, 2:23pm

Hello @Zelldon,

I did not add them with those values:

extraVolumes: {}
extraVolumeMounts: {}

I have added by modifying the StatefulSet like this:

        volumeMounts:
        - name: certs
          mountPath: /usr/local/zeebe/cert/
      volumes:
      - name: certs
        secret:
          secretName: zeebe-certs
          defaultMode: 256

But, is the same configuration even if I use the values or I modify the StatefulSet.

Zelldon · February 3, 2022, 2:27pm

Yes sure just wanted to verify how you do it Thanks. I can try tomorrow to reproduce this and ask also internally.

Mihnea_Staub · February 3, 2022, 2:30pm

Thank you very much @Zelldon for the help.

Maybe it will help what commands I have used in order to create the certificates:

openssl req -new -newkey rsa:2048 -nodes -out ca.csr -keyout ca.key
openssl x509 -trustout -signkey ca.key -days 365 -req -in ca.csr -out ca.pem

openssl genpkey -out nodeA.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048

openssl req -new -key nodeA.key -out nodeA.csr

openssl x509 -req -days 365 -in nodeA.csr -CA ca.pem -CAkey ca.key -set_serial 01 -out nodeA.pem

cat nodeA.pem ca.pem > chainNodeA.pem

After that I have change change the name of the pem file and key file to: chainNode.pem and node.key.

And after that create a Secret resource with those two files.

Zelldon · February 3, 2022, 4:55pm

One thing a colleague of mine mentioned to me is that it is necessary that the gateway is configure similar, to use TLS and adding the certificate etc. Otherwise it might cause the exception you’re seeing. Could you verify that?

Greets
Chris

Mihnea_Staub · February 4, 2022, 7:42am

Hey @Zelldon,

Thank you very much for this point, is working now from what I see from the logs, but when I want to see the actual cluster with the this command: "zbctl status --certPath ", I receive this error:

./zbctl status --certPath chainNode.pem
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

I have read a little bit, abut this problem and it seems to be a problem with the grpc client on the Gateway.

Zelldon · February 4, 2022, 8:02am

Hey @Mihnea_Staub

happy to hear that!

Would you mind to share the complete example you’re using so maybe we can reuse this for our docs? Or if other users stumble over this issue again.

Did you checked this section already ? Secure client communication | Camunda Cloud Docs

Greets
Chris

Mihnea_Staub · February 4, 2022, 8:26am

Hey @Zelldon,

It’s my pleasure to do that. The complete example looks like this:

Create the certificate following this procedure: Secure cluster communication | Camunda Cloud Docs
1.1. Please be aware that the same certificate should be used for the brokers and for the gateway. In conclusion you need to create one certificate, for how many brokers do you want and also for one gateway.
Deploy the zeebe-cluster with gateway in a kubernetes environment (e.g docker-desktop, minikube) using the helm chart
2.1. [Option 1] You can do that by modifying the values.yaml for brokers and gateway, using those values.yaml:

## Gateway
extraGatewayVolumeMounts: {}
extraGatewayVolume: {}

But you need to add also those in the gateway deploymetn env variables:
            - name: ZEEBE_GATEWAY_CLUSTER_SECURITY_ENABLED
              value: "true"
            - name: ZEEBE_GATEWAY_CLUSTER_SECURITY_CERTIFICATECHAINPATH
              value: "/usr/local/zeebe/cert/chainNode.pem"
            - name: ZEEBE_GATEWAY_CLUSTER_SECURITY_PRIVATEKEYPATH
              value: "/usr/local/zeebe/cert/node.key"

## Brokers
extraVolumes: {}
extraVolumeMounts: {}

But you need to add also those in the StatefulSet env variables:
        - name: ZEEBE_BROKER_NETWORK_SECURITY_ENABLED
          value: "true"
        - name: ZEEBE_BROKER_NETWORK_SECURITY_CERTIFICATECHAINPATH
          value: "/usr/local/zeebe/cert/chainNode.pem"
        - name: ZEEBE_BROKER_NETWORK_SECURITY_PRIVATEKEYPATH
          value: "/usr/local/zeebe/cert/node.key"

2.2 [Option2] You can change the Brokers StatefulSet and gateway deployment:

## Gateway

        - name: ZEEBE_BROKER_NETWORK_SECURITY_ENABLED
          value: "true"
        - name: ZEEBE_BROKER_NETWORK_SECURITY_CERTIFICATECHAINPATH
          value: "/usr/local/zeebe/cert/chainNode.pem"
        - name: ZEEBE_BROKER_NETWORK_SECURITY_PRIVATEKEYPATH
          value: "/usr/local/zeebe/cert/node.key"

        volumeMounts:
        - name: certs
          mountPath: /usr/local/zeebe/cert/
      volumes:
      - name: certs
        secret:
          secretName: zeebe-certs
          defaultMode: 256

## Broker
        - name: ZEEBE_BROKER_NETWORK_SECURITY_ENABLED
          value: "true"
        - name: ZEEBE_BROKER_NETWORK_SECURITY_CERTIFICATECHAINPATH
          value: "/usr/local/zeebe/cert/chainNode.pem"
        - name: ZEEBE_BROKER_NETWORK_SECURITY_PRIVATEKEYPATH
          value: "/usr/local/zeebe/cert/node.key"

        volumeMounts:
        - name: certs
          mountPath: /usr/local/zeebe/cert/
      volumes:
      - name: certs
        secret:
          secretName: zeebe-certs
          defaultMode: 256

You can test with this command: “zbctl --insecure status” and you will see all the brokers.
3.1 For now to see the brokers in a secure cluster with the zbctl command is not working

Also, I have checked that document and I have diff error that is specified over there:

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

It seems to be a connection error, not a “all SubConns are in TransientFailure”.

Also, I will test with another client to see if I receive the same error and I will put the answer here.

Thank you,
Mihnea

Mihnea_Staub · February 4, 2022, 9:11am

Hello @Zelldon,

I know that I have put to many questions for now, but do you know if the zeebe-operate support TLS enable? I did not found an answer in the documentation and this why I want to ask you.

Thank you,
Mihnea

npepinpe · February 4, 2022, 9:14am

Operate and Tasklist do not support connecting to Zeebe with TLS yet (see reference), though that will come eventually, most likely sometime this year, though I can’t give you anything more accurate.

Mihnea_Staub · February 4, 2022, 9:32am

Thank you for the answer @npepinpe and for the information

Zelldon · February 4, 2022, 9:39am

Awesome thanks for sharing @Mihnea_Staub !

Mihnea_Staub · February 4, 2022, 10:55am

Hello Guys,

I want to thank you again for all the information and help that you have provided.

But, I have came back with the answer from the client that need to authenticate in a secure cluster. I have tried to scenarios in order to connect to the cluster:

With the zbctl tool
1.1 I have run the command: zbctl status and this error appears:

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

1.2. I have run the command: zbctl status --certPath ca.pem and the same error appears:

Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"

With java client
I have tried to connect with the Java client and this error appears:

Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00001204000000000000037fffffff000400100000000600002000000004080000000000000f0001

If you need I can provide the full logs from the Java client, if you think will help, I did not want to post it to don’t create a huge post.

Please, let me know if you want to create another ticket/thread for this problem, because I have asked you already allot of things here.

Looking forward to your answer.

Thank you,
Mihnea

jwulf · February 4, 2022, 11:13am

What happens when you connect to the same cluster with zbctl status --insecure?

Mihnea_Staub · February 4, 2022, 11:21am

Hello @jwulf,

Is working as expected, with zbctl cli and also with the java client, but I don’t want to have a insecure connection between client and gateway. This why I want to use the certificates in order to connect to a secure cluster.

Mihnea_Staub · February 6, 2022, 10:25am

Hello Guys,

Thank you very much for all the guide and information.

I have managed to configure the TLS also for the client communication. I did not know that you need to create another certificate and also configure a little bit the gateway template in order to activate the client secure communication.

More information here: Secure client communication | Camunda Cloud Docs

Thank you,
Mihnea

ntt.duynh · May 10, 2023, 10:43am

Hi @Mihnea_Staub,

I’m trying to configure the same thing. Can you please share a step-by-step guide on how to create secrets and your camunda value file?

About the certificate, I ran the following commands, assuming I only need 1 cert right?

openssl req -config <(printf "[req]\ndistinguished_name=dn\n[dn]\n[ext]\nbasicConstraints=CA:TRUE,pathlen:0") -new -newkey rsa:2048 -nodes -subj "/C=DE/O=Test/OU=Test/ST=BE/CN=cluster.local" -x509 -extensions ext -keyout ca.key -out ca.pem

openssl genpkey -out node.key -algorithm RSA -pkeyopt rsa_keygen_bits:2048
openssl req -new -key node.key -out node.csr
openssl x509 -req -days 365 -in node.csr -CA ca.pem -CAkey ca.key -set_serial 01 -extfile <(printf "subjectAltName = DNS.1:*.cluster.local") -out node.pem
cat node.pem ca.pem > chainNode.pem