I’m trying to deploy a secure zeebe cluster, I have followed this procedure in order to create the certificates: Secure cluster communication | Camunda Cloud Docs. After I have create the certificates, this error appears in each broker:
can you please share a step by step guide how to reproduce this issue or at least what kind of configuration you have deployed together with your cluster. What kind of deployment method do you use?
I don’t know really, if I need to create a certificate for each replica or I can use one for all replicas. I have tried the both scenario and the same issue appear.
One thing a colleague of mine mentioned to me is that it is necessary that the gateway is configure similar, to use TLS and adding the certificate etc. Otherwise it might cause the exception you’re seeing. Could you verify that?
Thank you very much for this point, is working now from what I see from the logs, but when I want to see the actual cluster with the this command: "zbctl status --certPath ", I receive this error:
./zbctl status --certPath chainNode.pem
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
I have read a little bit, abut this problem and it seems to be a problem with the grpc client on the Gateway.
It’s my pleasure to do that. The complete example looks like this:
Create the certificate following this procedure: Secure cluster communication | Camunda Cloud Docs
1.1. Please be aware that the same certificate should be used for the brokers and for the gateway. In conclusion you need to create one certificate, for how many brokers do you want and also for one gateway.
Deploy the zeebe-cluster with gateway in a kubernetes environment (e.g docker-desktop, minikube) using the helm chart
2.1. [Option 1] You can do that by modifying the values.yaml for brokers and gateway, using those values.yaml:
## Gateway
extraGatewayVolumeMounts: {}
extraGatewayVolume: {}
But you need to add also those in the gateway deploymetn env variables:
- name: ZEEBE_GATEWAY_CLUSTER_SECURITY_ENABLED
value: "true"
- name: ZEEBE_GATEWAY_CLUSTER_SECURITY_CERTIFICATECHAINPATH
value: "/usr/local/zeebe/cert/chainNode.pem"
- name: ZEEBE_GATEWAY_CLUSTER_SECURITY_PRIVATEKEYPATH
value: "/usr/local/zeebe/cert/node.key"
## Brokers
extraVolumes: {}
extraVolumeMounts: {}
But you need to add also those in the StatefulSet env variables:
- name: ZEEBE_BROKER_NETWORK_SECURITY_ENABLED
value: "true"
- name: ZEEBE_BROKER_NETWORK_SECURITY_CERTIFICATECHAINPATH
value: "/usr/local/zeebe/cert/chainNode.pem"
- name: ZEEBE_BROKER_NETWORK_SECURITY_PRIVATEKEYPATH
value: "/usr/local/zeebe/cert/node.key"
2.2 [Option2] You can change the Brokers StatefulSet and gateway deployment:
You can test with this command: “zbctl --insecure status” and you will see all the brokers.
3.1 For now to see the brokers in a secure cluster with the zbctl command is not working
Also, I have checked that document and I have diff error that is specified over there:
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
It seems to be a connection error, not a “all SubConns are in TransientFailure”.
Also, I will test with another client to see if I receive the same error and I will put the answer here.
I know that I have put to many questions for now, but do you know if the zeebe-operate support TLS enable? I did not found an answer in the documentation and this why I want to ask you.
Operate and Tasklist do not support connecting to Zeebe with TLS yet (see reference), though that will come eventually, most likely sometime this year, though I can’t give you anything more accurate.
I want to thank you again for all the information and help that you have provided.
But, I have came back with the answer from the client that need to authenticate in a secure cluster. I have tried to scenarios in order to connect to the cluster:
With the zbctl tool
1.1 I have run the command: zbctl status and this error appears:
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
1.2. I have run the command: zbctl status --certPath ca.pem and the same error appears:
Error: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: first record does not look like a TLS handshake"
With java client
I have tried to connect with the Java client and this error appears:
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 00001204000000000000037fffffff000400100000000600002000000004080000000000000f0001
If you need I can provide the full logs from the Java client, if you think will help, I did not want to post it to don’t create a huge post.
Please, let me know if you want to create another ticket/thread for this problem, because I have asked you already allot of things here.
Is working as expected, with zbctl cli and also with the java client, but I don’t want to have a insecure connection between client and gateway. This why I want to use the certificates in order to connect to a secure cluster.
Thank you very much for all the guide and information.
I have managed to configure the TLS also for the client communication. I did not know that you need to create another certificate and also configure a little bit the gateway template in order to activate the client secure communication.