Raft Error deploying on cloud with kubernets

Camunda 8 Topics Discussion & Questions
1

Hi
I am trying to deploy Zeebe on AWS on a kube cluster with 2 worker nodes using zeebe-full helm chart. I only made one change to it - disabled elasticsearch. I have made sure that my security group on my VPC allows all traffic between my VMs. I see this from all 3 broker pods and they never go in Ready state -

  io.atomix.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - Poll request to 2 failed: java.net.ConnectException: Expected to send a message with subject 'raft-partition-partition-1-poll' to member '2', but member is not known. Known members are '[Member{id=1, address=zeebe-demo-zeebe-1.zeebe-demo-zeebe.default.svc.cluster.local:26502, properties={}}]'.

2021-04-30 21:59:22.948 [] [raft-server-1-raft-partition-partition-3] WARN
io.atomix.raft.roles.FollowerRole - RaftServer{raft-partition-partition-3}{role=FOLLOWER} - Poll request to 0 failed: java.net.ConnectException: Expected to send a message with subject ‘raft-partition-partition-3-poll’ to member ‘0’, but member is not known. Known members are ‘[Member{id=1, address=zeebe-demo-zeebe-1.zeebe-demo-zeebe.default.svc.cluster.local:26502, properties={}}]’.

It seems like a given broker pod can’t reach the other two. Initially, I do get an initialization completion message in the pod log. And then the above error is output in the loop.

Any help on why this could be the case?

Thanks a lot
AJ

2

Hey @atx123456

thanks for trying Zeebe and welcome in our forum :blush:

You’re right it seems Broker 1 cant reach the other node, Broker 0. My first question would be, how do you deploy zeebe? Are you using the helm charts? How does your condiguration look like? The cfg is printed in the beginning of the log, could you post it here?

Greets
Chris

3

May I ask if you have solved this problem? I also met the same problem.

4