Stuck Broker Pod - Cannot Connect With Other Nodes

Camunda 8 Topics
1

We are running zeebe 8.1.6 on aws using helm charts. Since last few days we are seeing that the broker pod does not start up and remains stuck with the below logs. It cannot connect with other members in the cluster. The config is for 2 gateways and 3 brokers - 3 partitions.We tried terminating all the ec2 instances (except the k3s master) and recreating them still the same error gets printed in the logs. This issue was observed earlier as detailed on this github issue but it was closed with some code fix.

Any thoughts on how to resolve this issue?

Broker logs

2023-03-14 12:09:42.682 [] [raft-server-0-raft-partition-partition-1] INFO
      io.atomix.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - Sending poll requests to all active members: [DefaultRaftMember{id=1, type=ACTIVE, updated=2023-03-14T12:08:55.177Z}]
2023-03-14 12:09:42.683 [] [raft-server-0-raft-partition-partition-1] WARN
      io.atomix.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - Poll request to 1 failed: java.net.ConnectException: Expected to send a message with subject 'raft-partition-partition-1-poll' to member '1', but member is not known. Known members are '[Member{id=0, address=c8-zeebe-0.c8-zeebe.default.svc:26502, properties={brokerInfo=EADJAAAAAwAAAAAAAwAAAAMAAAACAAAAAAABCgAAAGNvbW1hbmRBcGklAAAAYzgtemVlYmUtMC5jOC16ZWViZS5kZWZhdWx0LnN2YzoyNjUwMQUAAAwAAAUAAAA4LjEuNgUAAA==}}]'.

Thanks.

2

Ping to all the nodes work. I enabled debug logs on the zeebe broker and shared here. One of the debug log that struck me

[GatewayTopologyManager] [Broker-0-zb-actors-3] DEBUG
io.camunda.zeebe.gateway - Received metadata change from Broker 0, partitions {}, terms {} and health {}.

3

Looks like our networking team had blocked UDP ports which lead to this issue. We enabled UDP ports and the brokers came up

4

Did you start camuda in helm mode？I have the same error message as you。But how do i enable UDP ports in helm mode?

5

@taoyin - In my case we are deploying zeebe engine on aws. The security configuration of the ec2 instance had blocked UDP ports. Enable them from the aws web console resolved the error. Hope that helps

6