Has anyone experienced gateway-broker timeout issues in Camunda 8.3.10 on Kubernetes?
We’ve been struggling with persistent connectivity issues in our Camunda test cluster since July 22nd and wondering if anyone has faced something similar.
What we’re seeing:
statuscode.deadline_exceeded: Time out between gateway and broker:
Request timed out after PT15S
StatusCode.UNAVAILABLE: failed to connect to all addresses; last
error: UNKNOWN: ipv4:...:26500: Operation not permitted
Plus we’re also getting:
ZeebeBackpressureerrorsResource exhaustederrors
Setup:
- Camunda 8.3.10 on Kubernetes
- 3 Zeebe brokers + 3 gateway pods
- Multiple preview environments connecting to the cluster
Strange part:
- Cluster health endpoint reports everything as “healthy”
- Brokers running at low resource usage (5-10% CPU/memory)
- Getting connection timeouts, backpressure, AND resource exhausted errors all at once
- Complete cluster recreation didn’t help
We’ve tried resource scaling, backpressure tuning, and load reduction but the issue persists. The combination of network errors + backpressure + resource exhausted (while resources look fine)
Would like to understand if anyone else saw something similar ?? Any inputs are appreciated . Thanks