Request timed out between gateway and broker

Hello

We have 2 different VMs and we installed gateway one of them and broker to another one. When we started gateway and broker i can see from gateway logs that gateway gets information related with broker based on partition id etc. I can see several debug logs such as :

Received new broker BrokerInfo{nodeId=0, partitionsCount=1, clusterSize=1, replicationFactor=1, partitionRoles={1=LEADER}, partitionLeaderTerms={1=1}, partitionHealthStatuses={}, version=0.25.1}.
gateway | 2021-01-12 18:43:20.406 [GatewayTopologyManager] [gateway-scheduler-zb-actors-0] DEBUG io.zeebe.gateway - Received REACHABILITY_CHANGED for broker 0, do nothing.

Gateway prints this debug logs every few seconds. When i try to deploy flow using client i am getting following exception :

DEBUG io.zeebe.gateway - Expected to handle gRPC request, but request timed out between gateway and broker
gateway | java.util.concurrent.TimeoutException: Request timed out after PT15S
gateway | at io.zeebe.transport.impl.RequestContext.timeout(RequestContext.java:112) ~[zeebe-transport-0.25.1.jar:0.25.1]

gateway configuration :

version: “2”

networks:
zeebe_network:
driver: bridge

services:
gateway:
restart: always
container_name: gateway
image: camunda/zeebe:0.25.1
environment:
- ZEEBE_LOG_LEVEL=debug
- ZEEBE_STANDALONE_GATEWAY=true
- ZEEBE_GATEWAY_NETWORK_PORT=26500
- ZEEBE_GATEWAY_CLUSTER_CONTACTPOINT=10.61.137.204:26502
- ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED=false
ports:
- “26500:26500”
networks:
- zeebe_network

broker configuration :
version: “2”

networks:
zeebe_network:
driver: bridge

services:
node0:
container_name: zeebe_broker_1
image: camunda/zeebe:0.25.1
environment:
- ZEEBE__LOG_LEVEL=debug
- ZEEBE_NODE_ID=0
- ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT=1
- ZEEBE_REPLICATION_FACTOR=1
- ZEEBE_CLUSTER_SIZE=1
- ZEEBE_BROKER_GATEWAY_ENABLE=false
ports:
- “9600:9600”
- “26502:26502”
- “26501:26501”
networks:
- zeebe_network

Gateway configuration from startup log:

“network” : {
gateway | “host” : “0.0.0.0”,
gateway | “port” : 26500,
gateway | “minKeepAliveInterval” : “PT30S”
gateway | },
gateway | “cluster” : {
gateway | “contactPoint” : “10.61.137.204:26502”,
gateway | “requestTimeout” : “PT15S”,
gateway | “clusterName” : “zeebe-cluster”,
gateway | “memberId” : “gateway”,
gateway | “host” : “192.168.48.2”,
gateway | “port” : 26502,
gateway | “membership” : {
gateway | “broadcastUpdates” : false,
gateway | “broadcastDisputes” : true,
gateway | “notifySuspect” : false,
gateway | “gossipInterval” : “PT0.25S”,
gateway | “gossipFanout” : 2,
gateway | “probeInterval” : “PT1S”,
gateway | “probeTimeout” : “PT2S”,
gateway | “suspectProbes” : 3,
gateway | “failureTimeout” : “PT10S”,
gateway | “syncInterval” : “PT10S”
gateway | }
gateway | },
gateway | “threads” : {
gateway | “managementThreads” : 1
gateway | },
gateway | “monitoring” : {
gateway | “enabled” : false,
gateway | “host” : “0.0.0.0”,
gateway | “port” : 9600
gateway | },
gateway | “security” : {
gateway | “enabled” : false,
gateway | “certificateChainPath” : null,
gateway | “privateKeyPath” : null
gateway | },
gateway | “longPolling” : {
gateway | “enabled” : true
gateway | },
gateway | “initialized” : true
gateway | }
gateway | 2021-01-12 18:39:55.050 [] [main] INFO io.zeebe.gateway - Version: 0.25.1
gateway | 2021-01-12 18:39:55.051 [] [main] INFO io.zeebe.gateway - Starting gateway with configuration {
gateway | “network” : {
gateway | “host” : “0.0.0.0”,
gateway | “port” : 26500,
gateway | “minKeepAliveInterval” : “PT30S”
gateway | },
gateway | “cluster” : {
gateway | “contactPoint” : “10.61.137.204:26502”,
gateway | “requestTimeout” : “PT15S”,
gateway | “clusterName” : “zeebe-cluster”,
gateway | “memberId” : “gateway”,
gateway | “host” : “192.168.48.2”,
gateway | “port” : 26502,
gateway | “membership” : {
gateway | “broadcastUpdates” : false,
gateway | “broadcastDisputes” : true,
gateway | “notifySuspect” : false,
gateway | “gossipInterval” : “PT0.25S”,
gateway | “gossipFanout” : 2,
gateway | “probeInterval” : “PT1S”,
gateway | “probeTimeout” : “PT2S”,
gateway | “suspectProbes” : 3,
gateway | “failureTimeout” : “PT10S”,
gateway | “syncInterval” : “PT10S”
gateway | }
gateway | },
gateway | “threads” : {
gateway | “managementThreads” : 1
gateway | },
gateway | “monitoring” : {
gateway | “enabled” : false,
gateway | “host” : “0.0.0.0”,
gateway | “port” : 9600
gateway | },
gateway | “security” : {
gateway | “enabled” : false,
gateway | “certificateChainPath” : null,
gateway | “privateKeyPath” : null
gateway | },
gateway | “longPolling” : {
gateway | “enabled” : true
gateway | },
gateway | “initialized” : true
gateway | }

broker configuration log from startup :
zeebe_broker_1 | “network” : {
zeebe_broker_1 | “host” : “172.18.0.2”,
zeebe_broker_1 | “portOffset” : 0,
zeebe_broker_1 | “maxMessageSize” : “4MB”,
zeebe_broker_1 | “advertisedHost” : “172.18.0.2”,
zeebe_broker_1 | “commandApi” : {
zeebe_broker_1 | “host” : “172.18.0.2”,
zeebe_broker_1 | “port” : 26501,
zeebe_broker_1 | “advertisedHost” : “172.18.0.2”,
zeebe_broker_1 | “advertisedPort” : 26501,
zeebe_broker_1 | “address” : “172.18.0.2:26501”,
zeebe_broker_1 | “advertisedAddress” : “172.18.0.2:26501”
zeebe_broker_1 | },
zeebe_broker_1 | “internalApi” : {
zeebe_broker_1 | “host” : “172.18.0.2”,
zeebe_broker_1 | “port” : 26502,
zeebe_broker_1 | “advertisedHost” : “172.18.0.2”,
zeebe_broker_1 | “advertisedPort” : 26502,
zeebe_broker_1 | “address” : “172.18.0.2:26502”,
zeebe_broker_1 | “advertisedAddress” : “172.18.0.2:26502”
zeebe_broker_1 | },
zeebe_broker_1 | “monitoringApi” : {
zeebe_broker_1 | “host” : “172.18.0.2”,
zeebe_broker_1 | “port” : 9600,
zeebe_broker_1 | “advertisedHost” : “172.18.0.2”,
zeebe_broker_1 | “advertisedPort” : 9600,
zeebe_broker_1 | “address” : “172.18.0.2:9600”,
zeebe_broker_1 | “advertisedAddress” : “172.18.0.2:9600”
zeebe_broker_1 | },
zeebe_broker_1 | “maxMessageSizeInBytes” : 4194304
zeebe_broker_1 | },
zeebe_broker_1 | “cluster” : {
zeebe_broker_1 | “initialContactPoints” : [ ],
zeebe_broker_1 | “partitionIds” : [ 1 ],
zeebe_broker_1 | “nodeId” : 0,
zeebe_broker_1 | “partitionsCount” : 1,
zeebe_broker_1 | “replicationFactor” : 1,
zeebe_broker_1 | “clusterSize” : 1,
zeebe_broker_1 | “clusterName” : “zeebe-cluster”,
zeebe_broker_1 | “membership” : {
zeebe_broker_1 | “broadcastUpdates” : false,
zeebe_broker_1 | “broadcastDisputes” : true,
zeebe_broker_1 | “notifySuspect” : false,
zeebe_broker_1 | “gossipInterval” : “PT0.25S”,
zeebe_broker_1 | “gossipFanout” : 2,
zeebe_broker_1 | “probeInterval” : “PT1S”,
zeebe_broker_1 | “probeTimeout” : “PT2S”,
zeebe_broker_1 | “suspectProbes” : 3,
zeebe_broker_1 | “failureTimeout” : “PT10S”,
zeebe_broker_1 | “syncInterval” : “PT10S”
zeebe_broker_1 | }
zeebe_broker_1 | },
zeebe_broker_1 | “threads” : {
zeebe_broker_1 | “cpuThreadCount” : 2,
zeebe_broker_1 | “ioThreadCount” : 2
zeebe_broker_1 | },
zeebe_broker_1 | “data” : {
zeebe_broker_1 | “directories” : [ “/usr/local/zeebe/data” ],
zeebe_broker_1 | “logSegmentSize” : “512MB”,
zeebe_broker_1 | “snapshotPeriod” : “PT15M”,
zeebe_broker_1 | “logIndexDensity” : 100,
zeebe_broker_1 | “diskUsageMonitoringEnabled” : true,
zeebe_broker_1 | “diskUsageReplicationWatermark” : 0.99,
zeebe_broker_1 | “diskUsageCommandWatermark” : 0.97,
zeebe_broker_1 | “diskUsageMonitoringInterval” : “PT1S”,
zeebe_broker_1 | “rocksdb” : {
zeebe_broker_1 | “columnFamilyOptions” : { }
zeebe_broker_1 | },
zeebe_broker_1 | “logSegmentSizeInBytes” : 536870912,
zeebe_broker_1 | “atomixStorageLevel” : “DISK”,
zeebe_broker_1 | “freeDiskSpaceCommandWatermark” : 6153309512,
zeebe_broker_1 | “freeDiskSpaceReplicationWatermark” : 2051103171
zeebe_broker_1 | },
zeebe_broker_1 | “exporters” : { },
zeebe_broker_1 | “gateway” : {
zeebe_broker_1 | “network” : {
zeebe_broker_1 | “host” : “0.0.0.0”,
zeebe_broker_1 | “port” : 26500,
zeebe_broker_1 | “minKeepAliveInterval” : “PT30S”
zeebe_broker_1 | },
zeebe_broker_1 | “cluster” : {
zeebe_broker_1 | “contactPoint” : “172.18.0.2:26502”,
zeebe_broker_1 | “requestTimeout” : “PT15S”,
zeebe_broker_1 | “clusterName” : “zeebe-cluster”,
zeebe_broker_1 | “memberId” : “gateway”,
zeebe_broker_1 | “host” : “172.18.0.2”,
zeebe_broker_1 | “port” : 26502,
zeebe_broker_1 | “membership” : {
zeebe_broker_1 | “broadcastUpdates” : false,
zeebe_broker_1 | “broadcastDisputes” : true,
zeebe_broker_1 | “notifySuspect” : false,
zeebe_broker_1 | “gossipInterval” : “PT0.25S”,
zeebe_broker_1 | “gossipFanout” : 2,
zeebe_broker_1 | “probeInterval” : “PT1S”,
zeebe_broker_1 | “probeTimeout” : “PT2S”,
zeebe_broker_1 | “suspectProbes” : 3,
zeebe_broker_1 | “failureTimeout” : “PT10S”,
zeebe_broker_1 | “syncInterval” : “PT10S”
zeebe_broker_1 | }
zeebe_broker_1 | },
zeebe_broker_1 | “threads” : {
zeebe_broker_1 | “managementThreads” : 1
zeebe_broker_1 | },
zeebe_broker_1 | “monitoring” : {
zeebe_broker_1 | “enabled” : false,
zeebe_broker_1 | “host” : “172.18.0.2”,
zeebe_broker_1 | “port” : 9600
zeebe_broker_1 | },
zeebe_broker_1 | “security” : {
zeebe_broker_1 | “enabled” : false,
zeebe_broker_1 | “certificateChainPath” : null,
zeebe_broker_1 | “privateKeyPath” : null
zeebe_broker_1 | },
zeebe_broker_1 | “longPolling” : {
zeebe_broker_1 | “enabled” : true
zeebe_broker_1 | },
zeebe_broker_1 | “initialized” : true,
zeebe_broker_1 | “enable” : false
zeebe_broker_1 | },
zeebe_broker_1 | “backpressure” : {
zeebe_broker_1 | “enabled” : true,
zeebe_broker_1 | “algorithm” : “VEGAS”,
zeebe_broker_1 | “aimd” : {
zeebe_broker_1 | “requestTimeout” : “PT1S”,
zeebe_broker_1 | “initialLimit” : 100,
zeebe_broker_1 | “minLimit” : 1,
zeebe_broker_1 | “maxLimit” : 1000,
zeebe_broker_1 | “backoffRatio” : 0.9
zeebe_broker_1 | },
zeebe_broker_1 | “fixed” : {
zeebe_broker_1 | “limit” : 20
zeebe_broker_1 | },
zeebe_broker_1 | “vegas” : {
zeebe_broker_1 | “alpha” : 3,
zeebe_broker_1 | “beta” : 6,
zeebe_broker_1 | “initialLimit” : 20
zeebe_broker_1 | },
zeebe_broker_1 | “gradient” : {
zeebe_broker_1 | “minLimit” : 10,
zeebe_broker_1 | “initialLimit” : 20,
zeebe_broker_1 | “rttTolerance” : 2.0
zeebe_broker_1 | },
zeebe_broker_1 | “gradient2” : {
zeebe_broker_1 | “minLimit” : 10,
zeebe_broker_1 | “initialLimit” : 20,
zeebe_broker_1 | “rttTolerance” : 2.0,
zeebe_broker_1 | “longWindow” : 600
zeebe_broker_1 | }
zeebe_broker_1 | },
zeebe_broker_1 | “experimental” : {
zeebe_broker_1 | “maxAppendsPerFollower” : 2,
zeebe_broker_1 | “maxAppendBatchSize” : “0MB”,
zeebe_broker_1 | “disableExplicitRaftFlush” : false,
zeebe_broker_1 | “detectReprocessingInconsistency” : false,
zeebe_broker_1 | “maxAppendBatchSizeInBytes” : 32768
zeebe_broker_1 | },
zeebe_broker_1 | “stepTimeout” : “PT5M”,
zeebe_broker_1 | “executionMetricsExporterEnabled” : false
zeebe_broker_1 | }

I tested there is no connection issue between gateway and broker using telnet client. I could not understand why i am getting timeout exception when i try to deploy flow?

BR

ok found the issue i forgot to remove zeebe_networks and driver part and networks definition. I just added network_mode: host and it worked