Gateway Broker Connectivity Error - Request Timeout

Camunda 8 Topics Discussion & Questions
1

I am trying to run zeebe 8.1.1 locally through docker-compose file (below) with 2 brokers and separate gateway. After start up I still see constant errors in the zeebe-gateway logs as below.

I cannot figure out what is missing in the deployment configuration. Any help would be great!

Thanks.

2023-01-05 12:50:54.236 [io.camunda.zeebe.gateway.impl.broker.BrokerRequestManager] [gateway-scheduler-zb-actors-0] DEBUG
      io.camunda.zeebe.gateway - Expected to handle gRPC request, but request timed out between gateway and broker
java.util.concurrent.TimeoutException: Request ProtocolRequest{id=85582, subject=command-api-3, sender=zeebe-gateway:26502, payload=byte[]{length=634, hash=-1454275626}} to 172.21.0.2:26501 timed out in PT3S
	at io.atomix.cluster.messaging.impl.NettyMessagingService.lambda$sendAndReceive$4(NettyMessagingService.java:230) ~[zeebe-atomix-cluster-8.1.1.jar:8.1.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
	at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.82.Final.jar:4.1.82.Final]
	at java.lang.Thread.run(Unknown Source) ~[?:?]

version: '3'
services:

  zeebe-broker-1: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#zeebe
    image: camunda/zeebe:${CAMUNDA_PLATFORM_VERSION:-8.1.1}
    container_name: zeebe-broker-1
    network_mode: camunda-platform
    environment: # https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/environment-variables/
      - ZEEBE_BROKER_GATEWAY_ENABLE=false
      - ZEEBE_BROKER_CLUSTER_NODEID=0
      - ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT=4
      - ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR=2
      - ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=2
      - ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=zeebe-broker-1:26502,zeebe-broker-2:26502
      - ZEEBE_BROKER_CLUSTER_CLUSTERNAME=zeebe-cluster
      - ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998
      - ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999
    restart: always
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: "2400M"
    volumes:
      - zeebebroker1:/usr/local/zeebe/data

  zeebe-broker-2: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#zeebe
    image: camunda/zeebe:${CAMUNDA_PLATFORM_VERSION:-8.1.1}
    container_name: zeebe-broker-2
    network_mode: camunda-platform
    environment: # https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/environment-variables/
      - ZEEBE_BROKER_GATEWAY_ENABLE=false
      - ZEEBE_BROKER_CLUSTER_NODEID=1
      - ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT=4
      - ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR=2
      - ZEEBE_BROKER_CLUSTER_CLUSTERSIZE=2
      - ZEEBE_BROKER_CLUSTER_INITIALCONTACTPOINTS=zeebe-broker-1:26502,zeebe-broker-2:26502
      - ZEEBE_BROKER_CLUSTER_CLUSTERNAME=zeebe-cluster
      - ZEEBE_BROKER_DATA_DISKUSAGECOMMANDWATERMARK=0.998
      - ZEEBE_BROKER_DATA_DISKUSAGEREPLICATIONWATERMARK=0.999
    restart: always
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: "2400M"
    volumes:
      - zeebebroker2:/usr/local/zeebe/data

  zeebe-gateway: # https://docs.camunda.io/docs/self-managed/platform-deployment/docker/#zeebe
    image: camunda/zeebe:${CAMUNDA_PLATFORM_VERSION:-8.1.1}
    container_name: zeebe-gateway
    network_mode: camunda-platform
    ports:
      - "26500:26500"
      - "26502:26502"
    environment: # https://docs.camunda.io/docs/self-managed/zeebe-deployment/configuration/environment-variables/
      - ZEEBE_LOG_LEVEL=debug
      - ZEEBE_STANDALONE_GATEWAY=true
      - ZEEBE_GATEWAY_NETWORK_HOST=0.0.0.0
      - ZEEBE_GATEWAY_NETWORK_PORT=26500
      - ZEEBE_GATEWAY_CLUSTER_INITIALCONTACTPOINTS=zeebe-broker-1:26502,zeebe-broker-2:26502,zeebe-broker-3:26502
      - ZEEBE_GATEWAY_CLUSTER_PORT=26502
      - ZEEBE_GATEWAY_CLUSTER_HOST=zeebe-gateway
      - ZEEBE_GATEWAY_CLUSTER_CLUSTERNAME=zeebe-cluster
      - ZEEBE_GATEWAY_CLUSTER_MEMBERID=gateway
      - "JAVA_TOOL_OPTIONS=-Xms512m -Xmx512m"
    restart: always
    deploy:
      resources:
        limits:
          cpus: "1.0"
          memory: "768M"
    volumes:
      - zeebe:/usr/local/zeebe/data
    depends_on:
      - elasticsearch
      - zeebe-broker-1
      - hazelcast-1

volumes:
  zeebe:
  zeebebroker1:
  zeebebroker2:
2

On troubleshooting further I realised that this error comes from the benchmarking tool. The tool starts the process instance while waiting for results. The wait timeout is hard coded to 3 secs. When there are many processes started the process execution time crosses 3 secs and hence the error.
Changing the timeout value fixes the problem

3