Time out between gateway and broker: Request type command-api-1 timed out in 10000 milliseconds

bugbugmaker · September 8, 2021, 1:34am

korthout · September 8, 2021, 11:51am

Can you provide us with a bit more details? When did you see this happen and what were you trying to do? What does your configuration look like? And did you already check the Grafana dashboard for any irregularities that might indicate why a request timed out between the gateway and broker?

Also please note that this error message is often discussed on the forums. Did you already read through some of the other topics to see if any of them are related to your situation?

bugbugmaker · September 9, 2021, 5:51am

Sometimes debugging is needed. How to increase the timeout limit,

The configuration file is as follows

zeebe:
  broker:
    gateway:
      # Enable the embedded gateway to start on broker startup.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_ENABLE.
      enable: true

      network:
        # Sets the port the embedded gateway binds to.
        # This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_NETWORK_PORT.
        port: 26500

      security:
        # Enables TLS authentication between clients and the gateway
        # This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED.
        enabled: false

    network:
      # Controls the default host the broker should bind to. Can be overwritten on a
      # per binding basis for client, management and replication
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_NETWORK_HOST.
      host: 0.0.0.0

    data:
      # Specify a directory in which data is stored.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_DIRECTORY.
      directory: data
      # The size of data log segment files.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_LOGSEGMENTSIZE.
      logSegmentSize: 512MB
      # How often we take snapshots of streams (time unit)
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_SNAPSHOTPERIOD.
      snapshotPeriod: 15m

    cluster:
      # Specifies the Zeebe cluster size.
      # This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_CLUSTERSIZE.
      clusterSize: 1
      # Controls the replication factor, which defines the count of replicas per partition.
      # This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR.
      replicationFactor: 1
      # Controls the number of partitions, which should exist in the cluster.
      # This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT.
      partitionsCount: 1

    threads:
      # Controls the number of non-blocking CPU threads to be used.
      # WARNING: You should never specify a value that is larger than the number of physical cores
      # available. Good practice is to leave 1-2 cores for ioThreads and the operating
      # system (it has to run somewhere). For example, when running Zeebe on a machine
      # which has 4 cores, a good value would be 2.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_THREADS_CPUTHREADCOUNT
      cpuThreadCount: 2
      # Controls the number of io threads to be used.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_THREADS_IOTHREADCOUNT
      ioThreadCount: 2

korthout · September 9, 2021, 7:24am

This is a gateway configuration option. Since you’re using an embedded gateway, you can find it here:
https://github.com/camunda-cloud/zeebe/blob/develop/dist/src/main/config/broker.standalone.yaml.template#L55-L58. For all configuration templates, see: Configuration | Camunda 8 Docs

bugbugmaker · September 9, 2021, 7:48am

I have added the requesttimeout configuration, but it doesn’t work. The configuration file is as follows

# Zeebe Standalone Broker configuration file (with embedded gateway)

# This file is based on broker.standalone.yaml.template but stripped down to contain only a limited
# set of configuration options. These are a good starting point to get to know Zeebe.
# For advanced configuration options, have a look at the templates in this folder.

# !!! Note that this configuration is not suitable for running a standalone gateway. !!!
# If you want to run a standalone gateway node, please have a look at gateway.yaml.template

# ----------------------------------------------------
# Byte sizes
# For buffers and others must be specified as strings and follow the following
# format: "10U" where U (unit) must be replaced with KB = Kilobytes, MB = Megabytes or GB = Gigabytes.
# If unit is omitted then the default unit is simply bytes.
# Example:
# sendBufferSize = "16MB" (creates a buffer of 16 Megabytes)
#
# Time units
# Timeouts, intervals, and the likes, must be specified either in the standard ISO-8601 format used
# by java.time.Duration, or as strings with the following format: "VU", where:
#   - V is a numerical value (e.g. 1, 5, 10, etc.)
#   - U is the unit, one of: ms = Millis, s = Seconds, m = Minutes, or h = Hours
#
# Paths:
# Relative paths are resolved relative to the installation directory of the broker.
# ----------------------------------------------------

zeebe:
  broker:
    gateway:
      # Enable the embedded gateway to start on broker startup.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_ENABLE.
      enable: true

      network:
        # Sets the port the embedded gateway binds to.
        # This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_NETWORK_PORT.
        port: 26500

      security:
        # Enables TLS authentication between clients and the gateway
        # This setting can also be overridden using the environment variable ZEEBE_BROKER_GATEWAY_SECURITY_ENABLED.
        enabled: false

    network:
      # Controls the default host the broker should bind to. Can be overwritten on a
      # per binding basis for client, management and replication
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_NETWORK_HOST.
      host: 0.0.0.0

    data:
      # Specify a directory in which data is stored.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_DIRECTORY.
      directory: data
      # The size of data log segment files.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_LOGSEGMENTSIZE.
      logSegmentSize: 128MB
      # How often we take snapshots of streams (time unit)
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_DATA_SNAPSHOTPERIOD.
      snapshotPeriod: 15m

    cluster:
      # Specifies the Zeebe cluster size.
      # This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_CLUSTERSIZE.
      clusterSize: 1
      # Controls the replication factor, which defines the count of replicas per partition.
      # This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_REPLICATIONFACTOR.
      replicationFactor: 1
      # Controls the number of partitions, which should exist in the cluster.
      # This can also be overridden using the environment variable ZEEBE_BROKER_CLUSTER_PARTITIONSCOUNT.
      partitionsCount: 1
      
      requestTimeout: 120s

    threads:
      # Controls the number of non-blocking CPU threads to be used.
      # WARNING: You should never specify a value that is larger than the number of physical cores
      # available. Good practice is to leave 1-2 cores for ioThreads and the operating
      # system (it has to run somewhere). For example, when running Zeebe on a machine
      # which has 4 cores, a good value would be 2.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_THREADS_CPUTHREADCOUNT
      cpuThreadCount: 2
      # Controls the number of io threads to be used.
      # This setting can also be overridden using the environment variable ZEEBE_BROKER_THREADS_IOTHREADCOUNT
      ioThreadCount: 2

bugbugmaker · September 9, 2021, 8:09am

thank you ,I’ve solved the problem

bugbugmaker · September 9, 2021, 9:01am

Does the gateway.yaml.template configuration file not support the exporters attribute? Exporters are configured but do not work

korthout · September 9, 2021, 9:02am

I have added the requesttimeout configuration, but it doesn’t work.

Seems you configured zeebe.broker.cluster.requestTimeout, instead of zeebe.broker.gateway.cluster.requestTimeout

korthout · September 9, 2021, 9:04am

Does the gateway.yaml.template configuration file not support the exporters attribute?

Yes, the exporters can only be configured in the broker, not in the gateway.

Exporters are configured but do not work

Have a look at Configuration | Camunda 8 Docs to learn how to verify your configuration

yadav1990 · May 19, 2023, 10:54am

How did you solve this problem? Could you please update here as I am also facing the same problem

yadav1990 · May 22, 2023, 9:04am

I have the same issue, could you please help on this,

ProcessInstanceResult instanceResult = zeebeClient
			        .newCreateInstanceCommand()
			        .bpmnProcessId(processId)
			        .latestVersion()
			        .variables(processVariables)
			        .withResult()
			        .send().join();

Getting below error in springboot

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Time out between gateway and broker: Request ProtocolRequest{id=6, subject=command-api-1, sender=0.0.0.0:26502, payload=byte{length=283, hash=-951148236}} to 0.0.0.0:26501 timed out in PT10S

yadav1990 · May 23, 2023, 10:13am

Team,
Any help on this?

bugbugmaker · May 23, 2023, 10:20am

try to increase the timeout parameter value.
io.camunda.zeebe.client.api.ZeebeFuture.join(long timeout, TimeUnit)

YANGTAO · July 4, 2023, 4:07am

Have you solved this problem? I have the same problem now. How did you solve it

Karthikeyan_Raju · August 1, 2023, 9:55am

@yadav1990 have you got any solutions for the problem?

yadav1990 · August 1, 2023, 10:41am

@Karthikeyan_Raju No. And This is the existing problem in camunda 8.

TIA.

GotnOGuts · August 1, 2023, 4:55pm

If your process can take longer than the gateway timeout, then you have to use an Async model when calling it, rather than a Sync model.

You can start the process without the “withresult” which will give you the process instance ID. Then poll the Camunda engine for the outcome of the process (or design your UI server to be able to accept a message back from the Camunda engine)