Hi,
I am trying to start a cluster on multiple hosts without swarm or k8s.
So I am using the default zeebe.cfg.toml from zeebe-docker-compose/cluster example.
To start I have created three simple docker run commands to start the broker on each host:
HOST1(10.0.2.4):
docker run -d --rm --name broker1 \
-p 26500:26500 -p 26502:26502 -p 9600:9600 \
-e ZEEBE_HOST=10.0.2.4 \
-e ZEEBE_EMBED_GATEWAY=false \
-e ZEEBE_GATEWAY_CLUSTER_HOST=10.0.2.4 \
-e ZEEBE_LOG_LEVEL=debug \
-e ZEEBE_NODE_ID=0 \
-e ZEEBE_PARTITIONS_COUNT=2 \
-e ZEEBE_REPLICATION_FACTOR=3 \
-e ZEEBE_CLUSTER_SIZE=3 \
-e ZEEBE_CONTACT_POINTS=10.0.2.4:26502,10.0.2.5:26502,10.0.2.6:26502 \
-v /var/zeebe-docker-compose/cluster0/zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml \
camunda/zeebe:0.22.1
HOST2(10.0.2.6):
docker run -d --rm --name broker2 \
-p 26500:26500 -p 26502:26502 -p 9600:9600 \
-e ZEEBE_HOST=10.0.2.6 \
-e ZEEBE_EMBED_GATEWAY=false \
-e ZEEBE_GATEWAY_CLUSTER_HOST=10.0.2.6 \
-e ZEEBE_LOG_LEVEL=debug \
-e ZEEBE_NODE_ID=1 \
-e ZEEBE_PARTITIONS_COUNT=2 \
-e ZEEBE_REPLICATION_FACTOR=3 \
-e ZEEBE_CLUSTER_SIZE=3 \
-e ZEEBE_CONTACT_POINTS=10.0.2.4:26502,10.0.2.5:26502,10.0.2.6:26502 \
-v /var/zeebe-docker-compose/cluster1/zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml \
camunda/zeebe:0.22.1
HOST3(10.0.2.5):
docker run -d --rm --name broker3 \
-p 26500:26500 -p 26502:26502 -p 9600:9600 \
-e ZEEBE_HOST=10.0.2.5 \
-e ZEEBE_EMBED_GATEWAY=false \
-e ZEEBE_GATEWAY_CLUSTER_HOST=10.0.2.5 \
-e ZEEBE_LOG_LEVEL=debug \
-e ZEEBE_NODE_ID=2 \
-e ZEEBE_PARTITIONS_COUNT=2 \
-e ZEEBE_REPLICATION_FACTOR=3 \
-e ZEEBE_CLUSTER_SIZE=3 \
-e ZEEBE_CONTACT_POINTS=10.0.2.4:26502,10.0.2.5:26502,10.0.2.6:26502 \
-v /var/zeebe-docker-compose/cluster2/zeebe.cfg.toml:/usr/local/zeebe/conf/zeebe.cfg.toml \
camunda/zeebe:0.22.1
Logs from the brokers:
2020-05-18 15:26:56.512 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [6/10]: cluster services started in 18814 ms
2020-05-18 15:26:56.513 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [7/10]: topology manager
2020-05-18 15:26:56.525 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-0 [7/10]: topology manager started in 12 ms
2020-05-18 15:26:56.527 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [8/10]: metric's server
2020-05-18 15:26:56.757 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-0 [8/10]: metric's server failed with unexpected exception.
java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
2020-05-21 12:51:00.710 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-1 [6/10]: cluster services started in 7868 ms
2020-05-21 12:51:00.711 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-1 [7/10]: topology manager
2020-05-21 12:51:00.729 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-1 [7/10]: topology manager started in 17 ms
2020-05-21 12:51:00.731 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-1 [8/10]: metric's server
2020-05-21 12:51:00.766 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 0, partitions {} terms {}
2020-05-21 12:51:00.837 [Broker-1-TopologyManager] [Broker-1-zb-actors-0] DEBUG io.zeebe.broker.clustering - Received metadata change for 2, partitions {} terms {}
2020-05-21 12:51:00.892 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-1 [8/10]: metric's server failed with unexpected exception.
java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
2020-05-21 21:41:08.222 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-2 [2/10]: membership and replication protocol started in 17575 ms
2020-05-21 21:41:08.223 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-2 [3/10]: command api transport
2020-05-21 21:41:09.419 [] [main] DEBUG io.zeebe.broker.system - Bound command API to 10.0.2.5:26501
2020-05-21 21:41:09.500 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-2 [3/10]: command api transport started in 1276 ms
2020-05-21 21:41:09.502 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-2 [4/10]: command api handler
2020-05-21 21:41:09.683 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-2 [4/10]: command api handler started in 178 ms
2020-05-21 21:41:09.684 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-2 [5/10]: subscription api
2020-05-21 21:41:09.846 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-2 [5/10]: subscription api started in 157 ms
2020-05-21 21:41:09.847 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-2 [6/10]: cluster services
2020-05-21 21:41:19.681 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-2 [6/10]: cluster services started in 9833 ms
2020-05-21 21:41:19.684 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-2 [7/10]: topology manager
2020-05-21 21:41:19.700 [] [main] DEBUG io.zeebe.broker.system - Bootstrap Broker-2 [7/10]: topology manager started in 16 ms
2020-05-21 21:41:19.704 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-2 [8/10]: metric's server
2020-05-21 21:41:19.743 [Broker-2-TopologyManager] [Broker-2-zb-actors-1] DEBUG io.zeebe.broker.clustering - Received metadata change for 0, partitions {} terms {}
2020-05-21 21:41:19.949 [] [main] INFO io.zeebe.broker.system - Bootstrap Broker-2 [8/10]: metric's server failed with unexpected exception.
java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
at sun.nio.ch.Net.bind(Unknown Source) ~[?:?]
Disabling the metrics server in zeebe.cfg.toml:
# Controls if the prometheus metrics should be exporter over HTTP
# This setting can also be overridden using the environment variable ZEEBE_METRICS_HTTP_SERVER.
enableHttpServer = false
# Host to export metrics on, defaults to network.host
host = "127.0.0.1"
Did not work.
What am I missing here?
Maarten