Error running Zeebe 0.21.1

Camunda 8 Topics Discussion & Questions
1

Hi
I recently pulled down Zeebe using the zeebe-docker-compose / operate config. I was able to get Zeebe and Operate running locally fairly easily.

Yesterday I tried to pull the latest Zeebe (0.21.1) image from DockerHub (e.g. docker run camunda/zeebe:latest) and although Zeebe seems to install and run I get the following error when I use the zbctl tool to upload a bpmn file:

Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp 127.0.0.1:26500: connectex: No connection could be made because the target machine actively refused it.”

If I run the version of Zeebe (0.20.0) pulled by the operate/docker-compose file everything works as expected.

Do I need to remove any containers installed using the zeebe-docker-compose config before attempting to run the latest version of Zeebe?

Also, will the zeebe-docker-compose files be updated to pull the latest version of Zeebe?

I ran docker logs for the container, output is below.

Thanks
Andy.

Summary
  • configFile=/usr/local/zeebe/conf/zeebe.cfg.toml
    ++ hostname -i
  • export ZEEBE_HOST=172.17.0.2
  • ZEEBE_HOST=172.17.0.2
  • export ZEEBE_GATEWAY_CLUSTER_HOST=172.17.0.2
  • ZEEBE_GATEWAY_CLUSTER_HOST=172.17.0.2
  • ‘[’ false = true ‘]’
  • exec /usr/local/zeebe/bin/broker
    2019-11-01 08:37:04.557 [] [main] INFO io.zeebe.broker.system - Version: 0.21.1
    2019-11-01 08:37:04.570 [] [main] INFO io.zeebe.broker.system - Starting broker with configuration {
    “network”: {
    “host”: “172.17.0.2”,
    “portOffset”: 0,
    “maxMessageSize”: “4M”,
    “commandApi”: {
    “host”: “172.17.0.2”,
    “port”: 26501
    },
    “internalApi”: {
    “host”: “172.17.0.2”,
    “port”: 26502
    },
    “monitoringApi”: {
    “host”: “172.17.0.2”,
    “port”: 9600
    }
    },
    “cluster”: {
    “initialContactPoints”: [],
    “partitionIds”: [
    1
    ],
    “nodeId”: 0,
    “partitionsCount”: 1,
    “replicationFactor”: 1,
    “clusterSize”: 1,
    “clusterName”: “zeebe-cluster”
    },
    “threads”: {
    “cpuThreadCount”: 2,
    “ioThreadCount”: 2
    },
    “data”: {
    “directories”: [
    “/usr/local/zeebe/data”
    ],
    “logSegmentSize”: “512M”,
    “snapshotPeriod”: “15m”,
    “raftSegmentSize”: “512M”,
    “maxSnapshots”: 3
    },
    “exporters”: [],
    “gateway”: {
    “enable”: true,
    “network”: {
    “host”: “0.0.0.0”,
    “port”: 26500
    },
    “cluster”: {
    “contactPoint”: “172.17.0.2:26502”,
    “maxMessageSize”: “4M”,
    “requestTimeout”: “15s”,
    “clusterName”: “zeebe-cluster”,
    “memberId”: “gateway”,
    “host”: “172.17.0.2”,
    “port”: 26502
    },
    “threads”: {
    “managementThreads”: 1
    },
    “monitoring”: {
    “enabled”: false,
    “host”: “172.17.0.2”,
    “port”: 9600
    },
    “security”: {
    “enabled”: false
    }
    },
    “backpressure”: {
    “enabled”: true,
    “useWindowed”: true,
    “algorithm”: “vegas”
    }
    }
    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by org.agrona.nio.TransportPoller (file:/usr/local/zeebe/lib/agrona-1.0.7.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
    WARNING: Please consider reporting this to the maintainers of org.agrona.nio.TransportPoller
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    2019-11-01 08:37:04.815 [service-controller] [172.17.0.2:26501-zb-actors-1] INFO io.zeebe.transport - Bound commandApi.server to /172.17.0.2:26501
    2019-11-01 08:37:06.801 [] [zb-blocking-task-runner-1-172.17.0.2:26501] INFO io.zeebe.gateway - Version: 0.21.1
    2019-11-01 08:37:06.804 [] [zb-blocking-task-runner-1-172.17.0.2:26501] INFO io.zeebe.gateway - Starting gateway with configuration {
    “enable”: true,
    “network”: {
    “host”: “0.0.0.0”,
    “port”: 26500
    },
    “cluster”: {
    “contactPoint”: “172.17.0.2:26502”,
    “maxMessageSize”: “4M”,
    “requestTimeout”: “15s”,
    “clusterName”: “zeebe-cluster”,
    “memberId”: “gateway”,
    “host”: “172.17.0.2”,
    “port”: 26502
    },
    “threads”: {
    “managementThreads”: 1
    },
    “monitoring”: {
    “enabled”: false,
    “host”: “172.17.0.2”,
    “port”: 9600
    },
    “security”: {
    “enabled”: false
    }
    }
    2019-11-01 08:37:06.972 [io.zeebe.gateway.impl.broker.cluster.BrokerTopologyManagerImpl] [172.17.0.2:26501-zb-actors-1] INFO io.zeebe.transport.endpoint - Registering endpoint for node ‘0’ with address ‘172.17.0.2:26501’ on transport ‘gateway-broker-client’
    2019-11-01 08:37:07.254 [] [raft-server-system-partition-1] WARN io.atomix.utils.event.ListenerRegistry - Listener io.atomix.protocols.raft.roles.FollowerRole$$Lambda$416/0x0000000100488c40@5280cec3 not registered
    2019-11-01 08:37:07.575 [] [raft-server-raft-atomix-partition-1] WARN io.atomix.utils.event.ListenerRegistry - Listener io.atomix.protocols.raft.roles.FollowerRole$$Lambda$416/0x0000000100488c40@8998bdb not registered
    2019-11-01 08:37:08.913 [io.zeebe.broker.system.monitoring.BrokerHealthCheckService] [172.17.0.2:26501-zb-actors-1] INFO io.zeebe.broker.system - Partition ‘1’ is installed.
    2019-11-01 08:37:08.917 [io.zeebe.broker.system.monitoring.BrokerHealthCheckService] [172.17.0.2:26501-zb-actors-1] INFO io.zeebe.broker.system - All partitions are installed. Broker is ready!
    2019-11-01 08:37:08.985 [partition-1-processor] [172.17.0.2:26501-zb-actors-1] INFO io.zeebe.logstreams - Recovered state of partition 1 from snapshot at position -1
2

@andyh let me try to help you… how did you run the images? did you update your docker compose to use the latest tag of the images? because even if you pull the latest tag but the docker compose file is not pointing to those images it is not going to work.

Then… I guess that the zbctl problem is a different one… did you downloaded the latest release of zbctl ?

3

Breaking change - see here: https://zeebe.io/blog/2019/10/0.21-release/#tls

4

@salaboy - I have run the latest version of Zeebe using the following command: docker run camunda/zeebe:latest
I originally ran Zeebe using the zeebe-docker-compose, but none of those containers are running while I attempt to run the latest version of Zeebe. Should I update the zeebe-docker-compose/operate file with the latest version tag and try running .\operate docker-compose up?

I’m current running zbctl 0.20.0 (commit: 1ddfb216).

@jwulf - I have tried to run the zbctl using the --insecure e.g. .\zbctl deploy C:\bpmn\approval-process.bpmn --insecure

But I get the following error: Error: unknown flag: --insecure

I also added the ZEEBE_INSECURE_CONNECTION=true env var but I still get the error above when I attempt to deploy a bpmn file.

TLS is disabled by default in the broker, but enabled by default in the Go and Java clients, as well as zbctl .
This seems to suggest that the broker / gateway does not have TLS enabled, so as I have not updated my zbctl client will it not default to using an insecure connection?

Thanks for your help with this.
Andy

5

@andyh I think that you need to update your version of zbctl

1 Like
6

Either that or your docker run command is not mapping the port correctly. What is the command you are using to start it?

7

Ive installed the latest version of zbctl, I was using the version that came with the zeebe-docker-compose repo .
I ran npm i -g zbctl and now when I run zbctl version I get:
zbctl development (commit: 8cf92c8b)

When I run zbctl status --insecure I still get the following error:
Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp 127.0.0.1:26500: connectex: No connection could be made because the target machine actively refused it.”

The docker run command is:
docker run camunda/zeebe:latest

The output is:

  • configFile=/usr/local/zeebe/conf/zeebe.cfg.toml
    ++ hostname -i
  • export ZEEBE_HOST=172.17.0.2
  • ZEEBE_HOST=172.17.0.2
  • export ZEEBE_GATEWAY_CLUSTER_HOST=172.17.0.2
  • ZEEBE_GATEWAY_CLUSTER_HOST=172.17.0.2
  • ‘[’ false = true ‘]’
  • exec /usr/local/zeebe/bin/broker
    2019-11-01 17:32:17.363 [] [main] INFO io.zeebe.broker.system - Version: 0.21.1
    2019-11-01 17:32:17.396 [] [main] INFO io.zeebe.broker.system - Starting broker with configuration {
    “network”: {
    “host”: “172.17.0.2”,
    “portOffset”: 0,
    “maxMessageSize”: “4M”,
    “commandApi”: {
    “host”: “172.17.0.2”,
    “port”: 26501
    },
    “internalApi”: {
    “host”: “172.17.0.2”,
    “port”: 26502
    },
    “monitoringApi”: {
    “host”: “172.17.0.2”,
    “port”: 9600
    }
    },
    “cluster”: {
    “initialContactPoints”: [],
    “partitionIds”: [
    1
    ],
    “nodeId”: 0,
    “partitionsCount”: 1,
    “replicationFactor”: 1,
    “clusterSize”: 1,
    “clusterName”: “zeebe-cluster”
    },
    “threads”: {
    “cpuThreadCount”: 2,
    “ioThreadCount”: 2
    },
    “data”: {
    “directories”: [
    “/usr/local/zeebe/data”
    ],
    “logSegmentSize”: “512M”,
    “snapshotPeriod”: “15m”,
    “raftSegmentSize”: “512M”,
    “maxSnapshots”: 3
    },
    “exporters”: [],
    “gateway”: {
    “enable”: true,
    “network”: {
    “host”: “0.0.0.0”,
    “port”: 26500
    },
    “cluster”: {
    “contactPoint”: “172.17.0.2:26502”,
    “maxMessageSize”: “4M”,
    “requestTimeout”: “15s”,
    “clusterName”: “zeebe-cluster”,
    “memberId”: “gateway”,
    “host”: “172.17.0.2”,
    “port”: 26502
    },
    “threads”: {
    “managementThreads”: 1
    },
    “monitoring”: {
    “enabled”: false,
    “host”: “172.17.0.2”,
    “port”: 9600
    },
    “security”: {
    “enabled”: false
    }
    },
    “backpressure”: {
    “enabled”: true,
    “useWindowed”: true,
    “algorithm”: “vegas”
    }
    }
    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by org.agrona.nio.TransportPoller (file:/usr/local/zeebe/lib/agrona-1.0.7.jar) to field sun.nio.ch.SelectorImpl.selectedKeys
    WARNING: Please consider reporting this to the maintainers of org.agrona.nio.TransportPoller
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    2019-11-01 17:32:17.668 [service-controller] [172.17.0.2:26501-zb-actors-0] INFO io.zeebe.transport - Bound commandApi.server to /172.17.0.2:26501
    2019-11-01 17:32:19.467 [] [zb-blocking-task-runner-1-172.17.0.2:26501] INFO io.zeebe.gateway - Version: 0.21.1
    2019-11-01 17:32:19.470 [] [zb-blocking-task-runner-1-172.17.0.2:26501] INFO io.zeebe.gateway - Starting gateway with configuration {
    “enable”: true,
    “network”: {
    “host”: “0.0.0.0”,
    “port”: 26500
    },
    “cluster”: {
    “contactPoint”: “172.17.0.2:26502”,
    “maxMessageSize”: “4M”,
    “requestTimeout”: “15s”,
    “clusterName”: “zeebe-cluster”,
    “memberId”: “gateway”,
    “host”: “172.17.0.2”,
    “port”: 26502
    },
    “threads”: {
    “managementThreads”: 1
    },
    “monitoring”: {
    “enabled”: false,
    “host”: “172.17.0.2”,
    “port”: 9600
    },
    “security”: {
    “enabled”: false
    }
    }
    2019-11-01 17:32:19.629 [io.zeebe.gateway.impl.broker.cluster.BrokerTopologyManagerImpl] [172.17.0.2:26501-zb-actors-1] INFO io.zeebe.transport.endpoint - Registering endpoint for node ‘0’ with address ‘172.17.0.2:26501’ on transport ‘gateway-broker-client’
    2019-11-01 17:32:19.934 [] [raft-server-system-partition-1] WARN io.atomix.utils.event.ListenerRegistry - Listener io.atomix.protocols.raft.roles.FollowerRole$$Lambda$426/0x0000000100487040@299ca83 not registered
    2019-11-01 17:32:20.222 [] [raft-server-raft-atomix-partition-1] WARN io.atomix.utils.event.ListenerRegistry - Listener io.atomix.protocols.raft.roles.FollowerRole$$Lambda$426/0x0000000100487040@5662b4ed not registered
    2019-11-01 17:32:21.683 [io.zeebe.broker.system.monitoring.BrokerHealthCheckService] [172.17.0.2:26501-zb-actors-0] INFO io.zeebe.broker.system - Partition ‘1’ is installed.
    2019-11-01 17:32:21.690 [io.zeebe.broker.system.monitoring.BrokerHealthCheckService] [172.17.0.2:26501-zb-actors-0] INFO io.zeebe.broker.system - All partitions are installed. Broker is ready!
    2019-11-01 17:32:21.870 [partition-1-processor] [172.17.0.2:26501-zb-actors-0] INFO io.zeebe.logstreams - Recovered state of partition 1 from snapshot at position -1
8

docker run camunda/zeebe:0.21.1 -p 26500:26500

9

I ran docker run camunda/zeebe:0.21.1 -p 26500:26500 but I still get the same error.

So, I have removed all containers and images and re-ran the command above, but I still got the same error when I ran zbctl status --insecure:

Error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = “transport: Error while dialing dial tcp 127.0.0.1:26500: connectex: No connection could be made because the target machine actively refused it.”

I tried running the same command but with the container IP e.g. zbctl status --address 172.17.0.2:26500 --insecure which failed but with a different error:
Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded

So, I updated the zeebe-docker-compose/operate/docker-compose.yml to:

services:
zeebe:
image: camunda/zeebe:0.21.1

And I can now access the Zeebe Broker! :slight_smile:

When I run the command zbctl status --insecure I get the following response:

Cluster size: 1
Partitions count: 1
Replication factor: 1
Brokers:
Broker 0 - 172.19.0.4:26501
Partition 1 : Leader

Thanks for you help with this guys, greatly appreciated :+1:

2 Likes
10

Awesome!