Zeebe 0.23.0 cannot сonnection to elasticsearch?

Camunda 8 Topics Discussion & Questions
1

I decided the updated version of Zeebe from 0.22.1 to 0.23.0

I used to run docker-compose.yml from the official documentation, there I updated the versions zeebe and operate, replaced zeebe.cfg.toml with application.yaml. After that, the broker cannot export data to elasticsearch.

docker-compose.yml:

version: "2"

networks:
  zeebe_network:

volumes:
  zeebe_data:
  zeebe_elasticsearch_data:

services:
  zeebe:
    image: camunda/zeebe:0.23.0
    ports:
      - "26500:26500"
      - "9600:9600"
    volumes:
      - zeebe_data:/usr/local/zeebe/data
      - ./application.yaml:/usr/local/zeebe/config/application.yaml
    depends_on:
      - elasticsearch
    networks:
      - zeebe_network
  operate:
    image: camunda/operate:0.23.0
    ports:
      - "8080:8080"
    volumes:
      - ../lib/application.yml:/usr/local/operate/config/application.yml
    networks:
      - zeebe_network
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.1
    ports:
      - "9200:9200"
    environment:
      - discovery.type=single-node
      - cluster.name=elasticsearch
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - zeebe_elasticsearch_data:/usr/share/elasticsearch/data
    networks:
      - zeebe_network

log zeebe:

2020-04-27 09:54:32.891 [] [main] INFO  io.zeebe.broker.StandaloneBroker - Starting StandaloneBroker v0.23.1 on f1783073c055 with PID 6 (/usr/local/zeebe/lib/zeebe-distribution-0.23.1.jar started by root in /usr/local/zeebe)
2020-04-27 09:54:32.906 [] [main] INFO  io.zeebe.broker.StandaloneBroker - No active profile set, falling back to default profiles: default
2020-04-27 09:54:38.231 [] [main] INFO  io.zeebe.broker.StandaloneBroker - Started StandaloneBroker in 7.003 seconds (JVM running for 11.356)
2020-04-27 09:54:38.403 [] [main] INFO  io.zeebe.broker.system - Version: 0.23.1
2020-04-27 09:54:38.657 [] [main] INFO  io.zeebe.broker.system - Starting broker 0 with configuration {
  "network" : {
    "host" : "172.21.0.4",
    "portOffset" : 0,
    "maxMessageSize" : "4MB",
    "advertisedHost" : "0.0.0.0",
    "commandApi" : {
      "host" : "0.0.0.0",
      "port" : 26501,
      "advertisedHost" : "0.0.0.0",
      "advertisedPort" : 26501,
      "advertisedAddress" : "0.0.0.0:26501",
      "address" : "0.0.0.0:26501"
    },
    "internalApi" : {
      "host" : "172.21.0.4",
      "port" : 26502,
      "advertisedHost" : "0.0.0.0",
      "advertisedPort" : 26502,
      "advertisedAddress" : "0.0.0.0:26502",
      "address" : "172.21.0.4:26502"
    },
    "monitoringApi" : {
      "host" : "0.0.0.0",
      "port" : 9600,
      "advertisedHost" : "0.0.0.0",
      "advertisedPort" : 9600,
      "advertisedAddress" : "0.0.0.0:9600",
      "address" : "0.0.0.0:9600"
    },
    "maxMessageSizeInBytes" : 4194304
  },
  "cluster" : {
    "initialContactPoints" : [ ],
    "partitionIds" : [ 1 ],
    "nodeId" : 0,
    "partitionsCount" : 1,
    "replicationFactor" : 1,
    "clusterSize" : 1,
    "clusterName" : "zeebe-cluster",
    "gossipFailureTimeout" : 10000,
    "gossipInterval" : 250,
    "gossipProbeInterval" : 1000
  },
  "threads" : {
    "cpuThreadCount" : 2,
    "ioThreadCount" : 2
  },
  "data" : {
    "directories" : [ "/usr/local/zeebe/data" ],
    "logSegmentSize" : "512MB",
    "snapshotPeriod" : "PT15M",
    "logIndexDensity" : 100,
    "logSegmentSizeInBytes" : 536870912
  },
  "exporters" : {
    "elasticsearch" : {
      "jarPath" : null,
      "className" : "io.zeebe.exporter.ElasticsearchExporter",
      "args" : {
        "url" : "http://localhost:9200",
        "bulk" : {
          "delay" : 5,
          "size" : 1000
        },
        "index" : {
          "prefix" : "zeebe-record",
          "createTemplate" : true,
          "command" : false,
          "event" : true,
          "rejection" : false,
          "deployment" : true,
          "incident" : true,
          "job" : true,
          "message" : false,
          "messageSubscription" : false,
          "raft" : false,
          "workflowInstance" : true,
          "workflowInstanceSubscription" : false
        }
      },
      "external" : false
    },
    "debuglog" : {
      "jarPath" : null,
      "className" : "io.zeebe.broker.exporter.debug.DebugLogExporter",
      "args" : {
        "logLevel" : "debug"
      },
      "external" : false
    }
  },
  "gateway" : {
    "network" : {
      "host" : "0.0.0.0",
      "port" : 26500,
      "minKeepAliveInterval" : "PT30S"
    },
    "cluster" : {
      "contactPoint" : "172.21.0.4:26502",
      "requestTimeout" : "PT15S",
      "clusterName" : "zeebe-cluster",
      "memberId" : "gateway",
      "host" : "172.21.0.4",
      "port" : 26502
    },
    "threads" : {
      "managementThreads" : 1
    },
    "monitoring" : {
      "enabled" : false,
      "host" : "172.21.0.4",
      "port" : 9600
    },
    "security" : {
      "enabled" : false,
      "certificateChainPath" : null,
      "privateKeyPath" : null
    },
    "enable" : true
  },
  "backpressure" : {
    "enabled" : true,
    "algorithm" : "VEGAS"
  },
  "stepTimeout" : "PT5M"
}
2020-04-27 09:54:38.743 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [1/11]: actor scheduler
2020-04-27 09:54:38.786 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [2/11]: membership and replication protocol
2020-04-27 09:54:46.992 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [3/11]: command api transport
2020-04-27 09:54:47.548 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [4/11]: command api handler
2020-04-27 09:54:47.629 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [5/11]: subscription api
2020-04-27 09:54:47.693 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [6/11]: embedded gateway
2020-04-27 09:54:47.703 [] [main] INFO  io.zeebe.gateway - Version: 0.23.1
2020-04-27 09:54:47.704 [] [main] INFO  io.zeebe.gateway - Starting gateway with configuration {
  "network" : {
    "host" : "0.0.0.0",
    "port" : 26500,
    "minKeepAliveInterval" : "PT30S"
  },
  "cluster" : {
    "contactPoint" : "172.21.0.4:26502",
    "requestTimeout" : "PT15S",
    "clusterName" : "zeebe-cluster",
    "memberId" : "gateway",
    "host" : "172.21.0.4",
    "port" : 26502
  },
  "threads" : {
    "managementThreads" : 1
  },
  "monitoring" : {
    "enabled" : false,
    "host" : "172.21.0.4",
    "port" : 9600
  },
  "security" : {
    "enabled" : false,
    "certificateChainPath" : null,
    "privateKeyPath" : null
  },
  "enable" : true
}
2020-04-27 09:54:48.086 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [7/11]: cluster services
2020-04-27 09:54:49.371 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [8/11]: topology manager
2020-04-27 09:54:49.375 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [9/11]: metric's server
2020-04-27 09:54:49.431 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [10/11]: leader management request handler
2020-04-27 09:54:49.436 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 [11/11]: zeebe partitions
2020-04-27 09:54:49.443 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions [1/1]: partition 1
2020-04-27 09:54:51.304 [Broker-0-StreamProcessor-1] [Broker-0-zb-actors-0] INFO  org.camunda.feel.FeelEngine - Engine created. [value-mapper: CompositeValueMapper(List(io.zeebe.el.impl.feel.MessagePackValueMapper@271ff402)), function-provider: io.zeebe.el.impl.feel.FeelFunctionProvider@566f54ce, configuration: Configuration(false)]
2020-04-27 09:54:51.767 [Broker-0-StreamProcessor-1] [Broker-0-zb-actors-0] INFO  io.zeebe.logstreams - Recovered state of partition 1 from snapshot at position -1
2020-04-27 09:54:51.831 [Broker-0-StreamProcessor-1] [Broker-0-zb-actors-0] INFO  org.camunda.feel.FeelEngine - Engine created. [value-mapper: CompositeValueMapper(List(io.zeebe.el.impl.feel.MessagePackValueMapper@441d7f3a)), function-provider: io.zeebe.el.impl.feel.FeelFunctionProvider@66a17361, configuration: Configuration(false)]
2020-04-27 09:54:51.854 [Broker-0-StreamProcessor-1] [Broker-0-zb-actors-0] INFO  org.camunda.feel.FeelEngine - Engine created. [value-mapper: CompositeValueMapper(List(io.zeebe.el.impl.feel.MessagePackValueMapper@746850ea)), function-provider: io.zeebe.el.impl.feel.FeelFunctionProvider@1c28bdff, configuration: Configuration(false)]
2020-04-27 09:54:52.154 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 partitions succeeded. Started 1 steps in 2711 ms.
2020-04-27 09:54:52.155 [] [main] INFO  io.zeebe.broker.system - Bootstrap Broker-0 succeeded. Started 11 steps in 13412 ms.
2020-04-27 09:54:52.343 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-0] INFO  io.zeebe.broker.exporter.elasticsearch - Exporter opened
2020-04-27 10:02:04.117 [Broker-0-Exporter-1] [Broker-0-zb-fs-workers-1] ERROR io.zeebe.broker.exporter.elasticsearch - Error on exporting record with key 2251799813685252
io.zeebe.exporter.ElasticsearchExporterException: Failed to put index template
	at io.zeebe.exporter.ElasticsearchClient.putIndexTemplate(ElasticsearchClient.java:198) ~[zeebe-elasticsearch-exporter-0.23.1.jar:0.23.1]
	at io.zeebe.exporter.ElasticsearchClient.putIndexTemplate(ElasticsearchClient.java:187) ~[zeebe-elasticsearch-exporter-0.23.1.jar:0.23.1]
	at io.zeebe.exporter.ElasticsearchExporter.createRootIndexTemplate(ElasticsearchExporter.java:174) ~[zeebe-elasticsearch-exporter-0.23.1.jar:0.23.1]
	at io.zeebe.exporter.ElasticsearchExporter.createIndexTemplates(ElasticsearchExporter.java:127) ~[zeebe-elasticsearch-exporter-0.23.1.jar:0.23.1]
	at io.zeebe.exporter.ElasticsearchExporter.export(ElasticsearchExporter.java:86) ~[zeebe-elasticsearch-exporter-0.23.1.jar:0.23.1]
	at io.zeebe.broker.exporter.stream.ExporterDirector$RecordExporter.export(ExporterDirector.java:350) ~[zeebe-broker-0.23.1.jar:0.23.1]
	at io.zeebe.util.retry.BackOffRetryStrategy.run(BackOffRetryStrategy.java:51) ~[zeebe-util-0.23.1.jar:0.23.1]
	at io.zeebe.util.sched.ActorJob.invoke(ActorJob.java:73) [zeebe-util-0.23.1.jar:0.23.1]
	at io.zeebe.util.sched.ActorJob.execute(ActorJob.java:39) [zeebe-util-0.23.1.jar:0.23.1]
	at io.zeebe.util.sched.ActorTask.execute(ActorTask.java:115) [zeebe-util-0.23.1.jar:0.23.1]
	at io.zeebe.util.sched.ActorThread.executeCurrentTask(ActorThread.java:107) [zeebe-util-0.23.1.jar:0.23.1]
	at io.zeebe.util.sched.ActorThread.doWork(ActorThread.java:91) [zeebe-util-0.23.1.jar:0.23.1]
	at io.zeebe.util.sched.ActorThread.run(ActorThread.java:195) [zeebe-util-0.23.1.jar:0.23.1]
Caused by: java.net.ConnectException: Connection refused
	at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:959) ~[elasticsearch-rest-client-6.8.8.jar:6.8.8]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.8.jar:6.8.8]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar:6.8.8]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar:6.8.8]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar:6.8.8]
	at org.elasticsearch.client.IndicesClient.putTemplate(IndicesClient.java:1504) ~[elasticsearch-rest-high-level-client-6.8.8.jar:6.8.8]
	at io.zeebe.exporter.ElasticsearchClient.putIndexTemplate(ElasticsearchClient.java:195) ~[zeebe-elasticsearch-exporter-0.23.1.jar:0.23.1]
	... 12 more
Caused by: java.net.ConnectException: Connection refused
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) ~[?:?]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.13.jar:4.4.13]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.13.jar:4.4.13]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.13.jar:4.4.13]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar:4.1.4]
	at java.lang.Thread.run(Unknown Source) ~[?:?]
2

Hey @Sparda,

could you please show your application.yaml.
When I take a look at the old zeebe.cfg.toml from the docker compose I see that the elasticsearch url is
url = "http://elasticsearch:9200".

Hope this helps.

Greets
Chris

3

Thanks it helped, I was inattentive.

2 Likes
4

I will update the zeebe-docker-compose repo shortly.

1 Like
5

Looking forward to the changes @jwulf. I do not see any errors but Operate is not displaying information from Elasticsearch in my case. I used the latest settings in the docker-compose repo as well.

6

Same case on my side, I tried updating the current docker-compose to the latest version of zeebe and operate stop displaying info from elastic search. I also tried updating operate to the latest version but still the same situation. I am looking forward for the docker compose update so I can compare and see what I am doing wrong :slight_smile:

7

Sorry guys, but what was the solution? I’m still struggling with operate isn’t showing processes

8

The Zeebe Docker Compose repo is now updated to 0.23.1: https://github.com/zeebe-io/zeebe-docker-compose

1 Like
9