Zeebe operate search locally for Elastic search

I have deployed zeebe cluster

now I want is dashboard UI for workflow?

@ashu2108 can you please format your code correctly? There is no indentation here…

configmap:

values.yaml:

global:
  zeebe: "{{ .Release.Name }}-zeebe"
  elasticsearch:
    host: "https://xxxxxxxxxxxxxxxx.xxxxx.xxxx.xxx.ap-south-1.es.amazonaws.com"
    port: 443
    username: xxxxxxxxx
    password: xxxxxxxxxxx
    clusterName: "ph1-dev-elk"

sorry I have pasted it directly so it was not showing indentations.
Thanks in Advance!!

@ashu2108 I think that you are missing the second half of it
are you setting this also:

??

I have not set for the second half but those are also using the same values as you can see both the key and values are the same.

host: {{ .Values.global.elasticsearch.host }}

Thanks in Advance!

@ashu2108 what do you mean? You need to set them both… they are different properties, and the second ones are needed (even if they are the same) for the Zeebe Client inside Operate to connect as well.

I have updated as shown below now both are using same details of elastic search.

ERROR:

2020-07-07 09:58:11.627  INFO 7 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet engine: [Apache Tomcat/9.0.33]
2020-07-07 09:58:11.842  INFO 7 --- [           main] o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext
2020-07-07 09:58:11.842  INFO 7 --- [           main] o.s.w.c.ContextLoader                    : Root WebApplicationContext: initialization completed in 6294 ms
2020-07-07 09:58:13.843  INFO 7 --- [           main] o.c.o.WebappModuleConfiguration          : Starting module: webapp
2020-07-07 09:58:13.964 DEBUG 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Creating Elasticsearch connection...
2020-07-07 09:58:14.545 ERROR 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Error occurred while connecting to Elasticsearch: clustername [ph1-dev-elk], https://xxxxxxxxx.xxxxxxxxxx.xxxxxxxx.xxxxx-.ap-south-1.es.amazonaws.com:443. Will be retried...

java.io.IOException: https://xxxxxxx.xxxxxxxx.xxxxxxx.xxxxxxx-ap-south-1.es.amazonaws.com: Name or service not known
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:146) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.camunda.operate.es.ElasticsearchConnector.checkHealth(ElasticsearchConnector.java:89) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector.createEsClient(ElasticsearchConnector.java:75) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector.esClient(ElasticsearchConnector.java:51) ~[camunda-operate-common-0.23.0.jar!/:?]

@ashu2108 ok… so now it is a network problem… nothing to do with Zeebe … you need to ask why the host that is running Operate cannot reach that amazon url… I don’t know if I can help you with that…
Maybe try without the https:// at the beginning… but again… this is not a Zeebe problem.

@salaboy okay if it is network problem but this issue is only I’m facing in operate and the same URL, I was able to configure in broker as shown below:

logs of broker:

2020-07-07 12:06:37.785 [] [main] DEBUG io.zeebe.broker.exporter - Exporter configured with ElasticsearchExporterConfiguration{url='https://xxxxxxxxxxx.xxxxxxxxxxxxxxxxx.xxxxxxxxxx.xxxxxxxxxx.xxxxxxx.ap-south-1.es.amazonaws.com:443', index=IndexConfiguration{indexPrefix='zeebe-record', createTemplate=true, command=false, event=true, rejection=false, error=true, deployment=true, incident=true, job=true, message=false, messageSubscription=false, variable=true, variableDocument=true, workflowInstance=true, workflowInstanceCreation=false, workflowInstanceSubscription=false, ignoreVariablesAbove=8191}, bulk=BulkConfiguration{delay=5, size=1000}, authentication=AuthenticationConfiguration{username='xxxxxxxxxxxx'}}

as you can see both pods are running on the same namespaces.

Correct me if I’m wrong, but If this was the network issue then the same issue broker must-have. why only operate?

Thanks in Advance!!

@ashu2108 what is the topology of your cluster? How many nodes? Are they all hosted in the same place?
I would suggest to create a simple pod which does a rest call to that URL to test, you can use an existing container with bash to just do curl to that url from your cluster. I cannot think of any reason why it will work for a broker and not for Operate, besides a difference in the way that the rest calls are done.
@svetlana have you seen something like that before?

Hi @ashu2108,

please check the value provided in {{.Values.global.elasticsearch.clusterName}}. Does it correspond to Elasticsearch cluster name? It may be found in Elasticsearch config file or by requesting GET http://<els_host>:<els_port>.

Bers regards,
Svetlana

1 Like

Hi @svetlana
Same issue with that elasticsearch cluster name also. I took that clustername from GET http://<els_host>:<els_port> . but same issue is there

Does your Elasticsearch require authentication? If yes, it won’t work with Operate currently. This feature is planned, but not yet implemented.

1 Like

@svetlana yes elastic search is private so it needs authentication and is there an alternative way we can integrate with a broker and get UI running so we can access in the browser??
Thanks for the quick response.

Thank you in Advance.

Unfortunately currently it is not possible to connect Operate to Elasticsearch with enabled authentication. You will have to find some way to make ELS host visible to only Zeebe and Operate and to close it from all other connections.

@svetlana and @salaboy can we deploy zeebe simple monitor on kubernetes for UI purpose. Actually we deployed the simple-monitor on docker and it’s working fine but same configuration is not working on kubernetes. We are facing some issues on kubernetes please check this issue Zeebe-simple-monitor is trying to connect to localhost:5701. Please guide on it.

Thanks

@svetlana we tried with public aws elastic-search which don’t take username and password. Still facing same issue
You can check the logs:


  / __| /_\ |  \/  | | | | \| |   \ /_\    / _ \| _ \ __| _ \  /_\_   _| __|
 | (__ / _ \| |\/| | |_| | .` | |) / _ \  | (_) |  _/ _||   / / _ \| | | _|
  \___/_/ \_\_|  |_|\___/|_|\_|___/_/ \_\  \___/|_| |___|_|_\/_/ \_\_| |___|

                                                           0.23.0


2020-07-08 13:24:00.453  INFO 7 --- [           main] o.c.o.Application                        : Starting Application on zeebe-operate-5dc8b97f5c-j9vvt with PID 7 (/usr/local/operate/lib/camunda-operate-webapp-0.23.0-exec.jar started by root in /usr/local/operate)
2020-07-08 13:24:00.465 DEBUG 7 --- [           main] o.c.o.Application                        : Running with Spring Boot v2.2.6.RELEASE, Spring v5.2.5.RELEASE
2020-07-08 13:24:00.469  INFO 7 --- [           main] o.c.o.Application                        : The following profiles are active: auth
2020-07-08 13:24:06.187  INFO 7 --- [           main] o.s.b.w.e.t.TomcatWebServer              : Tomcat initialized with port(s): 8080 (http)
2020-07-08 13:24:06.239  INFO 7 --- [           main] o.a.c.h.Http11NioProtocol                : Initializing ProtocolHandler ["http-nio-8080"]
2020-07-08 13:24:06.241  INFO 7 --- [           main] o.a.c.c.StandardService                  : Starting service [Tomcat]
2020-07-08 13:24:06.243  INFO 7 --- [           main] o.a.c.c.StandardEngine                   : Starting Servlet engine: [Apache Tomcat/9.0.33]
2020-07-08 13:24:06.442  INFO 7 --- [           main] o.a.c.c.C.[.[.[/]                        : Initializing Spring embedded WebApplicationContext
2020-07-08 13:24:06.442  INFO 7 --- [           main] o.s.w.c.ContextLoader                    : Root WebApplicationContext: initialization completed in 5802 ms
2020-07-08 13:24:08.185  INFO 7 --- [           main] o.c.o.WebappModuleConfiguration          : Starting module: webapp
2020-07-08 13:24:08.352 DEBUG 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Creating Elasticsearch connection...
2020-07-08 13:24:08.922 ERROR 7 --- [           main] o.c.o.e.ElasticsearchConnector           : Error occurred while connecting to Elasticsearch: clustername [400307571006:es-test], https://xxxxxxxxxxxxxxx.us-east-2.es.amazonaws.com:443. Will be retried...

java.io.IOException: https://xxxxxxxxxxxxxx.us-east-2.es.amazonaws.com: Name or service not known
        at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:964) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233) ~[elasticsearch-rest-client-6.8.7.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.elasticsearch.client.ClusterClient.health(ClusterClient.java:146) ~[elasticsearch-rest-high-level-client-6.8.8.jar!/:6.8.7]
        at org.camunda.operate.es.ElasticsearchConnector.checkHealth(ElasticsearchConnector.java:89) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector.createEsClient(ElasticsearchConnector.java:75) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector.esClient(ElasticsearchConnector.java:51) ~[camunda-operate-common-0.23.0.jar!/:?]
        at org.camunda.operate.es.ElasticsearchConnector$$EnhancerBySpringCGLIB$$98f21047.CGLIB$esClient$0(<generated>) ~[camunda-operate-common-0.23.0.jar!/:?]

When we deploy zeebe-full that time zeebe full helm chart will deploy its own elastic search on kubernetes and that configuration is working fine. But when we are replacing that elasticsearch host with our aws elasticsearch(public) , the zeebe-operate is failing.

So now i have question can we connect aws elasticsearch to zeebe-operate. Is that configuration available??

@SujataKale97 can you share more information about your kubernetes cluster? is it EKS? how do you created it?

@salaboy we tried in both k8’s cluster Azure AKS and AWS EKS.

@svetlana @salaboy After doing some R&D we found that!!!

when we deployed zeebe-full operate was working fine!!

The elastic search is deployed by zeebe-full locally in the same namespaces as you can see above, zeebe-operate is working because operate was able to find elastic search locally. After modifying values of AWS elastic search in operate configmap with or without authentication it was not working in zeebe-full also.

We tried to deploy manually using operate helm chart it is trying to find elastic search locally so the pod is going to CrashLoopBackOff state, when we specify the AWS ELK (private or public) both are not working.

as @SujataKale97 has mentioned above she tried with both private and public elastic search with or without authentication still pods was in CrashLoopBackOff state because operate didn’t find elastic search locally.

I hope you understood what I’m trying to say or you can just give a try to what I was trying to do.
Thanks in Advance!! :slight_smile:

Saying that the pod goes to CrashLoopBackOff doesn’t help much, did you saw different logs or issues in each case? It is quite hard to understand what is going wrong in your setup, unfortunately I cannot reproduce as I don’t have an AWS ElasticSearch available to test. Also you have heavily modified the configMap, so at this point it is quite hard for me to even start trying to replicate your problem.
as @svetlana mention, if you are working without authentication it should work, but without looking at the logs it is impossible to know if your pod is not finding the host or why it is not connecting.

If it is a network problem, you might need to configure an Egress to enable outbound traffic from your Kube cluster to ElasticSearch, but if you haven’t done something like that for the broker and the broker connects to the ElasticSearch I am not sure what can be wrong to your setup.