Zeebe operator failed to connect to hosted elasticsearch

Camunda 8 Topics Discussion & Questions
1

Hi

We are using camunda-platform-8.4.9 , using hosted elasticsearch, operator is failing to connect to hosted elasticsearch with the below exceptions
updated below environment variables in helm opertor chart

CAMUNDA_OPERATE_ELASTICSEARCH_URL
CAMUNDA_OPERATE_ELASTICSEARCH_USERNAME
CAMUNDA_OPERATE_ELASTICSEARCH_PASSWORD
CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_USERNAME
CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_PASSWORD

Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [https://elasticsearch-logging-host], URI [/_component_template/operate_template?master_timeout=30s], status line [HTTP/1.1 403 Forbidden]

403 Forbidden

403 Forbidden

nginx 
    at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:347) ~[elasticsearch-rest-client-7.17.16.jar!/:7.17.16]
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:313) ~[elasticsearch-rest-client-7.17.16.jar!/:7.17.16]
    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:288) ~[elasticsearch-rest-client-7.17.16.jar!/:7.17.16]
    at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2699) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
    at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
    ... 44 more

2024-08-09 09:53:46.730 WARN 8 — [ main] i.c.o.u.RetryOperation : Retry Operation CreateComponentTemplate operate_template failed: Unable to parse response body

org.elasticsearch.ElasticsearchStatusException: Unable to parse response body
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2464) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.ClusterClient.putComponentTemplate(ClusterClient.java:274) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.lambda$createComponentTemplate$14(RetryElasticsearchClient.java:284) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.util.RetryOperation.retry(RetryOperation.java:118) ~[operate-common-8.4.10.jar!/:8.4.10]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.executeWithGivenRetries(RetryElasticsearchClient.java:556) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.executeWithRetries(RetryElasticsearchClient.java:543) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.executeWithRetries(RetryElasticsearchClient.java:539) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.createComponentTemplate(RetryElasticsearchClient.java:282) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.schema.elasticsearch.ElasticsearchSchemaManager.createDefaults(ElasticsearchSchemaManager.java:182) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.schema.elasticsearch.ElasticsearchSchemaManager.createSchema(ElasticsearchSchemaManager.java:75) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.schema.SchemaStartup.initializeSchema(SchemaStartup.java:47) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:600) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:313) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:975) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:962) [spring-context-6.1.5.jar!/:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:624) [spring-context-6.1.5.jar!/:6.1.5]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) [spring-boot-3.2.4.jar!/:3.2.4]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) [spring-boot-3.2.4.jar!/:3.2.4]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) [spring-boot-3.2.4.jar!/:3.2.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:334) [spring-boot-3.2.4.jar!/:3.2.4]
at io.camunda.operate.Application.main(Application.java:67) [!/:8.4.10]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91) [operate-webapp-8.4.10-exec.jar:8.4.10]
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53) [operate-webapp-8.4.10-exec.jar:8.4.10]
at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58) [operate-webapp-8.4.10-exec.jar:8.4.10]
Suppressed: java.lang.IllegalStateException: Unsupported Content-Type: text/html
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2481) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2461) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2184) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2154) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2118) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at org.elasticsearch.client.ClusterClient.putComponentTemplate(ClusterClient.java:274) ~[elasticsearch-rest-high-level-client-7.17.16.jar!/:7.17.16]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.lambda$createComponentTemplate$14(RetryElasticsearchClient.java:284) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.util.RetryOperation.retry(RetryOperation.java:118) ~[operate-common-8.4.10.jar!/:8.4.10]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.executeWithGivenRetries(RetryElasticsearchClient.java:556) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.executeWithRetries(RetryElasticsearchClient.java:543) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.executeWithRetries(RetryElasticsearchClient.java:539) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.store.elasticsearch.RetryElasticsearchClient.createComponentTemplate(RetryElasticsearchClient.java:282) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.schema.elasticsearch.ElasticsearchSchemaManager.createDefaults(ElasticsearchSchemaManager.java:182) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.schema.elasticsearch.ElasticsearchSchemaManager.createSchema(ElasticsearchSchemaManager.java:75) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at io.camunda.operate.schema.SchemaStartup.initializeSchema(SchemaStartup.java:47) ~[operate-els-schema-8.4.10.jar!/:8.4.10]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1778) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:600) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:522) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:326) ~[spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:324) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:313) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:200) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:975) [spring-beans-6.1.5.jar!/:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:962) [spring-context-6.1.5.jar!/:6.1.5]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:624) [spring-context-6.1.5.jar!/:6.1.5]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) [spring-boot-3.2.4.jar!/:3.2.4]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) [spring-boot-3.2.4.jar!/:3.2.4]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) [spring-boot-3.2.4.jar!/:3.2.4]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:334) [spring-boot-3.2.4.jar!/:3.2.4]
at io.camunda.operate.Application.main(Application.java:67) [!/:8.4.10]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:91) [operate-webapp-8.4.10-exec.jar:8.4.10]
at org.springframework.boot.loader.launch.Launcher.launch(Launcher.java:53) [operate-webapp-8.4.10-exec.jar:8.4.10]
at org.springframework.boot.loader.launch.JarLauncher.main(JarLauncher.java:58) [operate-webapp-8.4.10-exec.jar:8.4.10]

any support would be appreciated