Zeebe broker on AWS eks and AWS Opensearch issue

robertalison · May 27, 2024, 3:30pm

Hello, could someone please direct what the problem is? Got logs from zeebe broker pod which produce errors and helm parameters. I can’t figure out why it doesn’t work with Opensearch and it works with elasticsearch internally.

helm upgrade --install --kubeconfig kubeconfig
camunda camunda-platform
–repo https://helm.camunda.io
–version “10.0.5”
–namespace camunda
–create-namespace
–set identityKeycloak.postgresql.enabled=false
–set identityKeycloak.externalDatabase.host=“host”
–set identityKeycloak.externalDatabase.user=“user”
–set identityKeycloak.externalDatabase.password=“pass”
–set identityKeycloak.externalDatabase.database=“keycloak”
–set global.ingress.enabled=false
–set global.elasticsearch.enabled=false
–set elasticsearch.enabled=false
–set global.opensearch.enabled=true
–set global.opensearch.aws.enabled=true
–set global.opensearch.auth.username=“user”
–set global.opensearch.auth.password=“pass”
–set global.opensearch.url.protocol=“https”
–set global.opensearch.url.host=“endpoint”
–set global.opensearch.url.port=443

2024-05-27 15:05:24.313 [] [main] [] ERROR
      org.springframework.boot.SpringApplication - Application run failed
org.yaml.snakeyaml.parser.ParserException: while parsing a block mapping
 in 'reader', line 3, column 5:
        exporters: {}
        ^
expected <block end>, but found '<block mapping start>'
 in 'reader', line 4, column 7:
          opensearch:
          ^

        at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:654) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:161) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:57) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.comments.CommentEventsCollector$1.peek(CommentEventsCollector.java:43) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:136) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.comments.CommentEventsCollector.collectEvents(CommentEventsCollector.java:116) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:330) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:209) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:369) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:348) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:323) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:209) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:369) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:348) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:323) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:209) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeValueNode(Composer.java:369) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeMappingChildren(Composer.java:348) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeMappingNode(Composer.java:323) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.composeNode(Composer.java:209) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.composer.Composer.getNode(Composer.java:131) ~[snakeyaml-2.2.jar:?]
        at org.yaml.snakeyaml.constructor.BaseConstructor.getData(BaseConstructor.java:162) ~[snakeyaml-2.2.jar:?]
        at org.springframework.boot.env.OriginTrackedYamlLoader$OriginTrackingConstructor.getData(OriginTrackedYamlLoader.java:99) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.yaml.snakeyaml.Yaml$1.next(Yaml.java:515) ~[snakeyaml-2.2.jar:?]
        at org.springframework.beans.factory.config.YamlProcessor.process(YamlProcessor.java:203) ~[spring-beans-6.1.6.jar:6.1.6]
        at org.springframework.beans.factory.config.YamlProcessor.process(YamlProcessor.java:169) ~[spring-beans-6.1.6.jar:6.1.6]
        at org.springframework.boot.env.OriginTrackedYamlLoader.load(OriginTrackedYamlLoader.java:84) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.env.YamlPropertySourceLoader.load(YamlPropertySourceLoader.java:50) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.StandardConfigDataLoader.load(StandardConfigDataLoader.java:54) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.StandardConfigDataLoader.load(StandardConfigDataLoader.java:36) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.ConfigDataLoaders.load(ConfigDataLoaders.java:96) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.ConfigDataImporter.load(ConfigDataImporter.java:132) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.ConfigDataImporter.resolveAndLoad(ConfigDataImporter.java:87) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.ConfigDataEnvironmentContributors.withProcessedImports(ConfigDataEnvironmentContributors.java:115) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.ConfigDataEnvironment.processInitial(ConfigDataEnvironment.java:242) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.ConfigDataEnvironment.processAndApply(ConfigDataEnvironment.java:229) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.ConfigDataEnvironmentPostProcessor.postProcessEnvironment(ConfigDataEnvironmentPostProcessor.java:96) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.config.ConfigDataEnvironmentPostProcessor.postProcessEnvironment(ConfigDataEnvironmentPostProcessor.java:89) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:109) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:94) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:185) ~[spring-context-6.1.6.jar:6.1.6]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:178) ~[spring-context-6.1.6.jar:6.1.6]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:156) ~[spring-context-6.1.6.jar:6.1.6]
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:138) ~[spring-context-6.1.6.jar:6.1.6]
        at org.springframework.boot.context.event.EventPublishingRunListener.multicastInitialEvent(EventPublishingRunListener.java:136) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:81) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:64) ~[spring-boot-3.2.5.jar:3.2.5]
        at java.base/java.lang.Iterable.forEach(Unknown Source) ~[?:?]
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:118) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:112) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:63) ~[spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:369) [spring-boot-3.2.5.jar:3.2.5]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:329) [spring-boot-3.2.5.jar:3.2.5]
        at io.camunda.zeebe.broker.StandaloneBroker.main(StandaloneBroker.java:82) [camunda-zeebe-8.5.1.jar:8.5.1]

ruigrok · June 3, 2024, 1:31pm

I can confirm, having the same issue as OP.
I am using Opensearch too.

The issue was not there in version 10.0.2, after an upgrade to 10.0.5 it started to occur. When relaunching on 10.0.4, the problem disappeared, so it seems to be a bug in 10.0.5.

nathan.loding · June 3, 2024, 7:20pm

Hi @robertalison (welcome to the forums!) and @ruigrok - this is a bug that the engineering team has already fixed. The fix will be included in the next release of the Helm charts (which is some time next week I believe). If you need a fix sooner, you can use v10.0.4 as ruigrok noted, or you can clone the repository and use the main branch to deploy from.

github.com/camunda/camunda-platform-helm

[ISSUE] Disabling Elasticsearch and enabling OpenSearch causes incorrect syntax in zeebe configmap

opened 01:34PM - 28 May 24 UTC

closed 04:06PM - 28 May 24 UTC

hamza-m-masood

kind/bug kind/issue version/next support

Support ticket: https://jira.camunda.com/browse/SUPPORT-22100 **Describe the …issue:** When Elasticsearch is disabled and OpenSearch is enabled then the resulting zeebe configmap syntax is incorrect. **Actual behavior:** The zeebe configmap has an extra `{}` that should not be included after the `exporters` key: ``` data: application.yaml: | zeebe: broker: exporters: {} # invalid child element opensearch: # in a block mapping className: "io.camunda.zeebe.exporter.opensearch.OpensearchExporter" .... ``` **Expected behavior:** I expect to have correct syntax for the zeebe config when OpenSearch is enabled. **How to reproduce:**  Run the following command: ``` helm template test camunda/camunda-platform --version 10.0.5 --set global.elasticsearch.enabled=false,global.opensearch.enabled=true,global.opensearch.url.host=test > helm.yaml ``` Then check the content of `helm.yaml` for the above syntax error **Logs:** **Environment:** **Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.** - Platform: - Helm CLI version: 3.15.0 - Chart version: 10.0.5 - Values file:

system · June 10, 2024, 7:21pm

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.