I’m currently facing an issue while setting up Self-Managed Multi-tenancy Camunda8 (External Elasticsearch , External Keycloak),
and I’m seeking your expertise and insights to help resolve the problem.
1.What did you expect to happen? What did happen?
Camunda services (identity, operate, optimize, tasklist) are not working as expected. However, when I set global.multitenancy.enabled to false, the services work fine. This suggests that the issue may be related to the multi-tenancy configuration.
2.YAML configuration for Helm
global:
multitenancy:
enabled: true
identity:
auth:
publicIssuerUrl: "https://id.domain.com/realms/camunda-platform"
operate:
redirectUrl: "https://operate.domain.com"
tasklist:
redirectUrl: "https://tasklist.domain.com"
optimize:
redirectUrl: "https://optimize.domain.com"
webModeler:
redirectUrl: "https://wf.domain.com/webModeler"
keycloak:
internal: false
enabled: false
legacy: false
contextPath: "/"
realm: "realms/camunda-platform"
auth:
adminUser: "admin"
existingSecret: "dev-keycloak-secret"
existingSecretKey: "admin-password"
url:
protocol: "https"
host: "id.domain.com"
port: 443
elasticsearch:
url: "https://[REDACTED].elastic-cloud.com"
host: "[REDACTED].elastic-cloud.com"
port: 443
prefix: "zeebe-record"
zeebe:
clusterSize: "1"
partitionCount: "1"
replicationFactor: "1"
resources:
requests:
cpu: 100m
memory: 800Mi
limits:
cpu: 500m
memory: 1000Mi
pvcSize: "16Gi"
env:
- name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_AUTHENTICATION_USERNAME
value: "elastic"
- name: ZEEBE_BROKER_EXPORTERS_ELASTICSEARCH_ARGS_AUTHENTICATION_PASSWORD
value: "elastic_password"
zeebe-gateway:
replicas: 1
ingress:
enabled: true
annotations:
cert-manager.io/cluster-issuer: "cluster-issuer"
path: /
host: "gateway.domain.com"
tls:
enabled: true
secretName: gateway-dev-tls-secret
operate:
resources:
requests:
cpu: 600m
memory: 400Mi
limits:
cpu: 600m
memory: 400Mi
ingress:
annotations:
cert-manager.io/cluster-issuer: "cluster-issuer"
enabled: true
path: /
host: "operate.domain.com"
tls:
enabled: true
secretName: operate-dev-tls-secret
env:
- name: CAMUNDA_OPERATE_ELASTICSEARCH_URL
value: "https://[REDACTED].elastic-cloud.com"
- name: CAMUNDA_OPERATE_ELASTICSEARCH_USERNAME
value: "elastic"
- name: CAMUNDA_OPERATE_ELASTICSEARCH_PASSWORD
value: "elastic_password"
- name: CAMUNDA_OPERATE_ZEEBE_ELASTICSEARCH_URL
value: "https://[REDACTED].elastic-cloud.com"
- name: CAMUNDA_OPERATE_ZEEBE_ELASTICSEARCH_USERNAME
value: "elastic"
- name: CAMUNDA_OPERATE_ZEEBE_ELASTICSEARCH_PASSWORD
value: "elastic_password"
tasklist:
resources:
requests:
cpu: 400m
memory: 500Mi
limits:
cpu: 400m
memory: 600Mi
ingress:
annotations:
cert-manager.io/cluster-issuer: "cluster-issuer"
enabled: true
path: /
host: "tasklist.domain.com"
tls:
enabled: true
secretName: tasklist-dev-tls-secret
env:
- name: CAMUNDA_TASKLIST_ELASTICSEARCH_URL
value: "https://[REDACTED].elastic-cloud.com"
- name: CAMUNDA_TASKLIST_ELASTICSEARCH_USERNAME
value: "elastic"
- name: CAMUNDA_TASKLIST_ELASTICSEARCH_PASSWORD
value: "elastic_password"
- name: CAMUNDA_TASKLIST_ZEEBE_ELASTICSEARCH_URL
value: "https://[REDACTED].elastic-cloud.com"
- name: CAMUNDA_TASKLIST_ZEEBE_ELASTICSEARCH_USERNAME
value: "elastic"
- name: CAMUNDA_TASKLIST_ZEEBE_ELASTICSEARCH_PASSWORD
value: "elastic_password"
optimize:
partitionCount: "1"
ingress:
annotations:
cert-manager.io/cluster-issuer: "cluster-issuer"
enabled: true
path: /
host: "optimize.domain.com"
tls:
enabled: true
secretName: "dev-tls-secret"
env:
- name: OPTIMIZE_ELASTICSEARCH_HOST
value: "[REDACTED].elastic-cloud.com"
- name: OPTIMIZE_ELASTICSEARCH_HTTP_PORT
value: "443"
- name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_URL
value: "https://[REDACTED].elastic-cloud.com"
- name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SSL_ENABLED
value: "true"
- name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_USERNAME
value: "elastic"
- name: CAMUNDA_OPTIMIZE_ELASTICSEARCH_SECURITY_PASSWORD
value: "elastic_password"
- name: CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_URL
value: "https://id.domain.com"
- name: CAMUNDA_OPTIMIZE_IDENTITY_ISSUER_BACKEND_URL
value: "https://id.domain.com"
resources:
requests:
cpu: 600m
memory: 300Mi
limits:
cpu: 600m
memory: 500Mi
identity:
fullURL: "https://identity.domain.com"
ingress:
annotations:
cert-manager.io/cluster-issuer: "cluster-issuer"
enabled: true
path: /
host: "identity.domain.com"
tls:
enabled: true
secretName: identity-dev-tls-secret
keycloak:
enabled: false
resources:
requests:
cpu: 600m
memory: 400Mi
limits:
cpu: 600m
memory: 400Mi
env:
- name: KEYCLOAK_REALM
value: "camunda-platform"
- name: IDENTITY_CLIENT_ID
value: "camunda-identity"
- name: IDENTITY_CLIENT_SECRET
value: "PQcOrg4M4t7uuz3H8hwQx8VYE7CZTG0V"
- name: LOGGING_LEVEL_ROOT
value: TRACE
- name: IDENTITY_LOG_LEVEL
value: TRACE
- name: IDENTITY_AUTH_PROVIDER_BACKEND_URL
value: "https://id.domain.com/realms/camunda-platform"
externalDatabase:
enabled: true
host: "[REDACTED].azure.com"
port: 5432
username: "admin"
password: "[REDACTED]"
database: "camunda-identity"
elasticsearch:
enabled: false
postgresql:
enabled: false
console:
enabled: false
connectors:
enabled: true
3.What environment are you running C8 on?
- I have a Helm Camunda v8.3.1 deployment in an AKS (Azure Kubernetes Service) v1.27.3 environment.
- I’ve set up external Elasticsearch v8.9.2 with admin access.
- There’s an existing Keycloak v22.0.4 with SSL in place.
- The setup includes separate domains for each Camunda service (identity, operate, optimize, tasklist).
- Cert-Manager Cluster certificate issuer is in place and functioning correctly.
- The externalDatabase configuration in Camunda Identity service (identity.externalDatabase) has the correct username and password, with admin rights and the database is blank.
4.What version are you running?
Helm Camunda v8.3.1
5.Are all pods running?
No.
Running:
- camunda-zeebe-0
- camunda-zeebe-gateway
Pending/Restarting:
- camunda-connectors
- camunda-identity
- camunda-operate
- camunda-optimize
- camunda-tasklist
6.Error logs or stack traces
Connectors
Identity:
Operate:
Tasklist:
Is there a known compatibility or configuration issue with Camunda v8.3.1 when using multi-tenancy in AKS with external Elasticsearch and Keycloak?
Are there specific configurations or logs I should check to identify the root cause?
Any insights, suggestions, or experiences related to this issue would be greatly appreciated.
Thank you in advance for your help and support.
Hi @mahmoud_helal - welcome to the forums! There are no environment issues with multi-tenancy that I’m aware of: as long as the environment can run Kubernetes, you can enable multi-tenancy.
Unfortunately the weTransfer links are expired. Can you upload your logs directly to the post (there’s an attachment option)?
Hello @nathan.loding ,
I’ve updated the Helm chart to version 8.3.2, and now all services are functioning properly, with the exception of (operate and Tasklist). Below is a portion of the log that appears to be indicating the problem.
Tasklist:
2023-11-14 18:01:38.639 INFO 1 --- [ migration_5] i.c.t.s.m.e.ElasticSearchMigrator : No migration needed for variable, no previous indices found.
2023-11-14 18:01:38.641 INFO 1 --- [ migration_1] i.c.t.s.m.e.ElasticSearchMigrator : No migration needed for migration-steps-repository, no previous indices found.
2023-11-14 18:01:38.644 INFO 1 --- [ migration_3] i.c.t.s.m.e.ElasticSearchMigrator : No migration needed for draft-task-variable, no previous indices found.
2023-11-14 18:01:38.644 INFO 1 --- [ main] i.c.t.s.SchemaStartup : SchemaStartup: finished.
2023-11-14 18:01:38.645 INFO 1 --- [ main] i.c.t.ImportModuleConfiguration : Starting module: importer
2023-11-14 18:01:38.646 INFO 1 --- [ main] i.c.t.ArchiverModuleConfiguration : Starting module: archiver
2023-11-14 18:01:38.698 INFO 1 --- [ main] i.c.t.w.StartupBean : INIT: DONE
2023-11-14 18:01:39.243 INFO 1 --- [ main] i.c.t.z.ImportPositionHolderAbstract : INIT: Start import position updater...
2023-11-14 18:01:39.729 WARN 1 --- [-worker-ELG-1-1] i.c.z.c.i.ZeebeCallCredentials : The request's security level does not guarantee that the credentials will be confidential.
2023-11-14 18:01:39.831 WARN 1 --- [ main] i.c.t.z.PartitionHolder : Error occurred when requesting partition ids from Zeebe client: null
io.camunda.zeebe.client.api.command.ClientStatusException: null
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.3.2.jar:8.3.2]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.3.2.jar:8.3.2]
at io.camunda.tasklist.zeebe.PartitionHolder.getPartitionIdsFromZeebe(PartitionHolder.java:102) ~[tasklist-common-8.3.2.jar:?]
at io.camunda.tasklist.zeebe.PartitionHolder.getPartitionIdsWithWaitingTimeAndRetries(PartitionHolder.java:64) ~[tasklist-common-8.3.2.jar:?]
at io.camunda.tasklist.zeebe.PartitionHolder.getPartitionIds(PartitionHolder.java:47) ~[tasklist-common-8.3.2.jar:?]
at io.camunda.tasklist.zeebeimport.RecordsReaderHolder.getAllRecordsReaders(RecordsReaderHolder.java:49) ~[tasklist-importer-8.3.2.jar:?]
at io.camunda.tasklist.zeebeimport.ZeebeImporter.scheduleReaders(ZeebeImporter.java:45) ~[tasklist-importer-8.3.2.jar:?]
at io.camunda.tasklist.zeebeimport.ZeebeImporter.startImportingData(ZeebeImporter.java:40) ~[tasklist-importer-8.3.2.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1762) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) ~[spring-beans-6.0.13.jar:6.0.13]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:950) ~[spring-context-6.0.13.jar:6.0.13]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:616) ~[spring-context-6.0.13.jar:6.0.13]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.0.12.jar:3.0.12]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:733) ~[spring-boot-3.0.12.jar:3.0.12]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:435) ~[spring-boot-3.0.12.jar:3.0.12]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:311) ~[spring-boot-3.0.12.jar:3.0.12]
at io.camunda.tasklist.Application.main(Application.java:67) ~[classes/:?]
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNKNOWN
at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.3.2.jar:8.3.2]
... 29 more
Caused by: io.grpc.StatusRuntimeException: UNKNOWN
at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.58.0.jar:1.58.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.58.0.jar:1.58.0]
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.58.0.jar:1.58.0]
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.58.0.jar:1.58.0]
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.58.0.jar:1.58.0]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) ~[grpc-core-1.58.0.jar:1.58.0]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) ~[grpc-core-1.58.0.jar:1.58.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) ~[grpc-core-1.58.0.jar:1.58.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) ~[grpc-core-1.58.0.jar:1.58.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.58.0.jar:1.58.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.58.0.jar:1.58.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Operate:
2023-11-14 18:13:31.034 DEBUG 7 --- [ main] i.c.o.w.StartupBean : INIT: Generate demo data...
2023-11-14 18:13:31.034 INFO 7 --- [ main] i.c.o.w.StartupBean : INIT: Start operation executor...
2023-11-14 18:13:31.035 INFO 7 --- [ main] i.c.o.w.StartupBean : INIT: DONE
2023-11-14 18:13:31.064 DEBUG 7 --- [ Thread-2] i.c.o.s.e.ElasticsearchBatchRequest : Execute batchRequest with 0 requests and refresh
2023-11-14 18:13:31.065 DEBUG 7 --- [ Thread-2] i.c.o.w.e.w.BatchOperationWriter : 0 operations locked
2023-11-14 18:13:31.273 INFO 7 --- [ main] i.c.o.z.ZeebeImporter : INIT: Start importing data...
2023-11-14 18:13:31.274 DEBUG 7 --- [ main] i.c.o.z.PartitionHolder : Requesting partition ids from Zeebe client
2023-11-14 18:13:31.671 WARN 7 --- [-worker-ELG-1-2] i.c.z.c.i.ZeebeCallCredentials : The request's security level does not guarantee that the credentials will be confidential.
2023-11-14 18:13:31.794 WARN 7 --- [ main] i.c.o.z.PartitionHolder : Error occurred when requesting partition ids from Zeebe client: null
io.camunda.zeebe.client.api.command.ClientStatusException: null
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.3.2.jar!/:8.3.2]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.3.2.jar!/:8.3.2]
at io.camunda.operate.zeebe.PartitionHolder.getPartitionIdsFromZeebe(PartitionHolder.java:102) ~[operate-common-8.3.2.jar!/:8.3.2]
at io.camunda.operate.zeebe.PartitionHolder.getPartitionIdsWithWaitingTimeAndRetries(PartitionHolder.java:65) ~[operate-common-8.3.2.jar!/:8.3.2]
at io.camunda.operate.zeebe.PartitionHolder.getPartitionIds(PartitionHolder.java:50) ~[operate-common-8.3.2.jar!/:8.3.2]
at io.camunda.operate.zeebeimport.RecordsReaderHolder.getAllRecordsReaders(RecordsReaderHolder.java:51) ~[operate-importer-8.3.2.jar!/:8.3.2]
at io.camunda.operate.zeebeimport.ZeebeImporter.scheduleReaders(ZeebeImporter.java:55) ~[operate-importer-8.3.2.jar!/:8.3.2]
at io.camunda.operate.zeebeimport.ZeebeImporter.startImportingData(ZeebeImporter.java:48) ~[operate-importer-8.3.2.jar!/:8.3.2]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMethod.invoke(InitDestroyAnnotationBeanPostProcessor.java:457) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:401) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:219) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:419) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1762) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:598) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) ~[spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) [spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) [spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) [spring-beans-6.0.13.jar!/:6.0.13]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:950) [spring-context-6.0.13.jar!/:6.0.13]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:616) [spring-context-6.0.13.jar!/:6.0.13]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) [spring-boot-3.1.5.jar!/:3.1.5]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:738) [spring-boot-3.1.5.jar!/:3.1.5]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:440) [spring-boot-3.1.5.jar!/:3.1.5]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:316) [spring-boot-3.1.5.jar!/:3.1.5]
at io.camunda.operate.Application.main(Application.java:67) [classes!/:8.3.2]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [operate-webapp-8.3.2-exec.jar:8.3.2]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:95) [operate-webapp-8.3.2-exec.jar:8.3.2]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [operate-webapp-8.3.2-exec.jar:8.3.2]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) [operate-webapp-8.3.2-exec.jar:8.3.2]
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNKNOWN
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.3.2.jar!/:8.3.2]
... 37 more
Caused by: io.grpc.StatusRuntimeException: UNKNOWN
at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.58.0.jar!/:1.58.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.58.0.jar!/:1.58.0]
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.58.0.jar!/:1.58.0]
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.58.0.jar!/:1.58.0]
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.58.0.jar!/:1.58.0]
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567) ~[grpc-core-1.58.0.jar!/:1.58.0]
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71) ~[grpc-core-1.58.0.jar!/:1.58.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735) ~[grpc-core-1.58.0.jar!/:1.58.0]
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716) ~[grpc-core-1.58.0.jar!/:1.58.0]
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.58.0.jar!/:1.58.0]
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.58.0.jar!/:1.58.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2023-11-14 18:13:31.819 INFO 7 --- [ main] i.c.o.z.PartitionHolder : Partition ids can't be fetched from Zeebe. Try next round (1).
2023-11-14 18:13:32.820 DEBUG 7 --- [ main] i.c.o.z.PartitionHolder : Requesting partition ids from Zeebe client
2023-11-14 18:13:32.820 WARN 7 --- [ main] i.c.z.c.i.ZeebeCallCredentials : The request's security level does not guarantee that the credentials will be confidential.
2023-11-14 18:13:32.826 WARN 7 --- [ main] i.c.o.z.PartitionHolder : Error occurred when requesting partition ids from Zeebe client: null
Both services continue to restart repeatedly. Any suggestions or advice would be greatly appreciated.
I don’t think I’ve seen an UNKNOWN gRPC exception yet. My first guess is to try to manually configure the Zeebe connection for both Operate (docs) and Tasklist (docs), then see if you get the same error or something different.
Configuring it manually is not a permanent solution I guess. Running them in the cluster is a requirement. I’m Getting the same issue, any suggestions on what needs to be fixed?
Sharing docker file :
docker-compose.yaml (15.5 KB)
Hi @Harshdeep_Singh - is this related to your other thread? Have you tried using the latest docker-compose release (8.4)?
Hi @nathan.loding - other thread is resolved. I’ve found the error and added the solution in it’s comments.
However, coming back to this particular issue. After installing everything and booting up the containers, tasklist and operate are failing to run successfully. Error being :
io.camunda.zeebe.client.api.command.ClientStatusException: null
Please find the attached SS below :
@Harshdeep_Singh - are you using Helm to deploy to AKS? If not, please open a new topic!
