Not able to connect zeebe using

@nathan.loding - Today I have upgraded my self-managed cluster to 8.5. After upgrading I am not able to connect zeebe using my java client. I have enabled gRPC and Rest for zeebe gateway and I have validated that using Rest I am able to connect to zeebe via Postman.

image

grpc zeebe gateway ingress:

rest zeebe gateway ingress:

I always use below configuration to deploy from Web Modeler which is also successful:

I am using 8.5.0-rc2 version of spring-boot-starter-camunda and below is application.properties configuration that I am using -

camunda.client.auth.oidc-type=keycloak
camunda.client.auth.issuer=https://< AWS HOST >/auth/realms/camunda-platform
camunda.client.auth.client-id=camunda-api-client
camunda.client.auth.client-secret=
camunda.client.zeebe.audience=zeebe-api
camunda.client.zeebe.base-url=https://< AWS HOST >:443/
camunda.client.zeebe.enabled=true
camunda.client.mode=oidc

I am getting below error in my java code log -

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.2.4)

2024-04-19T19:55:45.821+05:30  INFO 17728 --- [           main] com.example.demo.DemoApplication         : Starting DemoApplication using Java 17.0.10 with PID 17728 (C:\Users\SAUM1310\OneDrive - LTIMindtree\Java_Workspace\zeebe-client-self-instance\target\classes started by SAUM1310 in C:\Users\SAUM1310\OneDrive - LTIMindtree\Java_Workspace\zeebe-client-self-instance)
2024-04-19T19:55:45.826+05:30  INFO 17728 --- [           main] com.example.demo.DemoApplication         : No active profile set, falling back to 1 default profile: "default"
2024-04-19T19:55:47.063+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.zeebe.spring.client.annotation.processor.AnnotationProcessorConfiguration' of type [io.camunda.zeebe.spring.client.annotation.processor.AnnotationProcessorConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.063+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'deploymentPostProcessor' of type [io.camunda.zeebe.spring.client.annotation.processor.ZeebeDeploymentAnnotationProcessor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.115+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'zeebe.client-io.camunda.zeebe.spring.client.properties.ZeebeClientConfigurationProperties' of type [io.camunda.zeebe.spring.client.properties.ZeebeClientConfigurationProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.165+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'camunda.client-io.camunda.zeebe.spring.client.properties.CamundaClientProperties' of type [io.camunda.zeebe.spring.client.properties.CamundaClientProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.168+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.zeebe.spring.client.configuration.ZeebeClientAllAutoConfiguration' of type [io.camunda.zeebe.spring.client.configuration.ZeebeClientAllAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.180+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'zeebeClientExecutorService' of type [io.camunda.zeebe.spring.client.jobhandling.ZeebeClientExecutorService] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.187+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'commandExceptionHandlingStrategy' of type [io.camunda.zeebe.spring.client.jobhandling.DefaultCommandExceptionHandlingStrategy] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.190+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.zeebe.spring.client.configuration.MetricsDefaultConfiguration' of type [io.camunda.zeebe.spring.client.configuration.MetricsDefaultConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.195+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'noopMetricsRecorder' of type [io.camunda.zeebe.spring.client.metrics.DefaultNoopMetricsRecorder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)     
2024-04-19T19:55:47.198+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.202+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperBuilderConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonObjectMapperBuilderConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.217+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$Jackson2ObjectMapperBuilderCustomizerConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$Jackson2ObjectMapperBuilderCustomizerConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.229+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'spring.jackson-org.springframework.boot.autoconfigure.jackson.JacksonProperties' of type [org.springframework.boot.autoconfigure.jackson.JacksonProperties] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.246+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$ParameterNamesModuleConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$ParameterNamesModuleConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.261+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'parameterNamesModule' of type [com.fasterxml.jackson.module.paramnames.ParameterNamesModule] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.266+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonMixinConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$JacksonMixinConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)   
2024-04-19T19:55:47.320+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jsonMixinModuleEntries' of type [org.springframework.boot.jackson.JsonMixinModuleEntries] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.324+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jsonMixinModule' of type [org.springframework.boot.jackson.JsonMixinModule] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.329+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.361+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jsonComponentModule' of type [org.springframework.boot.jackson.JsonComponentModule] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.363+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'standardJacksonObjectMapperBuilderCustomizer' of type [org.springframework.boot.autoconfigure.jackson.JacksonAutoConfiguration$Jackson2ObjectMapperBuilderCustomizerConfiguration$StandardJackson2ObjectMapperBuilderCustomizer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.381+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jacksonObjectMapperBuilder' of type [org.springframework.http.converter.json.Jackson2ObjectMapperBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.406+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jacksonObjectMapper' of type [com.fasterxml.jackson.databind.ObjectMapper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.408+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'io.camunda.zeebe.spring.client.configuration.JsonMapperConfiguration' of type [io.camunda.zeebe.spring.client.configuration.JsonMapperConfiguration$$SpringCGLIB$$0] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.409+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'zeebeJsonMapper' of type [io.camunda.zeebe.client.impl.ZeebeObjectMapper] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.412+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'parameterResolverStrategy' of type [io.camunda.zeebe.spring.client.jobhandling.parameter.DefaultParameterResolverStrategy] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.414+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'jobWorkerManager' of type [io.camunda.zeebe.spring.client.jobhandling.JobWorkerManager] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.439+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'propertyBasedZeebeWorkerValueCustomizer' of type [io.camunda.zeebe.spring.client.properties.PropertyBasedZeebeWorkerValueCustomizer] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.440+05:30  INFO 17728 --- [           main] trationDelegate$BeanPostProcessorChecker : Bean 'zeebeWorkerPostProcessor' of type [io.camunda.zeebe.spring.client.annotation.processor.ZeebeWorkerAnnotationProcessor] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2024-04-19T19:55:47.826+05:30  INFO 17728 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2024-04-19T19:55:47.833+05:30  INFO 17728 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2024-04-19T19:55:47.833+05:30  INFO 17728 --- [           main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.13]
2024-04-19T19:55:48.074+05:30  INFO 17728 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2024-04-19T19:55:48.076+05:30  INFO 17728 --- [           main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2137 ms
2024-04-19T19:55:48.594+05:30  INFO 17728 --- [           main] org.ehcache.core.EhcacheManager          : Cache 'tokenCache' created in EhcacheManager.
2024-04-19T19:55:48.625+05:30  INFO 17728 --- [           main] org.ehcache.core.EhcacheManager          : Cache 'tokenCache' created in EhcacheManager.
2024-04-19T19:55:48.662+05:30  INFO 17728 --- [           main] org.ehcache.core.EhcacheManager          : Cache 'tokenCache' created in EhcacheManager.
2024-04-19T19:55:48.683+05:30  INFO 17728 --- [           main] org.ehcache.core.EhcacheManager          : Cache 'tokenCache' created in EhcacheManager.
2024-04-19T19:55:48.706+05:30  INFO 17728 --- [           main] org.ehcache.core.EhcacheManager          : Cache 'tokenCache' created in EhcacheManager.
2024-04-19T19:55:48.777+05:30  INFO 17728 --- [           main] z.s.c.c.ZeebeClientProdAutoConfiguration : Creating ZeebeClient using ZeebeClientConfiguration [ZeebeClientConfiguration{properties=ZeebeClientConfigurationProperties{environment=ApplicationServletEnvironment {activeProfiles=[], defaultProfiles=[default], propertySources=[ConfigurationPropertySourcesPropertySource {name='configurationProperties'}, StubPropertySource {name='servletConfigInitParams'}, ServletContextPropertySource {name='servletContextInitParams'}, PropertiesPropertySource {name='systemProperties'}, OriginAwareSystemEnvironmentPropertySource {name='systemEnvironment'}, RandomValuePropertySource {name='random'}, OriginTrackedMapPropertySource {name='Config resource 'class path resource [application.properties]' via location 'optional:classpath:/''}, OriginTrackedMapPropertySource {name='application-camunda-oidc.yaml'}]}, connectionMode='null', defaultTenantId='<default>', defaultJobWorkerTenantIds=[<default>], applyEnvironmentVariableOverrides=false, enabled=true, broker=Broker{gatewayAddress='null', keepAlive=PT45S}, cloud=Cloud{clusterId='null', clientId='***', clientSecret='***', region='bru-2', scope='null', baseUrl='zeebe.camunda.io', authUrl='https://login.cloud.camunda.io/oauth/token', port=443, credentialsCachePath='null'}, worker=Worker{maxJobsActive=32, threads=1, defaultName='null', defaultType='null', override={}}, message=Message{timeToLive=PT1H, maxMessageSize=4194304}, security=Security{plaintext=false, overrideAuthority='null', certPath='null'}, job=Job{timeout=PT5M, pollInterval=PT0.1S}, ownsJobWorkerExecutor=false, defaultJobWorkerStreamEnabled=false, requestTimeout=PT10S}, camundaClientProperties=io.camunda.zeebe.spring.client.properties.CamundaClientProperties@3d3a1903, authentication=io.camunda.common.auth.SelfManagedAuthentication@26582ca, jsonMapper=io.camunda.zeebe.client.impl.ZeebeObjectMapper@153cb763, interceptors=[], zeebeClientExecutorService=io.camunda.zeebe.spring.client.jobhandling.ZeebeClientExecutorService@2c3158e0}]
2024-04-19T19:55:49.296+05:30  INFO 17728 --- [           main] z.s.c.a.p.ZeebeWorkerAnnotationProcessor : Configuring 1 Zeebe worker(s) of bean 'sampleWorker': [ZeebeWorkerValue{type='test-worker', name='testWorker', timeout=PT-0.001S, maxJobsActive=-1, requestTimeout=PT-1S, pollInterval=PT-0.001S, autoComplete=true, fetchVariables=[], enabled=true, methodInfo=io.camunda.zeebe.spring.client.bean.MethodInfo@13d289c7, tenantIds=[], forceFetchAllVariables=false, streamEnabled=true, streamTimeout=PT-0.001S}]
2024-04-19T19:55:49.312+05:30  INFO 17728 --- [           main] com.example.demo.ZeebeClientImpl         : Going to start process :  Test_Process
2024-04-19T19:55:50.481+05:30  WARN 17728 --- [           main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'startProcess' defined in com.example.demo.DemoApplication: Failed to instantiate [void]: Factory method 'startProcess' threw exception with message: HTTP status code 404
invalid content-type: application/problem+json
headers: Metadata(:status=404,date=Fri, 19 Apr 2024 14:25:50 GMT,content-type=application/problem+json,cache-control=no-cache, no-store, max-age=0, must-revalidate,pragma=no-cache,expires=0,x-content-type-options=nosniff,strict-transport-security=max-age=15724800; includeSubDomains,x-frame-options=DENY,x-xss-protection=0,referrer-policy=no-referrer,content-length=116)
DATA-----------------------------
{"type":"about:blank","title":"Not Found","status":404,"instance":"/gateway_protocol.Gateway/CreateProcessInstance"}
DATA-----------------------------

2024-04-19T19:55:50.529+05:30  INFO 17728 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2024-04-19T19:55:50.547+05:30  INFO 17728 --- [           main] .s.b.a.l.ConditionEvaluationReportLogger : 

Error starting ApplicationContext. To display the condition evaluation report re-run your application with 'debug' enabled.
2024-04-19T19:55:50.586+05:30 ERROR 17728 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'startProcess' defined in com.example.demo.DemoApplication: Failed to instantiate [void]: Factory method 'startProcess' threw exception with message: HTTP status code 404
invalid content-type: application/problem+json
headers: Metadata(:status=404,date=Fri, 19 Apr 2024 14:25:50 GMT,content-type=application/problem+json,cache-control=no-cache, no-store, max-age=0, must-revalidate,pragma=no-cache,expires=0,x-content-type-options=nosniff,strict-transport-security=max-age=15724800; includeSubDomains,x-frame-options=DENY,x-xss-protection=0,referrer-policy=no-referrer,content-length=116)
DATA-----------------------------
{"type":"about:blank","title":"Not Found","status":404,"instance":"/gateway_protocol.Gateway/CreateProcessInstance"}
DATA-----------------------------

        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:654) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:488) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1332) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1162) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:560) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:520) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:325) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:323) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:973) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942) ~[spring-context-6.0.12.jar:6.0.12]
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:608) ~[spring-context-6.0.12.jar:6.0.12]
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:146) ~[spring-boot-3.2.4.jar:3.2.4]
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) ~[spring-boot-3.2.4.jar:3.2.4]
        at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:456) ~[spring-boot-3.2.4.jar:3.2.4]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:334) ~[spring-boot-3.2.4.jar:3.2.4]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1354) ~[spring-boot-3.2.4.jar:3.2.4]
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) ~[spring-boot-3.2.4.jar:3.2.4]
        at com.example.demo.DemoApplication.main(DemoApplication.java:26) ~[classes/:na]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [void]: Factory method 'startProcess' threw exception with message: HTTP status code 404
invalid content-type: application/problem+json
headers: Metadata(:status=404,date=Fri, 19 Apr 2024 14:25:50 GMT,content-type=application/problem+json,cache-control=no-cache, no-store, max-age=0, must-revalidate,pragma=no-cache,expires=0,x-content-type-options=nosniff,strict-transport-security=max-age=15724800; includeSubDomains,x-frame-options=DENY,x-xss-protection=0,referrer-policy=no-referrer,content-length=116)
DATA-----------------------------
{"type":"about:blank","title":"Not Found","status":404,"instance":"/gateway_protocol.Gateway/CreateProcessInstance"}
DATA-----------------------------

        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:171) ~[spring-beans-6.0.12.jar:6.0.12]
        at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:650) ~[spring-beans-6.0.12.jar:6.0.12]
        ... 19 common frames omitted
Caused by: io.camunda.zeebe.client.api.command.ClientStatusException: HTTP status code 404
invalid content-type: application/problem+json
headers: Metadata(:status=404,date=Fri, 19 Apr 2024 14:25:50 GMT,content-type=application/problem+json,cache-control=no-cache, no-store, max-age=0, must-revalidate,pragma=no-cache,expires=0,x-content-type-options=nosniff,strict-transport-security=max-age=15724800; includeSubDomains,x-frame-options=DENY,x-xss-protection=0,referrer-policy=no-referrer,content-length=116)
DATA-----------------------------
{"type":"about:blank","title":"Not Found","status":404,"instance":"/gateway_protocol.Gateway/CreateProcessInstance"}
DATA-----------------------------

        at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.4.5.jar:8.4.5]
        at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.4.5.jar:8.4.5]
        at com.example.demo.ZeebeClientImpl.createInstance(ZeebeClientImpl.java:28) ~[classes/:na]
        at com.example.demo.DemoApplication.startProcess(DemoApplication.java:30) ~[classes/:na]
        at com.example.demo.DemoApplication$$SpringCGLIB$$0.CGLIB$startProcess$1(<generated>) ~[classes/:na]
        at com.example.demo.DemoApplication$$SpringCGLIB$$FastClass$$1.invoke(<generated>) ~[classes/:na]   
        at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:258) ~[spring-core-6.1.5.jar:6.1.5]
        at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-6.0.12.jar:6.0.12]
        at com.example.demo.DemoApplication$$SpringCGLIB$$0.startProcess(<generated>) ~[classes/:na]        
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[na:na]      
        at java.base/java.lang.reflect.Method.invoke(Unknown Source) ~[na:na]
        at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:139) ~[spring-beans-6.0.12.jar:6.0.12]
        ... 20 common frames omitted
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNIMPLEMENTED: HTTP status code 404
invalid content-type: application/problem+json
headers: Metadata(:status=404,date=Fri, 19 Apr 2024 14:25:50 GMT,content-type=application/problem+json,cache-control=no-cache, no-store, max-age=0, must-revalidate,pragma=no-cache,expires=0,x-content-type-options=nosniff,strict-transport-security=max-age=15724800; includeSubDomains,x-frame-options=DENY,x-xss-protection=0,referrer-policy=no-referrer,content-length=116)
DATA-----------------------------
{"type":"about:blank","title":"Not Found","status":404,"instance":"/gateway_protocol.Gateway/CreateProcessInstance"}
DATA-----------------------------

        at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source) ~[na:na]
        at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52) ~[zeebe-client-java-8.4.5.jar:8.4.5]
        ... 32 common frames omitted
Caused by: io.grpc.StatusRuntimeException: UNIMPLEMENTED: HTTP status code 404
invalid content-type: application/problem+json
headers: Metadata(:status=404,date=Fri, 19 Apr 2024 14:25:50 GMT,content-type=application/problem+json,cache-control=no-cache, no-store, max-age=0, must-revalidate,pragma=no-cache,expires=0,x-content-type-options=nosniff,strict-transport-security=max-age=15724800; includeSubDomains,x-frame-options=DENY,x-xss-protection=0,referrer-policy=no-referrer,content-length=116)
DATA-----------------------------
{"type":"about:blank","title":"Not Found","status":404,"instance":"/gateway_protocol.Gateway/CreateProcessInstance"}
DATA-----------------------------

        at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.60.1.jar:1.60.1]
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481) ~[grpc-stub-1.60.1.jar:1.60.1]
        at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489) ~[grpc-core-1.60.1.jar:1.60.1]
        at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453) ~[grpc-core-1.60.1.jar:1.60.1]
        at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486) ~[grpc-core-1.60.1.jar:1.60.1]
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574) ~[grpc-core-1.60.1.jar:1.60.1]
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72) ~[grpc-core-1.60.1.jar:1.60.1]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742) ~[grpc-core-1.60.1.jar:1.60.1]
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723) ~[grpc-core-1.60.1.jar:1.60.1]
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.60.1.jar:1.60.1]     
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ~[grpc-core-1.60.1.jar:1.60.1]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[na:na]
        at java.base/java.lang.Thread.run(Unknown Source) ~[na:na]

Though I tested with SaaS configuration as below which is successful -

camunda.client.mode=saas
camunda.client.auth.client-id=
camunda.client.auth.client-secret=
camunda.client.region=jfk-1
camunda.client.cluster-id=

Can you please guide me what I am missing here?

Hi @arijit.chanda - I don’t see anything obvious in what you’ve supplied. I think I would try connecting to the gRPC gateway without spring-zeebe to see if the issue is with your code (or with spring-zeebe), or if the issue is something else. For instance, Postman supports gRPC, so I would try that. If it works, the issue is with your code, configuration, or a bug with spring-zeebe or the Zeebe Java client; if it fails, the issue is more likely with your deployment (perhaps a routing issue on the ingress somehow?).

1 Like

Hi @nathan.loding thanks for looking into the issue. I have changed ingress configuration of zee gateway and post that I am able to connect using gRPC gateway from java client whereas Rest gateway is not working now. I have defined path as “zeebe” for Rest zeebe gateway in values file. It’s giving me below error -
URL that I am using https://< AWS Host>/zeebe/v1/topology

I am also experiencing additional 2 issues. There could be some wrong configuration I have did in values file.

  1. Whenever I am logging off from identity it’s showing below screen instead of redirecting back to login screen.

  2. When I trying to open console using https://< AWS Host>/console, it’s giving me “Invalid parameter: redirect_uri”.

I have attached the values file that I am using, it would be really helpful if you can help me to resolve those issues.

camunda-values-v8.5.yaml (10.2 KB)

@arijit.chanda - since you have an enterprise license, it might be best to reach out to the support team. This is a community forum, not an official support channel, and you may get more direct support from them.

I’ve not worked with a Self Managed deployment behind a load balancer with the redirects you have configured, so I am not sure if that’s configured fully or if there’s additional configuration needed. Nothing jumps out to me as being incorrectly configured, but there is a lot of environment specific information that I can’t validate.