Trying to connect Job Worker/zbctl

Hi everybody,
my Camunda 8 self managed cluster is ready and every pod is running.
I can deploy bpmn diagrams from the desktop modeler but i couldn’t connect with my cluster using zbctl with the credentials i use for the desktop modeler. Should i use different data or which configuration should i check?
I have ingress enabled but without TLS.

I tried the Java Client and zbctl with the data from the java code:

package com.camunda.academy;

import io.camunda.zeebe.client.ZeebeClient;
import io.camunda.zeebe.client.api.response.DeploymentEvent;
import io.camunda.zeebe.client.api.response.Topology;
import io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider;
import io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProviderBuilder;

import java.net.URI;

public class test {

    public static void main(String[] args) {
        // Setzen Sie Ihre Konfigurationswerte
        String zeebeGrpc = "grpc://zeebecamunda.local"; // Normalerweise der gRPC-Port des Zeebe Gateways
        String oAuthAPI = "http://camunda.local/auth/realms/camunda-platform/protocol/openid-connect/token";
        String audience = "zeebe-api";
        String clientId = "zeebe";
        String clientSecret = "****";

        // Erstellen des OAuthCredentialsProvider
        OAuthCredentialsProvider credentialsProvider = new OAuthCredentialsProviderBuilder()
                .authorizationServerUrl(oAuthAPI)
                .audience(audience)
                .clientId(clientId)
                .clientSecret(clientSecret)
                .build();

        // Erstellen und Konfigurieren des ZeebeClients
        try (ZeebeClient client = ZeebeClient.newClientBuilder()
                .grpcAddress(new URI(zeebeGrpc))  // Setzen der Adresse des Zeebe Gateways
                .credentialsProvider(credentialsProvider)  // Hinzufügen der Authentifizierungsdaten
                .build()) {

            Topology topology = client.newTopologyRequest().send().join();
            System.out.println("Topology: " + topology);

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

By Running the Java code i get the following error:
io.camunda.zeebe.client.api.command.ClientStatusException: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54)
at com.camunda.academy.test.main(test.java:35)
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52)
… 1 more
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
Channel Pipeline: [SslHandler#0, ProtocolNegotiators$ClientTlsHandler#0, WriteBufferingAndExceptionHandler#0, DefaultChannelPipeline$TailContext#0]
at io.grpc.Status.asRuntimeException(Status.java:533)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489)
at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453)
at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshakeException(ReferenceCountedOpenSslEngine.java:1927)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:848)
at java.base/javax.net.ssl.SSLEngine.wrap(SSLEngine.java:564)
at io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:1131)
at io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:975)
at io.netty.handler.ssl.SslHandler.access$1900(SslHandler.java:171)
at io.netty.handler.ssl.SslHandler$SslTasksRunner.resumeOnEventExecutor(SslHandler.java:1832)
at io.netty.handler.ssl.SslHandler$SslTasksRunner.access$2000(SslHandler.java:1711)
at io.netty.handler.ssl.SslHandler$SslTasksRunner$2.run(SslHandler.java:1872)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
… 1 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:388)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:271)
at java.base/sun.security.validator.Validator.validate(Validator.java:256)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at io.netty.handler.ssl.EnhancingX509ExtendedTrustManager.checkServerTrusted(EnhancingX509ExtendedTrustManager.java:69)
at io.netty.handler.ssl.ReferenceCountedOpenSslClientContext$ExtendedTrustManagerVerifyCallback.verify(ReferenceCountedOpenSslClientContext.java:235)
at io.netty.handler.ssl.ReferenceCountedOpenSslContext$AbstractCertificateVerifier.verify(ReferenceCountedOpenSslContext.java:797)
at io.netty.internal.tcnative.CertificateVerifierTask.runTask(CertificateVerifierTask.java:36)
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:48)
at io.netty.internal.tcnative.SSLTask.run(SSLTask.java:42)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.runAndResetNeedTask(ReferenceCountedOpenSslEngine.java:1533)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.access$700(ReferenceCountedOpenSslEngine.java:94)
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$TaskDecorator.run(ReferenceCountedOpenSslEngine.java:1505)
at io.netty.handler.ssl.SslHandler$SslTasksRunner.run(SslHandler.java:1889)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
… 1 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:148)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:129)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:383)
… 17 more

zbctl error:
Error: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: EOF”

My Gateway Config is the standard with ingress enabled:

# Zeebe.
####    ##   ##### ###### #    #   ##   #   #
#    #  #  #    #   #      #    #  #  #   # #
#      #    #   #   #####  #    # #    #   #
#  ### ######   #   #      # ## # ######   #
#    # #    #   #   #      ##  ## #    #   #
####  #    #   #   ###### #    # #    #   #
## @section ZeebeGateway Parameters
## @extra Gateway configuration to define properties related to the standalone gateway
zeebeGateway:
  ## @param zeebeGateway.replicas defines how many standalone gateways are deployed
  replicas: 2
  ## @extra zeebeGateway.image configuration to configure the ZeebeGateway image specifics
  image:
    ## @param zeebeGateway.image.registry can be used to set container image registry.
    registry: ""
    ## @param zeebeGateway.image.repository defines which image repository to use
    repository: camunda/zeebe
    ## @param zeebeGateway.image.tag can be set to overwrite the global tag, which should be used in that chart
    tag:
    ## @param zeebeGateway.image.pullSecrets can be used to configure image pull secrets https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
    pullSecrets: []

  ## @param zeebeGateway.sidecars can be used to attach extra containers to the ZeebeGateway deployment
  sidecars: []

  ## @param zeebeGateway.podAnnotations can be used to define extra gateway pod annotations
  podAnnotations: {}
  ## @param zeebeGateway.podLabels can be used to define extra gateway pod labels
  podLabels: {}

  ## @param zeebeGateway.logLevel defines the log level which is used by the gateway
  logLevel: warn
  ## @param zeebeGateway.log4j2 can be used to overwrite the log4j2 configuration of the gateway
  log4j2: ""
  ## @param zeebeGateway.javaOpts can be used to set java options for the ZeebeGateway
  javaOpts: >-
    -XX:+ExitOnOutOfMemoryError

  ## @param zeebeGateway.env can be used to set extra environment variables in each gateway container
  env: []
  ## @extra zeebeGateway.configMap configuration which will be applied to the mounted config map.
  configMap:
    ## @param zeebeGateway.configMap.defaultMode can be used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.
    # See https://github.com/kubernetes/api/blob/92d2eac37b2a33380a2099a20aa9b209e7ab7cab/core/v1/types.go#L1615-L1623
    defaultMode: 0744
  ## @param zeebeGateway.command can be used to override the default command provided by the container image. See https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/
  command: []

  ## @extra zeebeGateway.podDisruptionBudget configuration to configure a pod disruption budget for the gateway pods https://kubernetes.io/docs/tasks/run-application/configure-pdb/
  podDisruptionBudget:
    ## @param zeebeGateway.podDisruptionBudget.enabled if true a pod disruption budget is defined for the gateways
    enabled: false
    ## @param zeebeGateway.podDisruptionBudget.minAvailable can be used to set how many pods should be available. Be aware that if minAvailable is set, maxUnavailable will not be set (they are mutually exclusive).
    minAvailable: 1
    ## @param zeebeGateway.podDisruptionBudget.maxUnavailable can be used to set how many pods should be at max. unavailable
    maxUnavailable:

  ## @extra zeebeGateway.resources configuration to set request and limit configuration for the container https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits
  ## @param zeebeGateway.resources.requests.cpu
  ## @param zeebeGateway.resources.requests.memory
  ## @param zeebeGateway.resources.limits.cpu
  ## @param zeebeGateway.resources.limits.memory
  resources:
    requests:
      cpu: 400m
      memory: 450Mi
    limits:
      cpu: 400m
      memory: 450Mi

  ## @param zeebeGateway.priorityClassName can be used to define the gateway pods priority https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass
  priorityClassName: ""

  ## @extra zeebeGateway.podSecurityContext defines the security options the gateway pod should be run wit
  podSecurityContext:
    ## @param zeebeGateway.podSecurityContext.runAsNonRoot
    runAsNonRoot: true
    ## @param zeebeGateway.podSecurityContext.fsGroup
    fsGroup: 1001

  ## @extra zeebeGateway.containerSecurityContext defines the security options the gateway container should be run with
  containerSecurityContext:
    ## @param zeebeGateway.containerSecurityContext.allowPrivilegeEscalation
    allowPrivilegeEscalation: false
    ## @param zeebeGateway.containerSecurityContext.privileged
    privileged: false
    ## @param zeebeGateway.containerSecurityContext.readOnlyRootFilesystem
    readOnlyRootFilesystem: true
    ## @param zeebeGateway.containerSecurityContext.runAsNonRoot
    runAsNonRoot: true
    ## @param zeebeGateway.containerSecurityContext.runAsUser
    runAsUser: 1001

  ## @extra zeebeGateway.startupProbe configuration
  startupProbe:
    ## @param zeebeGateway.startupProbe.enabled if true, the startup probe is enabled in app container
    enabled: false
    ## @param zeebeGateway.startupProbe.scheme defines the startup probe schema used on calling the probePath
    scheme: HTTP
    ## @param zeebeGateway.startupProbe.probePath defines the startup probe route used on the app
    probePath: /actuator/health/startup
    ## @param zeebeGateway.startupProbe.initialDelaySeconds defines the number of seconds after the container has started before
    # the probe is initiated.
    initialDelaySeconds: 30
    ## @param zeebeGateway.startupProbe.periodSeconds defines how often the probe is executed
    periodSeconds: 30
    ## @param zeebeGateway.startupProbe.successThreshold defines how often it needs to be true to be marked as ready, after failure
    successThreshold: 1
    ## @param zeebeGateway.startupProbe.failureThreshold defines when the probe is considered as failed so the Pod will be marked Unready
    failureThreshold: 5
    ## @param zeebeGateway.startupProbe.timeoutSeconds defines the seconds after the probe times out
    timeoutSeconds: 1

  ## @extra zeebeGateway.readinessProbe configuration
  readinessProbe:
    ## @param zeebeGateway.readinessProbe.enabled if true, the readiness probe is enabled in app container
    enabled: true
    ## @param zeebeGateway.readinessProbe.scheme defines the startup probe schema used on calling the probePath
    scheme: HTTP
    ## @param zeebeGateway.readinessProbe.probePath defines the readiness probe route used on the app
    probePath: /actuator/health/readiness
    ## @param zeebeGateway.readinessProbe.initialDelaySeconds defines the number of seconds after the container has started before
    ## @extra zeebeGateway.the probe is initiated.
    initialDelaySeconds: 30
    ## @param zeebeGateway.readinessProbe.periodSeconds defines how often the probe is executed
    periodSeconds: 30
    ## @param zeebeGateway.readinessProbe.successThreshold defines how often it needs to be true to be marked as ready, after failure
    successThreshold: 1
    ## @param zeebeGateway.readinessProbe.failureThreshold defines when the probe is considered as failed so the Pod will be marked Unready
    failureThreshold: 5
    ## @param zeebeGateway.readinessProbe.timeoutSeconds defines the seconds after the probe times out
    timeoutSeconds: 1

  ## @extra zeebeGateway.livenessProbe configuration
  livenessProbe:
    ## @param zeebeGateway.livenessProbe.enabled if true, the liveness probe is enabled in app container
    enabled: false
    ## @param zeebeGateway.livenessProbe.scheme defines the startup probe schema used on calling the probePath
    scheme: HTTP
    ## @param zeebeGateway.livenessProbe.probePath defines the liveness probe route used on the app
    probePath: /actuator/health/liveness
    ## @param zeebeGateway.livenessProbe.initialDelaySeconds defines the number of seconds after the container has started before
    initialDelaySeconds: 30
    ## @param zeebeGateway.livenessProbe.periodSeconds defines how often the probe is executed
    periodSeconds: 30
    ## @param zeebeGateway.livenessProbe.successThreshold defines how often it needs to be true to be considered successful after having failed
    successThreshold: 1
    ## @param zeebeGateway.livenessProbe.failureThreshold defines when the probe is considered as failed so the container will be restarted
    failureThreshold: 5
    ## @param zeebeGateway.livenessProbe.timeoutSeconds defines the seconds after the probe times out
    timeoutSeconds: 1

  ## Metrics configuration
  metrics:
    ## @param zeebeGateway.metrics.prometheus Prometheus metrics endpoint
    prometheus: /actuator/prometheus

  ## @param zeebeGateway.nodeSelector can be used to define on which nodes the gateway pods should run
  nodeSelector: {}
  ## @param zeebeGateway.tolerations can be used to define pod toleration's https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
  tolerations: []
  ## @extra zeebeGateway.affinity can be used to define pod affinity or anti-affinity https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#affinity-and-anti-affinity
  # The default defined PodAntiAffinity allows constraining on which nodes the ZeebeGateway pods are scheduled on https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
  # It uses a hard requirement for scheduling and works based on the ZeebeGateway pod labels
  ## @skip zeebeGateway.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].labelSelector.matchExpressions[0].key
  ## @skip zeebeGateway.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].labelSelector.matchExpressions[0].operator
  ## @skip zeebeGateway.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].labelSelector.matchExpressions[0].values
  ## @skip zeebeGateway.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].topologyKey
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: "app.kubernetes.io/component"
                operator: In
                values:
                  - zeebe-gateway
          topologyKey: "kubernetes.io/hostname"

  ## @param zeebeGateway.extraVolumeMounts can be used to mount extra volumes for the gateway pods, useful for enabling tls between gateway and broker
  extraVolumeMounts: []
  ## @param zeebeGateway.extraVolumes can be used to define extra volumes for the gateway pods, useful for enabling tls between gateway and broker
  extraVolumes: []
  ## @param zeebeGateway.extraInitContainers (Deprecated - use `initContainers` instead) can be used to set up extra init containers for the gateway pods, useful for adding interceptors
  extraInitContainers: []
  ## @param zeebeGateway.initContainers can be used to set up extra init containers for the application Pod
  initContainers: []

  ## @extra zeebeGateway.service configuration for the gateway service
  service:
    ## @param zeebeGateway.service.annotations can be used to define annotations, which will be applied to the zeebe-gateway service
    annotations: {}
    ## @param zeebeGateway.service.type defines the type of the service https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
    type: ClusterIP
    ## @param zeebeGateway.service.loadBalancerIP defines public ip of the load balancer if the type is LoadBalancer
    loadBalancerIP: ""
    ## @param zeebeGateway.service.loadBalancerSourceRanges defines list of allowed source ip address ranges if the type is LoadBalancer
    loadBalancerSourceRanges: []
    ## @param zeebeGateway.service.httpPort defines the port of the http endpoint, where for example metrics are provided
    httpPort: 9600
    ## @param zeebeGateway.service.httpName defines the name of the http endpoint, where for example metrics are provided
    httpName: "http"
    ## @param zeebeGateway.service.grpcPort defines the port of the gateway gRPC endpoint, where client commands (grpc) are sent to
    grpcPort: 26500
    ## @param zeebeGateway.service.grpcName defines the name of the gateway gRPC endpoint, where client commands (grpc) are sent to
    grpcName: "gateway"
    ## @param zeebeGateway.service.restPort defines the REST port of the gateway REST endpoint, where client commands (REST) are sent to
    restPort: 8080
    ## @param zeebeGateway.service.restName defines the name of the gateway REST endpoint, where client commands (REST) are sent to
    restName: "rest"
    ## @param zeebeGateway.service.internalPort defines the port of the internal api endpoint, which is used for internal communication
    internalPort: 26502
    ## @param zeebeGateway.service.internalName defines the name of the internal api endpoint, which is used for internal communication
    internalName: "internal"

  ## @extra zeebeGateway.serviceAccount configuration for the service account where the gateway pods are assigned to
  serviceAccount:
    ## @param zeebeGateway.serviceAccount.enabled if true, enables the gateway service account
    enabled: true
    ## @param zeebeGateway.serviceAccount.name can be used to set the name of the gateway service account
    name: ""
    ## @param zeebeGateway.serviceAccount.annotations can be used to set the annotations of the gateway service account
    annotations: {}
    ## @param zeebeGateway.serviceAccount.automountServiceAccountToken can be used to control whether the service account token should be automatically mounted
    automountServiceAccountToken: false

  ingress:
    grpc:
      ## @param zeebeGateway.ingress.grpc.enabled if true, an ingress resource is deployed with the Zeebe gateway deployment. Only useful if an ingress controller is available, like nginx.
      enabled: true
      ## @param zeebeGateway.ingress.grpc.className defines the class or configuration of ingress which should be used by the controller
      className: nginx
      ## @param zeebeGateway.ingress.grpc.annotations [object] defines the ingress related annotations, consumed mostly by the ingress controller
      ## @skip zeebeGateway.ingress.grpc.annotations.ingress.kubernetes.io/rewrite-target
      ## @skip zeebeGateway.ingress.grpc.annotations.nginx.ingress.kubernetes.io/ssl-redirect
      ## @skip zeebeGateway.ingress.grpc.annotations.nginx.ingress.kubernetes.io/backend-protocol
      ## @skip zeebeGateway.ingress.grpc.annotations.nginx.ingress.kubernetes.io/proxy-buffer-size
      annotations:
        ingress.kubernetes.io/rewrite-target: "/"
        nginx.ingress.kubernetes.io/ssl-redirect: "false"
        nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
        nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
      ## @param zeebeGateway.ingress.grpc.path defines the path which is associated with the Zeebe gateway's gRPC service and port https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-rules
      path: /
      ## @param zeebeGateway.ingress.grpc.pathType can be used to define the Ingress path type. https://kubernetes.io/docs/concepts/services-networking/ingress/#path-types
      pathType: Prefix
      ## @param zeebeGateway.ingress.grpc.host can be used to define the host of the ingress rule. https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-rules
      # If not specified the rules applies to all inbound gRPC traffic, if specified the rule applies to that host.
      host: "zeebecamunda.local"
      ## @extra zeebeGateway.ingress.grpc.tls configuration for tls on the ingress resource https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
      tls:
        ## @param zeebeGateway.ingress.grpc.tls.enabled if true, then tls is configured on the ingress resource. If enabled the Ingress.host need to be defined.
        enabled: false
        ## @param zeebeGateway.ingress.grpc.tls.secretName defines the secret name which contains the TLS private key and certificate
        secretName: camunda-platform-zeebe-gateway-grpc
    rest:
      ## @param zeebeGateway.ingress.rest.enabled if true, an ingress resource is deployed with the Zeebe gateway deployment. Only useful if an ingress controller is available, like nginx.
      enabled: false
      ## @param zeebeGateway.ingress.rest.className defines the class or configuration of ingress which should be used by the controller
      className: nginx
      ## @param zeebeGateway.ingress.rest.annotations [object] defines the ingress related annotations, consumed mostly by the ingress controller
      ## @skip zeebeGateway.ingress.rest.annotations.ingress.kubernetes.io/rewrite-target
      ## @skip zeebeGateway.ingress.rest.annotations.nginx.ingress.kubernetes.io/ssl-redirect
      ## @skip zeebeGateway.ingress.rest.annotations.nginx.ingress.kubernetes.io/backend-protocol
      ## @skip zeebeGateway.ingress.rest.annotations.nginx.ingress.kubernetes.io/proxy-buffer-size
      annotations:
        ingress.kubernetes.io/rewrite-target: "/"
        nginx.ingress.kubernetes.io/ssl-redirect: "false"
        nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
        nginx.ingress.kubernetes.io/proxy-buffer-size: "128k"
      ## @param zeebeGateway.ingress.rest.path defines the path which is associated with the Zeebe gateway's REST service and port https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-rules
      path: /
      ## @param zeebeGateway.ingress.rest.pathType can be used to define the Ingress path type. https://kubernetes.io/docs/concepts/services-networking/ingress/#path-types
      pathType: Prefix
      ## @param zeebeGateway.ingress.rest.host can be used to define the host of the ingress rule. https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-rules
      # If not specified the rules applies to all inbound gRPC traffic, if specified the rule applies to that host.
      host: ""
      ## @extra zeebeGateway.ingress.rest.tls configuration for tls on the ingress resource https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
      tls:
        ## @param zeebeGateway.ingress.rest.tls.enabled if true, then tls is configured on the ingress resource. If enabled the Ingress.host need to be defined.
        enabled: false
        ## @param zeebeGateway.ingress.rest.tls.secretName defines the secret name which contains the TLS private key and certificate
        secretName: camunda-platform-zeebe-gateway-rest

  # contextPath: "/zeebe"
  ## @param zeebeGateway.contextPath can be used to make Zeebe web application works on a custom sub-path. This is mainly used to run Camunda web applications under a single domain.
  contextPath: "/zeebe"

  ## @param zeebeGateway.configuration if specified, contents will be used as the application.yaml
  configuration: ""
  ## @param zeebeGateway.extraConfiguration if specified, contents will be used for any extra configuration files such as log4j2.xml
  extraConfiguration: {}

Hi @ben051199 - it’s odd since you say you can deploy from Modeler. First, I would try the URL http://zeebecamunda.local:26500 instead of the grpc:// address you have. However, many ingress controllers (for instance, nginx) require SSL for HTTP/2 communication, which is what gRPC uses, so it is possible you need to configure a certificate.

Can you share a screenshot of how you have Modeler configured? Can you see the deployed model in Operate?

hi @nathan.loding,

i use nginx as ingress controller. So you think the solution is to generate a SSL certificate for the local URL?
After i created a SSL certificate i have to add it to the cluster update my Helm config file, enable TLS and add the right name for the certificate am i right? Is there a instruction how to add a certificate and use it in camunda?

small update I have created a certificate and I reach Tasklist (camunda.local/tasklist) with https:// (certificate was approved locally, added in flags.json of the desktop modeler).

Now I get the following error:
zbctl --address zeebecamunda.local:443 --clientId zeebe --clientSecret **** --authzUrl https://camunda.local/auth/realms/camunda-platform/protocol/openid-connect/token --audience zeebe-api status
Error: rpc error: code = Canceled desc = failed to apply token: failed to obtain access token: oauth2: “invalid_scope” “Invalid scopes: Zeebe”

When i configure no scope the error is: “invalid_scope” "Invalid scopes: "

Update:

If I first send the request via the terminal with the zbctl client and then via the Java code, I also get a result. As soon as I wait a bit and only execute the Java code, I get an error again.

Unfortunately, I cannot explain this behavior. Does anyone have an idea what the problem could be?

package com.camunda.academy;

import io.camunda.zeebe.client.ZeebeClient;
import io.camunda.zeebe.client.api.response.BrokerInfo;
import io.camunda.zeebe.client.api.response.DeploymentEvent;
import io.camunda.zeebe.client.api.response.PartitionInfo;
import io.camunda.zeebe.client.api.response.Topology;
import io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider;
import io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProviderBuilder;

import java.net.URI;
import java.net.URISyntaxException;
import java.time.Duration;

public class test {

    public static void main(String[] args) throws URISyntaxException {
        // Konfigurieren des OAuth2 Credentials Providers
        var credentialsProvider = new OAuthCredentialsProviderBuilder()
                .audience("zeebe-api")
                .authorizationServerUrl("https://camunda.local/auth/realms/camunda-platform/protocol/openid-connect/token")
                .clientId("zeebe")
                .clientSecret("****")
                .scope("zeebe-test")
                .build();

        // Konfigurieren und Erstellen des ZeebeClients
        ZeebeClient client = ZeebeClient.newClientBuilder()
                .grpcAddress(new URI("https://zeebecamunda.local"))
                .gatewayAddress("zeebecamunda.local:443")
                .defaultRequestTimeout(Duration.ofSeconds(30))
                //.usePlaintext()
                .credentialsProvider(credentialsProvider)
                .build();

        System.out.println(client.getConfiguration().toString());

        // Abrufen der Topologie und Anzeigen der Cluster-Details
        try {
            Topology topology = client.newTopologyRequest().send().join();

            System.out.println("Cluster size: " + topology.getBrokers().size());
            System.out.println("Partitions count: " + topology.getPartitionsCount());
            System.out.println("Replication factor: " + topology.getReplicationFactor());
            System.out.println("Gateway version: " + topology.getGatewayVersion());

            for (BrokerInfo broker : topology.getBrokers()) {
                System.out.println("Broker " + broker.getNodeId() + " - " + broker.getAddress());
                System.out.println("  Version: " + broker.getVersion());
                for (PartitionInfo partition : broker.getPartitions()) {
                    System.out.println("  Partition " + partition.getPartitionId() + " : " +
                            (partition.isLeader() ? "Leader" : "Follower") + ", " );
                }
            }
        } catch (Exception e){
            e.printStackTrace();
        }
        finally {
            // Schließen des Clients
            client.close();
        }
    }
}

Errormessage:

io.camunda.zeebe.client.api.command.ClientStatusException
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116)
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54)
	at com.camunda.academy.test.main(test.java:41)
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: CANCELLED
	at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
	at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073)
	at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:52)
	... 1 more
Caused by: io.grpc.StatusRuntimeException: CANCELLED
	at io.grpc.Status.asRuntimeException(Status.java:533)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
	at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:489)
	at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:453)
	at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:486)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:574)
	at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:72)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:723)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:751)
	at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636)
	at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
	at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1237)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1123)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1446)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1417)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:219)
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.fetchCredentials(OAuthCredentialsProvider.java:150)
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsCache.computeIfMissingOrInvalid(OAuthCredentialsCache.java:100)
	at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.applyCredentials(OAuthCredentialsProvider.java:79)
	at io.camunda.zeebe.client.impl.ZeebeCallCredentials.lambda$applyRequestMetadata$0(ZeebeCallCredentials.java:49)
	... 3 more

Used Config for client:

gatewayAddress: zeebecamunda.local:443
grpcAddress: https://zeebecamunda.local
restAddress: https://0.0.0.0:8080
defaultTenantId: <default>
jobWorkerMaxJobsActive: 32
numJobWorkerExecutionThreads: 1
defaultJobWorkerName: default
defaultJobTimeout: PT5M
defaultJobPollInterval: PT0.1S
defaultMessageTimeToLive: PT1H
defaultRequestTimeout: PT30S
overrideAuthority: null
maxMessageSize: 4194304
jobWorkerExecutor: null
ownsJobWorkerExecutor: false
streamEnabled: false
preferRestOverGrpc: false

@ben051199 - if your first Modeler screenshot was before you added a certificate, then I would try removing the certificate changes and use the http://zeebecamunda.local address rather than the grpc:// address you were using (my first suggestion, assuming deployments from Modeler were working). You may not need a certificate, it was one observation; a lot depends on your specific deployment and environment.

One other thing: If you are using Camunda 8.5, Zeebe ships with a REST API that will eventually fully replace the gRPC API. It may be simplest to just use the REST API.

@nathan.loding thanks for your answer. Before i added the certificate i tried although http:// and https://

So i can use the Rest api instead of grpc and all works the same ? I just have to update the values and no certificate is needed?

ingress.grpc.enabled: false
ingress.rest.enabled: true

@ben051199 - yes, the REST API provides the same functionality as the gRPC API. I don’t know for sure a certificate was needed to fix your gRPC issue, I just know that’s a common requirement for ingress controllers.

Does camunda’s version 8.5.0 REST zeebe already have an endpoint for Create a process instance?

Hi @Domingos_Dias - not currently. It is in active development, and we are hoping to have it in the 8.6 release but as it’s in active development, it is not guaranteed.

1 Like