Camunda 8.3.4 - Operate not able to connect to Zeebe Partitions

Hi Team,
I am installing Camunda 8.3.4 version on my Non prod Openshift environment using camunda helm charts. However, Operate fails to start . I get an error -
Caused by: java.io.IOException: Failed while requesting access token with status code 401 and message Unauthorized.
4617 at io.camunda.zeebe.client.impl.oauth.OAuthCredentialsProvider.fetchCredentials(OAuthCredentialsProvider.java:158) ~[zeebe-client-java-8.3.4.jar!/:8.3.4]

46252024-01-02 11:54:27.029 INFO 7 — [ main] i.c.o.z.PartitionHolder : Partition ids can’t be fetched from Zeebe. Return empty partition ids list.

To fix this I tried setting the env property in zeebee gaeway - ZEEBE_GATEWAY_SECURITY_AUTHENTICATION_MODE as none. But the error still persists. I even tried updating the zeebeGateway template>deployment/yaml with this property.But it didnt worked as well.

Please provide some guidance. I had same issue with the 8.2.11 version and this property fixed the issue.

Hi @ShilpaN.Chalke - do you want to run your stack with Identity for authentication or with no authentication? Can you share your values.yaml file (with secrets redacted)?

Hi @nathan.loding , not sure If I understood your qns correctly
I would prefer to run the entire camunda stack (except connectors, webmodeler and console ) with Identity for authentication (i.e basic authentication and auto generated clientId and secrets by helm) , but if there is an issue… for now for non prod environment I am ok with just basic authentication as well (just with basic demo/demo ) should also work.

So these are the values i have configured for latest helm -

for operate.global

 operate:
        ## @param global.identity.auth.operate.clientId defines the client id, which is used by Operate in athentication flows.
        clientId: operate
        ## @param global.identity.auth.operate.audience defines the audience, which is used by Operate.
        audience: operate-api
        ## @param global.identity.auth.operate.existingSecret can be used to reference an existing secret. If not set, a random secret is generated.
        # The existing secret should contain an `operate-secret` field, which will be used as secret for the identity-Operate communication.
        existingSecret:
        ## @param global.identity.auth.operate.redirectUrl defines the redirect URL, which is used by Keycloak to access Operate.
        # Should be publicly accessible, the default value works if a port-forward to Operate is created to 8081.
        # Can be overwritten if ingress is in use and an external IP is available.
        redirectUrl: "http://xyz.com"


for Zeebe-Gateway

zeebe-gateway:
  ## @param zeebe-gateway.replicas defines how many standalone gateways are deployed
  replicas: 2
  ## @extra zeebe-gateway.image configuration to configure the zeebe-gateway image specifics
  image:
    ## @param zeebe-gateway.image.registry can be used to set container image registry.
    registry: ""
    ## @param zeebe-gateway.image.repository defines which image repository to use
    repository: repoName
    ## @param zeebe-gateway.image.tag can be set to overwrite the global tag, which should be used in that chart
    tag: 8.3.4
    ## @param zeebe-gateway.image.pullSecrets can be used to configure image pull secrets https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
    pullSecrets: []

  ## @param zeebe-gateway.sidecars can be used to attach extra containers to the zeebe gateway deployment
  sidecars: []

  ## @param zeebe-gateway.podAnnotations can be used to define extra gateway pod annotations
  podAnnotations: {}
  ## @param zeebe-gateway.podLabels can be used to define extra gateway pod labels
  podLabels: {}

  ## @param zeebe-gateway.logLevel defines the log level which is used by the gateway
  logLevel: info
  ## @param zeebe-gateway.log4j2 can be used to overwrite the log4j2 configuration of the gateway
  log4j2: ''
  ## @param zeebe-gateway.javaOpts can be used to set java options for the zeebe gateways
  javaOpts: >-
    -XX:+ExitOnOutOfMemoryError

  ## @param zeebe-gateway.env can be used to set extra environment variables in each gateway container
 
  env:
    - name: ZEEBE_GATEWAY_SECURITY_AUTHENTICATION_MODE
      value: NONE  /// it by default set this value to identity

for operate
## @section Operate Parameters
## @extra.operate configuration for the Operate sub chart.
operate:
  ## @param operate.enabled if true, the Operate deployment and its related resources are deployed via a helm release
  enabled: true

  ## @extra operate.image configuration to configure the Operate image specifics
  image:
    ## @param operate.image.registry can be used to set container image registry.
    registry: ""
    ## @param operate.image.repository defines which image repository to use
    repository: repoName
    ## @param operate.image.tag can be set to overwrite the global tag, which should be used in that chart
    tag: 8.3.4
    ## @param operate.image.pullSecrets can be used to configure image pull secrets https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
    pullSecrets: []

  ## @param operate.sidecars can be used to attach extra containers to the operate deployment
  sidecars: []
  ## @param operate.initContainers can be used to set up extra init containers for the operate pods, useful for additional exporters
  initContainers: []
  # contextPath: "/operate"
  ## @param operate.contextPath can be used to make Operate web application works on a custom sub-path. This is mainly used to run Camunda web applications under a single domain.
  contextPath: ""

  ## @param operate.podAnnotations can be used to define extra Operate pod annotations
  podAnnotations: {}
  ## @param operate.podLabels can be used to define extra Operate pod labels
  podLabels: {}

  ## @extra operate.logging configuration for the Operate logging. This template will be directly included in the Operate configuration yaml file
  ## @param operate.logging.level.ROOT
  ## @param operate.logging.level.io.camunda.operate
  logging:
    level:
      ROOT: INFO
      io.camunda.operate: DEBUG

  ## @extra operate.service configuration to configure the Operate service.
  service:
    ## @param operate.service.type defines the type of the service https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types
    type: ClusterIP
    ## @param operate.service.port defines the port of the service, where the Operate web application will be available
    port: 80
    ## @param operate.service.annotations can be used to define annotations, which will be applied to the Operate service
    annotations: {}

  ## @extra operate.resources configuration to set request and limit configuration for the container https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#requests-and-limits
  ## @param operate.resources.requests.cpu
  ## @param operate.resources.requests.memory
  ## @param operate.resources.limits.cpu
  ## @param operate.resources.limits.memory
  resources:
    requests:
      cpu: 600m
      memory: 400Mi
    limits:
      cpu: 2000m
      memory: 2Gi

  ## @param operate.env can be used to set extra environment variables in each Operate container

  env:
    - name: CAMUNDA_OPERATE_ENTERPRISE
      value: "false"
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_USERNAME
      value: "username"
    - name: CAMUNDA_OPERATE_ELASTICSEARCH_PASSWORD
      valueFrom:
        secretKeyRef:
          name:keyname
          key: key
    - name: CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_USERNAME
      value: "username"
    - name: CAMUNDA_OPERATE_ZEEBEELASTICSEARCH_PASSWORD
      valueFrom:
        secretKeyRef:
          name: keyname
          key: key
    - name: CAMUNDA_OPERATE_IDENTITY_ISSUER_URL
      value: "http://keycloak/auth/realms/camunda-platform"
  ## @extra operate.configMap configuration which will be applied to the mounted config map.
  configMap:
    ## @param operate.configMap.defaultMode can be used to set permissions on created files by default. Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.
    # See https://github.com/kubernetes/api/blob/84c4768022421f4d420a8d74e6699077021563ce/core/v1/types.go#L1615-L1623
    defaultMode: 0744

Hi @ShilpaN.Chalke - is that the complete values.yaml file? It looks edited. Can you share the full file and only edit the values where needed (like the secrets)?

If you want to use Identity, then the authentication mode should be set to identity and not none.

I am using Identity mode of authetication
global.identity.auth.enabled is true

All the pods of camunda are running except for Operate -
It fails with error -

83 DEBUG 7 — [SelectorManager] j.i.h.debug : [HttpClient-4-SelectorManager] [282s 546ms] HttpClientImpl(4) Next deadline is 3000
2024-01-17 10:41:50.213 WARN 7 — [ main] i.c.o.z.PartitionHolder : Error occurred when requesting partition ids from Zeebe client: null
io.camunda.zeebe.client.api.command.ClientStatusException: null
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.transformExecutionException(ZeebeClientFutureImpl.java:116) ~[zeebe-client-java-8.4.0.jar!/:8.4.0]
at io.camunda.zeebe.client.impl.ZeebeClientFutureImpl.join(ZeebeClientFutureImpl.java:54) ~[zeebe-client-java-8.4.0.jar!/:8.4.0]

Caused by: io.grpc.StatusRuntimeException: CANCELLED
at io.grpc.Status.asRuntimeException(Status.java:537) ~[grpc-api-1.60.0.jar!/:1.60.0]
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)

Caused by: java.net.SocketTimeoutException: Connect timed out
at sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:551) ~[?:?]

2024-01-17 10:42:14.256 INFO 7 — [ main] i.c.o.z.PartitionHolder : Partition ids can’t be fetched from Zeebe. Try next round (51).

Let me see if I can edit and my values.yaml here

Yes, please share the complete values.yaml file (with secrets removed). It’s hard to say what might be causing issues without seeing the full configuration.

@nathan.loding I am unable to attach the file here.

@ShilpaN.Chalke - hmm, that’s strange. Can you email it perhaps? I’ll send you a DM.

@nathan.loding The issue is resolved. We were pointing to incorrect token url in helm chart.

1 Like

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.