Problem Setting the Cluster Endpoint

LexEgg · June 11, 2024, 4:25pm

Hello,
I have a rancher kubernetes setup with the camunda helm chart installed. Now I wanted to deploy a spimple process. But there is an error and I don’t know what to do (I’m new to Camunda Self-Managed).

All pods in my cluster are running but there is this warning in the logs of the zeebe-gateway: “Unexpected error while handling message cluster-topology-sync from camunda-platform-zeebe-0.camunda-platform-zeebe.kw-b07d10-default.svc:26502
io.atomix.cluster.messaging.MessagingException$NoSuchMemberException: Failed to handle message, host camunda-platform-zeebe-0.camunda-platform-zeebe.kw-b07d10-default.svc:26502 is not a known cluster member”.
(there is also the same message for zeebe-1 and zeebe-2)

Does anyone know what I can do?

Thanks a lot

Domingos_Dias · June 30, 2024, 10:30pm

Check if your zeebe is in fact running on localhost:26500, it may have a different port configured or IP defined

LexEgg · July 1, 2024, 12:33pm

The connection was possible with zbctl. I guess thats not the problem.
Now I am switching to combined ingress. But there is also a connection problem. I can connect to keycloak but when I am trying to connect to operate or tasklist I am forwarded to a localhost address…
But I cannot see any settings that are related to a localhost.

Any ideas?

Domingos_Dias · July 1, 2024, 7:47pm

You can check the redirect settings of operate and tasklist on clients in keycloak, if you have a specific IP configured on your artifacts, change the routes according to the correct address

LexEgg · July 2, 2024, 8:12am

I did that but if I want to connect to operate or tasklist using the specified url, I am still forwarded to localhost:18080…

Domingos_Dias · July 2, 2024, 8:40am

Can you share your client settings on keycloak?

LexEgg · July 2, 2024, 10:03am

The Keycloak clients settings for operate (and tasklist - it’s the same):

I blacked out the sensible data. For example it’s like “https://camunda.namespace.rancher-host.com”.

That’s the values.yaml:

global:
  ingress:
    enabled: true
    className: nginx
    host: "camunda.namespace.rancher-host.com"
  identity:
    auth: 
      publicIssuerUrl: "https://camunda.namespace.rancher-host.com/auth/realms/camunda-platform"
      operate:
        redirectUrl: "https://camunda.namespace.rancher-host.com/operate"
      tasklist:
        redirectUrl: "https://camunda.namespace.rancher-host.com/tasklist"

identity:
  auth:
    publicUrl: "https://camunda.namespace.rancher-host.com/identity" 
  contextPath: "/identity"

operate:
  contextPath: "/operate"
  webAppUrl: "https://camunda.namespace.rancher-host.com/operate"

tasklist:
  contextPath: "/tasklist"
  webAppUrl: "https://camunda.namespace.rancher-host.com/tasklist"

zeebe-gateway:
  ingress:
    grpc:
      enabled: true
      className: nginx
      host: "camunda-gw.namespace.rancher-host.com"

# Disable Optimize
optimize:
  enabled: false

(The configurations for ingress are managed by the company I work for. It’s configured in the k8-cluster)

nathan.loding · July 2, 2024, 6:39pm

@LexEgg - when is the redirect to localhost happening? The usual flow would be something like:

Open https://…/operate in your browser
The token is unavailable or expired
Redirect to an Identity URL to start the OAuth flow
Redirect to the login page
Enter login credentials
Browser redirects to Identity
Browser redirects to https://…/operate after authentication is complete

At what step is the redirect to localhost happening? (I also probably got something wrong in that list, but you get the idea! )

LexEgg · July 4, 2024, 10:46am

@nathan.loding
Thanks a lot for having a look at my problem. Actually I solved that part of the problem. I just needed to apply the ingress.yaml with my install:
helm install camunda camunda/camunda-platform -f ingress.yam
It’s because the helm chart used “port forward” in its default values.

But there is still one problem:
I can’t connect to my zeebe-gateway. I always get this warning in my Desktop Modeler in the deploy window:

I made an extra ingress for the gateway. All other ingresses are working and i can connect to operate and tasklist using the specified URLs. I also tested the connection to the zeebe-gateway using zbctl but I also get the error “unavailable”.

(In a YouTube video I saw someone setting up a Loadbalancer. With “kubectl get services” I don’t get any loadbalancer in my list. Could this be a problem?)

C:\Users\eggart>kubectl get services
NAME                                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
camunda-platform-connectors                ClusterIP   10.218.109.76   <none>        8080/TCP                       15h
camunda-platform-elasticsearch             ClusterIP   10.218.77.118   <none>        9200/TCP,9300/TCP              15h
camunda-platform-elasticsearch-master-hl   ClusterIP   None            <none>        9200/TCP,9300/TCP              15h
camunda-platform-identity                  ClusterIP   10.218.80.118   <none>        80/TCP,82/TCP                  15h
camunda-platform-keycloak                  ClusterIP   10.218.75.46    <none>        80/TCP                         15h
camunda-platform-keycloak-headless         ClusterIP   None            <none>        80/TCP                         15h
camunda-platform-operate                   ClusterIP   10.218.116.44   <none>        80/TCP                         15h
camunda-platform-optimize                  ClusterIP   10.218.121.25   <none>        80/TCP,8092/TCP                15h
camunda-platform-postgresql                ClusterIP   10.218.124.19   <none>        5432/TCP                       15h
camunda-platform-postgresql-hl             ClusterIP   None            <none>        5432/TCP                       15h
camunda-platform-tasklist                  ClusterIP   10.218.79.137   <none>        80/TCP                         15h
camunda-platform-zeebe                     ClusterIP   None            <none>        9600/TCP,26502/TCP,26501/TCP   15h
camunda-platform-zeebe-gateway             ClusterIP   10.218.75.80    <none>        9600/TCP,26500/TCP,8080/TCP    15h

nathan.loding · July 5, 2024, 2:31pm

@LexEgg - if zbctl isn’t connecting either, and just gives “unavailable,” it sounds like there is a networking issue between your client and the zeebe-gateway service. The ingress configuration is the first place I would look; I’m personally not familiar with Rancher and I’m not sure if anything additional might be needed. You should only need a load balancer if you need load balancing, and from what you’ve described so far, I don’t think you do.