Running Zeebe-Helm-Chart on Kubernetes with Kind - Pending ElasticSearch Pods

Thomas Heinrichs: Hey,
I am currently trying to install the Zeebe full Helm chart on my laptop using Kind Kubernetes. Apparently I am running into problems since elasticsearch won’t leave the “pending state”.

Taking a look into the description of the pods I saw that:
• Elasticsearch-master-0 shows a failing Readiness probe since it is waiting for the elasticsearch cluster to become ready
• Elasticsearch-master-1 and 2 show that the scheduling failed due to " 1 node(s) didn’t match pod anti-affinity rules"

I will also attach my kind-values.yaml in case I configured something wrong in there.

Hopefully someone can help me getting it up to work. :slight_smile:
Thanks in advance!


Rin Oliver [they/them] - Slack Administrator: IIRC, based off my own reasearch and experience – If you’re running a single replica cluster add the following helm value:

clusterHealthCheckParams: "wait_for_status=yellow&timeout=1s"

Your status will never go green with a single replica cluster.

The following values might work:

replicas: 1
minimumMasterNodes: 1
clusterHealthCheckParams: 'wait_for_status=yellow&timeout=1s'

You also have three pods, you could try deleting the pod with the highest cpu consumption [elastic-master-0] and then elastic-master-1 and elastic-master-2 should ideally ready up

Thomas Heinrichs: Thanks Rin for your answer!
I tried deleting the pod with elastic-search-master-0 and apparently nothing has changed.

In order to try out the upper part let me ask you: I do need to set those additional parameters in the kind-values.yaml right? (Especially since I am relying on Helm install)

Rin Oliver [they/them] - Slack Administrator: Yes that’s correct

zell: Thanks @Thomas Heinrichs for raising this! I actually wanted to take a look at this since a while. Today I finally had some time to did it. Based on your values and some values https://github.com/camunda-community-hub/zeebe-helm-profiles/blob/751286076a4cb1bbcd4f63b74651e43b120d99a6/zeebe-dev-profile.yaml|here I found a working configuration. I will update the example repo for that.

As @Rin Oliver [they/them] - Slack Administrator mentioned it makes more sense to set the replica to 1, per default it is set to 3. We also need to decrease some of the claimed resources plus very important is to set the storage class to standard, since normally you will not have an “ssd” storage class installed on your kind installation.

Make sure if you try this values file, first uninstall your installed helm release and remove all claimed PV, otherwise it will reuse the pvc’s. Like k delete pvc elastic* k delete pvc data-* etc.

zell: Motto: Pics or didn’t happen

zell: As an optimization we could also remove the standalone gateway and use an embedded gateway for zeebe, but these are details I would say.

Rin Oliver [they/them] - Slack Administrator: Thank you so much <@U6WCLLNGJ>! I’m actually going to try to install these charts on my machine too, so this fix you just pushed will be super helpful!

zell: Already done https://github.com/camunda-community-hub/zeebe-helm-profiles/blob/master/zeebe-dev-profile.yaml :slightly_smiling_face:

zell: @Thomas Heinrichs did it help?

Thomas Heinrichs: Hey @zell :slightly_smiling_face: - I was just trying it out

Good news: Your fixed yaml did boot up all pods properly
Downside: My nginx ingress controller refuses all connections from localhost even though I have forwarded port 80 and 26500

Have you tried to access Operate afterwards or did you just boot up the pods?

zell: Ah damn. No I havent tried that just creating and waiting until they became ready sorry

Thomas Heinrichs: I found out so far that I am able to run
kubectl port-forward svc/zeebe-self-managed-zeebe-operate-helm 8080:80 in order to forward and access Operate.
but apparently this doesn’t work for the Tasklist application.

Regarding the Tasklist it seems that problems regarding ElasticSearch persist. :thinking_face:
I attached a screenshot from a part of the Tasklist stacktrace below.

Thomas Heinrichs: Another idea I had is that…

If you are running in Kubernetes KIND, you will need to port-forward to the ingress controller main entry point due KIND doesn't support LoadBalancers. You can do that by running in a different terminal:

&gt; kubectl port-forward svc/&lt;RELEASE NAME&gt;-nginx-ingress-controller 8080:80 

… this sentence from our documentation might indicate that KIND isn’t working with ingress at all. Due to this I tried the upper part where I wanted to forward each application individually.

zell: @Thomas Heinrichs can you send the complete exception? I see only the end the exception message would be interesting.

With port forwarding it works?

zell: @Thomas Heinrichs I found the issue it is using old pre 1.x properties see here https://github.com/camunda-community-hub/camunda-cloud-helm/blob/main/charts/zeebe-tasklist-helm/templates/configmap.yaml#L9 we have to change them to camunda.tasklist I will do it after lunch and release a new version :slightly_smiling_face:

Thomas Heinrichs: Cool that you found the issue @zell :slightly_smiling_face: - Apparently I haven’t seen your previous message. Somehow the notification has slipped through.

Thomas Heinrichs: Additionally kubectl logs -f didn’t return the full stack trace. :disappointed: Due to this I have just checked out what might be meaningful :smile:

Thomas Heinrichs: Have a great lunch!
I will probably test it out on tuesday

Note: This post was generated by Slack Archivist from a conversation in the Zeebe Slack, a source of valuable discussions on Zeebe (get an invite). Someone in the Slack thought this was worth sharing!

If this post answered a question for you, hit the Like button - we use that to assess which posts to put into docs.

zell: Was fixed by community contribution https://github.com/camunda-community-hub/camunda-cloud-helm/pull/59 :tada:

zell: Ok new helm chart release is out https://github.com/camunda-community-hub/camunda-cloud-helm/releases/tag/zeebe-tasklist-helm-1.0.6 which is included in https://github.com/camunda-community-hub/camunda-cloud-helm/releases/tag/zeebe-full-helm-1.2.2 Tasklist seems to be able to connect now.

It looks like tasklist was generating lot of data, because some dev profile were on. https://github.com/camunda-community-hub/camunda-cloud-helm/blob/main/charts/zeebe-tasklist-helm/templates/deployment.yaml#L22 I will remove this and update the kind values file for now.

zell: Elastic disk size needs to be a bit bigger to get it working, after increasing to 15 Gig it works for me. Let me know if you see any other issues. Happy to help :slightly_smiling_face:

Thomas Heinrichs: Awesome it worked! Thanks for the quick fix.
Nevertheless I think it won’t work with the ingress controller. Somehow it shows up some problems with the session (see below). Forwarding Tasklist and Cockpit individually works perfectly fine.

Do you have the same “findings” regarding ingress. In case you do, let me know @zell and I will update the documentation. :slightly_smiling_face:

1 Like