Thomas Heinrichs: Hey,
I am currently trying to install the Zeebe full Helm chart on my laptop using Kind Kubernetes. Apparently I am running into problems since elasticsearch won’t leave the “pending state”.
Taking a look into the description of the pods I saw that:
• Elasticsearch-master-0 shows a failing Readiness probe since it is waiting for the elasticsearch cluster to become ready
• Elasticsearch-master-1 and 2 show that the scheduling failed due to " 1 node(s) didn’t match pod anti-affinity rules"
I will also attach my kind-values.yaml in case I configured something wrong in there.
Hopefully someone can help me getting it up to work.
Thanks in advance!
Rin Oliver [they/them] - Slack Administrator: IIRC, based off my own reasearch and experience – If you’re running a single replica cluster add the following helm value:
Your status will never go green with a single replica cluster.
The following values might work:
replicas: 1 minimumMasterNodes: 1 clusterHealthCheckParams: 'wait_for_status=yellow&timeout=1s'
You also have three pods, you could try deleting the pod with the highest cpu consumption [elastic-master-0] and then elastic-master-1 and elastic-master-2 should ideally ready up
Thomas Heinrichs: Thanks Rin for your answer!
I tried deleting the pod with elastic-search-master-0 and apparently nothing has changed.
In order to try out the upper part let me ask you: I do need to set those additional parameters in the kind-values.yaml right? (Especially since I am relying on Helm install)
Rin Oliver [they/them] - Slack Administrator: Yes that’s correct
zell: Thanks @Thomas Heinrichs for raising this! I actually wanted to take a look at this since a while. Today I finally had some time to did it. Based on your values and some values https://github.com/camunda-community-hub/zeebe-helm-profiles/blob/751286076a4cb1bbcd4f63b74651e43b120d99a6/zeebe-dev-profile.yaml|here I found a working configuration. I will update the example repo for that.
As @Rin Oliver [they/them] - Slack Administrator mentioned it makes more sense to set the replica to 1, per default it is set to 3. We also need to decrease some of the claimed resources plus very important is to set the storage class to standard, since normally you will not have an “ssd” storage class installed on your kind installation.
Make sure if you try this values file, first uninstall your installed helm release and remove all claimed PV, otherwise it will reuse the pvc’s. Like
k delete pvc elastic*
k delete pvc data-* etc.
zell: Motto: Pics or didn’t happen
zell: As an optimization we could also remove the standalone gateway and use an embedded gateway for zeebe, but these are details I would say.
Rin Oliver [they/them] - Slack Administrator: Thank you so much <@U6WCLLNGJ>! I’m actually going to try to install these charts on my machine too, so this fix you just pushed will be super helpful!
zell: @Thomas Heinrichs did it help?
Thomas Heinrichs: Hey @zell - I was just trying it out
Good news: Your fixed yaml did boot up all pods properly
Downside: My nginx ingress controller refuses all connections from localhost even though I have forwarded port 80 and 26500
Have you tried to access Operate afterwards or did you just boot up the pods?
zell: Ah damn. No I havent tried that just creating and waiting until they became ready sorry
Thomas Heinrichs: I found out so far that I am able to run
kubectl port-forward svc/zeebe-self-managed-zeebe-operate-helm 8080:80 in order to forward and access Operate.
but apparently this doesn’t work for the Tasklist application.
Regarding the Tasklist it seems that problems regarding ElasticSearch persist.
I attached a screenshot from a part of the Tasklist stacktrace below.
Thomas Heinrichs: Another idea I had is that…
If you are running in Kubernetes KIND, you will need to port-forward to the ingress controller main entry point due KIND doesn't support LoadBalancers. You can do that by running in a different terminal: > kubectl port-forward svc/<RELEASE NAME>-nginx-ingress-controller 8080:80
… this sentence from our documentation might indicate that KIND isn’t working with ingress at all. Due to this I tried the upper part where I wanted to forward each application individually.
zell: @Thomas Heinrichs can you send the complete exception? I see only the end the exception message would be interesting.
With port forwarding it works?
zell: @Thomas Heinrichs I found the issue it is using old pre 1.x properties see here https://github.com/camunda-community-hub/camunda-cloud-helm/blob/main/charts/zeebe-tasklist-helm/templates/configmap.yaml#L9 we have to change them to camunda.tasklist I will do it after lunch and release a new version
Thomas Heinrichs: Cool that you found the issue @zell - Apparently I haven’t seen your previous message. Somehow the notification has slipped through.
Thomas Heinrichs: Additionally
kubectl logs -f didn’t return the full stack trace. Due to this I have just checked out what might be meaningful
Thomas Heinrichs: Have a great lunch!
I will probably test it out on tuesday
Note: This post was generated by Slack Archivist from a conversation in the Zeebe Slack, a source of valuable discussions on Zeebe (get an invite). Someone in the Slack thought this was worth sharing!
If this post answered a question for you, hit the Like button - we use that to assess which posts to put into docs.