Does anyone have experience with backup and restore of Camunda 8 running in Google Kubernetes Engine?
I can create the backup, but when it comes to restoring from Zeebe, the instructions are incomprehensible to me.
Zeebe backs up to a bucket and how do I then perform the restore? Zeebe runs as a container and I also found the restore script in the container. If I simply call it, it aborts because there is data under /usr/local/zeebe/data. Isn’t it the case that when I start the Zeebe container, there will always be data there and thus the restore cannot work?
The really important question is how do you restore Zeebe when it is running in a container in Kubernetes?
Hi @jrh.koenig - when restoring a Zeebe backup, you are effectively creating a new cluster rather than restoring data to an existing broker. The restore script provided takes a Zeebe distribution and installs it to a node with the backup. Or put another way, you aren’t restoring the data, you are restoring the cluster.
Does that help at all? I’m not sure if that answers your question or not! 
hi @nathan.loding, thank you for your reply.
I now understand that it is not just a data recovery but a cluster recovery including the data.
As the process is quite complex, I wonder if it is not easier with Velero. Velero takes snapshots of the persistent volumes and the containers. When restoring with Velero, the PV and the containers are restored. Maybe pause Zeebe before the Velero backup and then, when Velero is finished with the backup, export Zeebe again.
As it says in the Camunda instructions, I don’t have a tar if it is in the GCS bucket to copy it to a node, unpack it and run the restore script. I have a pod with a container in Kubernetes and not a normal Linux server.
The Velero thing is what I am thinking of doing to simplify things.
@jrh.koenig - there are definitely other ways to back up pods in Kubernetes, and they all should work as long as everything being backed up and restored. The downside to those approaches is exactly what you mentioned: you have to stop processing in Zeebe. The backups done by Camunda are hot backups that don’t significantly impact processing. Depending on your needs, one solution might be better than the other!
Hi @jrh.koenig!
maybe this example helps: GitHub - ManuelDittmar/backup-restore-demo
This blog post describes very well why zeebe has it’s own backup procedure: The Inner Workings of Hot Backups in Zeebe | Camunda
Hope that helps.
Kind Regards,
Manu
2 Likes
Hi Manu,
thank you very much. That helps.
Kind Regards
Johannes
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.