Oh boy that’s a loaded question for me.
Started migrating my SPOF server setup with docker-compose, that hosts my media and home automation setup to a k3s 4 node cluster deployment, in order to get things to be mostly HA.
I have the k3s cluster setup with Cilium and it does L2 ARP broadcasts to make the control plane HA alongside a few other apps like Traefik and PiHole. I also have Vault setup to store all my secrets and cert-manager to generate letsEncrypt certificates for all my services.
The idea was to have all my media moved to an NFS and to use longhorn as a distributed storage for my configs and DBs. Unfortunately it turns out that longhorns performance is less than ideal, and my fallback of storing my DBs and configs temporarily on my old server acting as NFS also did not work extremely well, most likely because of a network bottleneck.
So for now I have the Pods running with local storage with the exception of a few things like PiHole and Vault that I definitely want to be HA. And I did a full DR simulation and know I can restore from backup and do a full data recovery from the cloud in about 3hrs (data restore). I’ll eventually tackle moving configs and DBs off local storage again but not sure when.
I now have my full set of media (plex and *arr) apps running on k8s. I’ll also be migrating the home automation stuff soon.
On a side note I’ve grown to hate Duplicati it’s extremely slow and 90% of the time just plain fails to restore files. I’ve ended up moving to Kopia which seems to be working ok but isn’t the most intuitive.
P.S. Please forgive the unorganized brain dump, it’s late and it was a long day.
I can’t say for GKE, but I believe for EKS all features enabled by default on the K8S version are an enabled on EKS.