Really enjoyed reading through this whole thread, the Pi cluster approach is great and honestly the constraints are half the value. Making it work on nodes with 1GB of RAM teaches you more about what Kubernetes is actually doing than throwing a beefy box at it ever would.
I run a similar setup but on the other end of the hardware spectrum, three-node k3s on top of Proxmox instead of bare Pis, and it’s funny how many of the same lessons show up regardless of scale. The namespace-per-experiment workflow you described is exactly how I work too, spin it up in its own namespace, and if the test goes sideways you just delete the namespace and it’s like it never happened. Cleanest undo button in tech.
On the Velero thing, that one hit home. I run Velero for nightly cluster backups too, and I’ll admit I did the thing everyone does: scheduled them for months and never actually tested a restore. Finally made myself wipe the cluster state on purpose just to find out if the backups were real. Came back in about eleven seconds, clean, and that was the moment I actually trusted it. Scheduling a backup and restoring one are two completely different levels of confidence, and you don’t earn the second one until you’ve done the scary part. Your go-velero-manager is a cool answer to the cleanup annoyance, deleting old backups through the CLI is genuinely tedious, so building the tool you wished existed is the whole spirit of this hobby.
And to the person who suggested adding a health check after each node reboot in the Ansible play, listen to that, it’s the kind of thing you don’t think you need until a node comes back up but doesn’t actually rejoin the cluster and you’re chasing a ghost for an hour.