I know that for data storage the best bet is a NAS and RAID1 or something in that vein, but what about all the docker containers you are running, carefully configured services on your rpi, installed *arr services on your PC, etc.?
Do you have a simple way to automate backups and re-installs of these as well or are you just resigned to having to eventually reconfigure them all when the SD card fails, your OS needs a reinstall or the disk dies?
There’s lots of very good approaches in the comments.
But I’d like to play the devil’s advocate: how many of you have actually recovered from a disaster that way? Ideally as a test, of course.
A backup system that has never done a restore operations must be assumed to be broken. similar logic should be applied to disaster recovery.
And no: I use Ansible/Docker combined approach that I’m reasonably sure could quite easily recover most stuff, but I’ve not yet fully rebuilt from just that yet.
My configuration and deployment is managed entirely via an Ansible playbook repository. In case of absolute disaster, I just have to redeploy the playbook. I do run all my stuff on top of mirrored drives so a single failure isn’t disastrous if I replace the drive quickly enough.
For when that’s not enough, the data itself is backed up hourly (via ZFS snapshots) to a spare pair of drives and nightly to S3 buckets in the cloud (via restic). Everything automated with systemd timers and some scripts. The configuration for these backups is part of the playbooks of course. I test the backups every 6 months by trying to reproduce all the services in a test VM. This has identified issues with my restoration procedure (mostly due to potential UID mismatches).
And yes, I have once been forced to reinstall from scratch and I managed to do that rather quickly through a combination of playbooks and well tested backups.
Radical suggestion:
- Once a year you buy a hard drive that can handle all of your data.
- rsync everything to it
- unplug it, put it back in cold storage
Once a… year? There’s a lot that can change in a year. Cloud storage can be pretty cheap these days. Backup to something like backblaze, S3 or Glacier nightly instead.
- Most systems are provisioned in proxmox with terraform.
- Configuration and setup is handled via ansible playbooks after the server is available. 2.a) Do NOT make changes on the server without updating your ansible scripts - except during troubleshooting. 2.b) Once troubleshooting is done delete and re-create the VM from scratch using only scripts to ensure it works.
- VM storage is considered to be ephemeral. All long-term data/config that can’t be re-created with ansible is either stored on an NFS server with a RAID5 dive configuration or backed up to that same file-server using rsnapshot.
- NFS server is backed-up nightly to backblaze using duplicacy.
- Any other non-VM systems like personal laptops and the like are backed up nightly to the file-server using rsnapshot. Those snapshots are then backed up to backblaze using duplicacy.
The most useful philosophy I’ve come across is “make the OS instance disposable.” That means an almost backups-first approach. Everything of importance to me is thoroughly backed up so once main box goes kaput, I just have to pull the most recent copy of the dataset and provision it on a new OS, maybe new hardware if needed. These days, it’s not that difficult. Docker makes scripting backups easy as pie. You write your docker-compose so all config and program state lives in a single directory. Back up the directory, and all you need to get up and running again with your services is access to Docker Hub to fetch the application code.
Some downsides with this approach (Docker’s security model sorta assumes you can secure/segment your home network better than most people are actually able to), but honestly, for throwing up a small local service quickly it’s kind of fantastic. Also, if you decide to move away from Docker the experience will give you insight into what amounts to program state for the applications you use which will make doing the same thing without Docker that much easier.
virtualize the machine with proxmox, use proxmox backup server, load vm on new system if you get catastrophic failure on the machine running the vm currently.
I rsync my root and everything under it to a NAS, will hopefully save my data. I wrote some scripts manually to do that.
I think the next best thing to do is to doco your setup as mich as possible. Either by typed up notes, or ansible/packer/whatever, any documentation is better than nothing if you have to rebuild.
I run history and then clean it up so i have a guide to follow on the next setup. It’s not even so much for drive failure but to move to the newer OS versions when available.
The ‘data’ is backed up by scripts that tar folders up and scp them off to another server.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters Git Popular version control system, primarily for code HA Home Assistant automation software ~ High Availability LXC Linux Containers NAS Network-Attached Storage Plex Brand of media server package RAID Redundant Array of Independent Disks for mass storage RPi Raspberry Pi brand of SBC SBC Single-Board Computer SSD Solid State Drive mass storage
8 acronyms in this thread; the most compressed thread commented on today has 6 acronyms.
[Thread #287 for this sub, first seen 18th Nov 2023, 10:35] [FAQ] [Full list] [Contact] [Source code]
