I could not make sense of this work without research. After three years at CERN, back in Bilbao I decided to work on my own and keep researching and learning. I believe entrepreneurship is the ultimate professional challenge — and, at the same time, an expression of creativity and a claim to freedom.
Zetesis comes from the Greek ζήτησις, inquiry: find the cause before proposing the solution.
Which data is worth backing up and in what order it gets restored is covered in Setting up a homelab from scratch: backups and restore, so I will not repeat it here. As in the previous posts, I do not give machine names, addresses or domains.
Every machine backs itself up with restic
File backups are made by restic, a backup program that stores data in a repository: a folder, or a bucket in S3-compatible storage, with the data split into chunks and encrypted. Each workload has its own repository, so the production backups never mix with a VPS's.
What changes from one machine to another is who launches restic. In the production cluster it is K8up, a Kubernetes operator: each namespace declares in a Schedule resource which volumes it backs up, to which repository and when, and K8up starts a pod running restic at that time. In the platform cluster it is VolSync, another operator, which takes a ZFS snapshot of the volume before backing it up and clones it into a temporary one. Restic reads that clone, a still picture of the disk, instead of a volume that is being written to.
The VPSs have no Kubernetes. There restic runs in a container next to the services, with their volumes mounted read-only, and schedules itself. Almost everything is backed up in the early hours.
Restic encrypts and deduplicates before anything leaves
Before sending anything, restic cuts each file into chunks based on its content and computes a fingerprint for each chunk. If a chunk is already in the repository, it is not uploaded again: a file that has not changed since yesterday does not travel, and two identical files take the space of one. Then it encrypts every chunk with the repository key.
Only encrypted chunks reach the storage: without the password, a restic repository cannot be read. And each night's backup takes little more than what changed that day.
PostgreSQL goes its own way
The clusters' databases have their own path, with tools that understand PostgreSQL. They are managed by CloudNativePG, the operator that runs PostgreSQL inside Kubernetes, and its Barman plugin stores two things in MinIO: the continuous archive of the WAL, the log where PostgreSQL writes down every change, and a base backup every day. Together they let me recover a database up to a specific moment, and they are kept for 30 days.
On top of that, every night K8up starts a pod that dumps the main databases with pg_dump before backing up the data namespace, and restic stores that dump in its repository like any other file. It is a second copy in another format, which can be restored into any PostgreSQL.
Every Sunday, a test restores each database
A backup that has never been restored is a hypothesis. For PostgreSQL I have a weekly test: a CronJob, a scheduled Kubernetes task, that goes through the production cluster's databases early on Sunday morning. For each one it creates an empty database with its own prefix, loads the original into it with pg_dump and pg_restore, checks that it has tables and deletes it. If one fails, the test carries on with the rest and ends in error, which fires an alert.
The test starts from the live database, not from the copy stored in MinIO. What it checks is that each database can be dumped and rebuilt in full, with no corrupted objects or permissions getting in the way. It skips databases over 2 GB, because the disposable copy takes as much space as the original, and it writes down which ones it skipped, so that «nothing failed» is never mistaken for «everything was checked».
Backups travel over Tailscale to a single port
Every backup, restic's and PostgreSQL's alike, goes to the same place: MinIO, an S3-compatible object store running on the TrueNAS in the other house. They travel over Tailscale, the private network between machines I described in the network post. The clusters go out through a Tailscale egress, a proxy that the Tailscale operator runs inside the cluster to carry an internal service's traffic to the NAS. The VPSs use their own Tailscale connection or a gateway container that shares its network with the restic one.
The Tailscale policy lets the four VPSs and the two clusters reach the NAS on a single port, MinIO's. In the access table of the network post, it is the backup storage row, the one with «just one» port.
A 16 TB disk in the other house and another in the main one
MinIO keeps the repositories on a 16 TB disk in the TrueNAS. Being in the other house is what matters: a fire or a burglary in the main house would take the servers, but not their backups.
Every now and then I copy the TrueNAS's file root by hand to another 16 TB disk I keep in the main house, with FreeFileSync, a program that compares two folders and copies the differences. That way, if one house goes down, a copy survives in the other: if the main house goes, the backups are on the TrueNAS, and if the other one goes, they are on the disk at home.
The backups are declared in git too
All of the above is configuration in the infrastructure repository: K8up's Schedule resources, VolSync's sources, CloudNativePG's backups with their retention, the Sunday test, the Compose files for the VPS containers and the Tailscale policy. In the clusters, Argo CD, the agent that keeps each cluster in sync with git, applies it on merge into main. Changing the time of a backup or adding a volume is a commit that gets reviewed beforehand and stays on record afterwards.
01
02
03
04
