I could not make sense of this work without research. After three years at CERN, back in Bilbao I decided to work on my own and keep researching and learning. I believe entrepreneurship is the ultimate professional challenge — and, at the same time, an expression of creativity and a claim to freedom.
Zetesis comes from the Greek ζήτησις, inquiry: find the cause before proposing the solution.
Building a homelab from scratch: security and operations - Homelab (04/06)
Case studyHomelabSecurityOperations
In the Post 3 we saw how applications go from Git to running containers. But we skipped a critical question: how do secrets get into those containers?
A database password, an API token, a TLS certificate — none of this can live in Git in plain text. And once deployed, they need to be backed up. And if everything fails, you need a way to rebuild from scratch.
This post covers the three layers of secrets management, the backup strategy, home automation, and what you would need to recover from a total disaster. It is the boring but essential part that makes the difference between a weekend project and infrastructure you can rely on.
The secrets problem
The entire homelab is managed from a single Git repository. Great for reproducibility and auditing, but it creates a tension: configurations have to be in Git for GitOps deployment to work, secrets cannot be in Git for obvious security reasons, and yet secrets have to reach the services automatically at deploy time.
There is no single tool that solves this. There are three, each covering one part of the problem.
Layer 1: Infisical (the vault)
Infisical is a self-hosted secrets manager. Think of it as a password vault for infrastructure. All the homelab secrets — database passwords, API tokens, OAuth credentials, SMTP passwords — live in Infisical as the single source of truth.
Infisical runs on the escipion cluster (Roma network) and is reachable at secrets.zetesis.localhost. Because it is self-hosted, secrets never leave the homelab's control.
Two ways out
For Docker Compose services — ptolomeo (the CD agent) fetches the secrets from Infisical at deploy time and injects them as environment variables:
# .doco-cd.yaml — ptolomeo lee esto
external_secrets:
DB_PASSWORD: <project-id>:prod:/my-app/DB_PASSWORD
SMTP_HOST: <project-id>:prod:/my-app/SMTP_HOST
When ptolomeo deploys a stack, it calls the Infisical API, fetches the values, writes them to .env and runs docker compose up. The container sees DB_PASSWORD=xyz123 as a normal environment variable.
For Kubernetes — the External Secrets Operator (ESO) runs inside the cluster and periodically syncs Infisical secrets as native Kubernetes Secrets:
ESO creates a Kubernetes Secret called web-secrets with the values from Infisical. Pods mount it as environment variables or files — without knowing where the secrets come from.
ESO refreshes every minute. If you rotate a password in Infisical, the Kubernetes Secret updates automatically. No redeploy needed.
The ClusterSecretStore pattern
In alejandro, secrets are organized with Kustomize components following the same base/overlay pattern as the infrastructure (from Post 3):
manifests/infisical/
base/
cluster-secret-store.yaml — Template with placeholders
prod/kustomization.yaml — Patches store name + environment slug
staging/kustomization.yaml
The base defines a ClusterSecretStore template. Each overlay patches it with the correct Infisical project and environment. That way, production reads from the Infisical prod environment and staging reads from staging — same structure, different credentials.
Layer 2: SOPS/age (the file lock)
There are files with secrets that have to exist in the Git repository. For example, Talos Linux needs the cluster certificates and bootstrap tokens to generate the machine configurations. These live in talsecret.sops.yaml next to talconfig.yaml.
SOPS encrypts the values of a YAML file while leaving the keys readable. Combined with age (an encryption tool), it produces files that can be committed safely:
# Lo que vive en Git (cifrado)
cluster:
id: ENC[AES256_GCM,data:QqUK9Xr6cWv0ed9F...,tag:tnMuAX...,type:str]
secret: ENC[AES256_GCM,data:ncoTvU2yA5He7n2r...,tag:o5SrlE...,type:str]
You can see the structure (cluster.id, cluster.secret) but not the values. Decrypting requires the age private key — which is stored in Infisical, never in Git.
The .sops.yaml file at the repo root tells SOPS which public key to use:
Infisical provides the age private key as an environment variable. SOPS uses it to decrypt the files. The decrypted output goes to clusterconfig/ (gitignored, never committed).
Everything else stays in Infisical and never touches Git.
Layer 3: the chicken-and-egg bootstrap
The puzzle: ESO needs credentials to connect to Infisical. But those credentials are a Kubernetes Secret. And ESO is precisely what creates Kubernetes Secrets from Infisical. So where do ESO's credentials come from?
The answer: Talos inline manifests. The inline-secrets.sops.yaml file contains SOPS-encrypted Kubernetes Secret manifests. When Talos brings up the control plane, it applies these manifests directly — before ArgoCD or ESO even start. That breaks the circular dependency:
Loading diagram...
These bootstrap secrets are the only ones managed via SOPS. From there on, everything is automatic through ESO.
How the three layers fit together
Loading diagram...
Infisical is the single source of truth. ptolomeo bridges Infisical and the Docker containers (environment variables). ESO bridges Infisical and the Kubernetes pods (native Secrets). And SOPS/age covers the special case of files that have to exist in Git.
Backup strategy
All homelab data is backed up to a single place: MinIO en spinoza (10.1.0.11). The backup tool is Restic — an encrypted, deduplicating backup program that supports S3.
Docker hosts: tolstoi
All VPSs and VMs run a service called tolstoi — a Restic wrapper that runs daily at 3:00 AM. It backs up Docker volumes to MinIO through the Tailscale gateway (cervantes):
Each deployment extends this base with its own volumes and S3 bucket:
Host
Bucket
What gets backed up
von-braun
von-braun-restic
Caddy data, CrowdSec DB
escohotado
escohotado-restic
PostgreSQL, Ghost, Typesense
unamuno
unamuno-restic
Forums, PostgreSQL
Kubernetes: K8up
On the Kubernetes clusters, K8up replaces tolstoi. It is a Kubernetes-native backup operator that discovers and backs up PVCs (Persistent Volume Claims) automatically. The schedule is defined as a Kubernetes resource:
apiVersion: k8up.io/v1
kind: Schedule
metadata:
name: backup-schedule
namespace: turing
spec:
backend:
s3:
endpoint: http://minio-atenas.data.svc.cluster.local:9000
bucket: escipion-restic
backup:
schedule: '30 2 * * *' # 2:30 AM a diario
prune:
schedule: '0 4 * * 0' # Domingos a las 4 AM
retention:
keepDaily: 7
keepWeekly: 4
keepMonthly: 6
check:
schedule: '0 5 * * 0' # Domingos a las 5 AM
Three automated operations: backup daily at 2:30, which snapshots all PVCs in the namespace; prune weekly on Sundays, which deletes old snapshots according to the retention policy; and check weekly after the prune, which verifies backup integrity.
K8up is configured per namespace. On escipion there are separate schedules for turing (Infisical), gauss (Harbor) and traefik. Each one backs up to its own prefix in the bucket.
Everything converges on spinoza
Loading diagram...
The Docker hosts reach MinIO through the cervantes Tailscale gateway. escipion and alejandro reach it over the Tailscale mesh (Roma to Atenas).
Home automation
The homelab is not just servers — it also controls the house. Two services run on the aristoteles VM on the Atenas network (marco-aurelio).
rothbard: Zigbee2MQTT
Zigbee2MQTT bridges home Zigbee devices (lights, sensors, switches) and MQTT, a lightweight messaging protocol. It runs alongside a Mosquitto MQTT broker:
The Zigbee coordinator (a USB dongle) talks to the devices directly — with no cloud service in between. Zigbee2MQTT translates device messages into MQTT topics, making them available to any automation system.
mises: Homebridge
Homebridge exposes non-HomeKit devices to Apple Home. It runs in host network mode to handle mDNS discovery:
Homebridge picks up the devices published to MQTT by Zigbee2MQTT and exposes them as HomeKit accessories. A "Hey Siri, turn off the living room lights" ends up triggering an MQTT message that goes from Homebridge to Zigbee2MQTT, from there to the Zigbee radio, and from the radio to the light bulb.
Both services are reachable through the local Caddy reverse proxy (trajano) at rothbard.zetesis.localhost and mises.zetesis.localhost.
Disaster recovery
If everything goes wrong, you need exactly six values stored outside the infrastructure to rebuild:
Secret
What for
age private key
Decrypt the SOPS files in the repo
Backup repository username
Access the Infisical backup in MinIO
Backup repository password
Access the Infisical backup in MinIO
Backup encryption password
Decrypt the Restic backup
Secrets manager encryption key
Boot Infisical from the backup
Secrets manager bootstrap secret
Boot Infisical from the backup
With these six values and a clone of the Git repository, the rebuild path is:
Decrypt the SOPS files with the age key
Boot the first Talos node with talhelper genconfig + talosctl apply-config
Bootstrap ArgoCD with kubectl apply -f bootstrap.yaml
Restore Infisical from the MinIO backup using the Restic and encryption keys
Wait — ESO connects to Infisical, secrets propagate, services start
Once Infisical is up, the rest is automatic. ESO syncs secrets, ArgoCD deploys applications, ptolomeo pulls on the Docker hosts. The full rebuild is documented in the FIRST_STEPS.md file in the repo.
Loss scenarios
What is lost
Impact
Recovery
Only the age key
SOPS files cannot be decrypted
Retrieve it from Infisical
Only Infisical
New services cannot be deployed
Restore from the backup in MinIO
age key + Infisical
Everything
Regenerate all secrets from scratch
Only the Git repo
The configuration is lost
Re-clone from GitHub
The essential part: the age key and the Infisical backup credentials have to exist somewhere outside the homelab — a password manager, a piece of paper in a safe, an offline USB drive. Without them, you start from zero.
GitGuardian: the safety net
Since all the infrastructure is in Git, there is one more layer: GitGuardian scans every push and every pull request for accidentally committed secrets:
If someone accidentally commits a plaintext password, GitGuardian catches it before it reaches the main branch.
What we have built
Let's zoom out. Over the course of this series we have covered:
The big picture — Hardware, naming, what runs where
Networking — Site-to-site WireGuard, Tailscale mesh, Cloudflare at the edge, Caddy reverse proxy
Kubernetes and GitOps — Talos Linux, ArgoCD App-of-Apps, ApplicationSets, ptolomeo
Security and operations — Three layers of secrets (Infisical, SOPS, ESO), backups with Restic, home automation
Everything runs from a single Git repository. Two CD systems (ArgoCD + ptolomeo) deploy to Kubernetes and Docker hosts respectively. Secrets flow from Infisical through three different mechanisms depending on the destination. Everything is backed up to MinIO on spinoza.
Is it overkill for a homelab? Probably. But every piece exists because I ran into a real problem — and the solution taught me something I now use professionally. That is the real value of a homelab: a playground where the stakes are low, but the lessons are real.
With secrets sorted and a written answer to "what if everything burns down?", what remains is watching the system run and knowing how to recover it. That is what the last two posts cover.
01
02
03
04
