Mid-June, while preparing an infrastructure review, I ask myself a dumb question: if worker-2 burned out tonight, what would I actually lose? I list my “backups” out loud, and the answer fits in one embarrassing sentence: I have no real backup. A RAID1, Longhorn snapshots stored on the cluster itself, a tar of the Home Assistant config sitting on the same array as what it is supposed to protect. Pure theatre.
Two weeks later, on July 2, the last gap closed: the 93 GB of Nextcloud were finally leaving, encrypted, towards an off-site copy. In between, a full redesign into 3-2-1-1-0, a token that leaked twice, a circular dependency in Longhorn, and an upload crawling at 1 Mbit/s. Here is the path, mistakes included.
The finding: “no real backup”
An honest audit starts by naming what is not a backup. In my setup, almost everything fell into that category:
- The NAS’s RAID1 (2× WD Red 6 TB, ~5.5 TB usable) provides availability: if one disk dies, the other takes over. But an unfortunate
rm -rf, ransomware or logical corruption is replicated instantly onto both disks. RAID protects against the hardware failure of one disk, not against me. - The cluster’s S3 backup lived on the cluster. Longhorn’s backup target pointed to storage hosted on worker-2 — the very machine whose volumes I wanted to be able to recover. Losing that node meant potentially losing the volumes and their backups in one go.
- My girlfriend’s 4 TB external drive — her photos, her documents — was a single copy, zero redundancy. One drop, one USB controller giving up, and it is a straight loss.
- The photos on both phones and the personal Nextcloud: irreplaceable, with no verified second copy.
What all of this has in common is the confusion between fault tolerance and backup. A real backup is a copy decoupled in time and in space from the original: I can go back to yesterday, and the copy survives the destruction of the source. Nothing I had ticked both boxes at once.
The 3-2-1-1-0 rule and tiering by value
The framework I settled on is an extension of the classic 3-2-1 rule:
3 copies · 2 different media · 1 off-site · 1 offline or immutable · 0 errors (proven by a restore test).
The trap, when you discover this rule, is wanting to apply it to everything. But I cannot afford to put 5.5 TB of media library in a cloud sized in single-digit terabytes — and above all I do not need to. The key is to classify data by value and not treat it all the same:
| Tier | Data | Strategy | Replaceable? |
|---|---|---|---|
| T0 — Irreplaceable | Personal photos/videos, documents, Nextcloud, the 4 TB data, password vaults | strict 3-2-1, versioned, encrypted, off-site | Never |
| T1 — Hard to recreate | Cluster app data (Gitea repositories, n8n database, Home Assistant config), PC profiles | automatic daily backup + off-site | Painful |
| T2 — Recreatable from code | Manifests, docker-compose, .env, IaC | Git (Gitea + GitHub mirror) | Rebuildable |
| T3 — Re-downloadable | Media library (films/series) | RAID1 is enough + back up only the metadata (arr/Jellyfin databases), not the blobs | Yes |
That table did most of the design work. It states in black and white that the media library does not need to go off-site — only its metadata, a few megabytes that make it possible to rebuild the libraries after a re-download. As a result, the genuinely precious scope fits in a few terabytes, not a few dozen.
The architecture: the NAS as convergence hub
Rather than backing up every machine to the cloud independently, everything converges on the NAS first, which becomes the single point to back up to the outside.
Two ideas structure the diagram:
- Workstations and phones push their data to Nextcloud (desktop and mobile clients). Phone photos land automatically on the NAS; the Documents/Desktop/Pictures folders of the PCs are synced there. A far from negligible bonus: Nextcloud versioning already allows recovering an overwritten file.
- The cluster does not store its own backups — it pushes them to the NAS. A script on master-1 (
[email protected]) performs the database dumps and an etcd snapshot every night, then sends them over SSH to/mnt/storage/backupson the NAS (10.0.0.30):
# cron sur master-1 — 45 2 * * *
pg_dump gitea > /tmp/gitea-$(date +%F).sql
pg_dump n8n > /tmp/n8n-$(date +%F).sql
k3s etcd-snapshot save # snapshot etcd frais
rsync -a /tmp/*.sql /var/lib/rancher/k3s/server/db/snapshots/ \
[email protected]:/mnt/storage/backups/db/
Nextcloud’s MariaDB database, for its part, is dumped locally on the NAS (30 2 * * *). Direct consequence: the cluster can be lost entirely without taking the databases or the etcd with it. That is exactly the separation the old setup did not have.
Once everything has converged on the NAS, it backs itself up to two targets with restic: a local disk (fast restore) and an off-site cloud (encrypted, deduplicated, incremental) through rclone. The off-site repository is end-to-end encrypted — the cloud provider only ever sees opaque blobs, never the content. That matters, because my home on the NAS also contains application secrets that are out of the question to push in the clear.
Longhorn breaking its own circular dependency
The nastiest point of the old setup deserved its own fix. Longhorn’s S3 backup target — where volume backups land — ran inside the cluster, on worker-2. In other words: the backup meant to protect me from losing a node shared its failure domain with that node. A fire, a dead SD card, a mistake of mine on worker-2, and I would lose the original and its copy in the same gesture.
The fix: move the target outside the cluster, onto the NAS. I deployed a MinIO in a Docker container on the NAS (data on the RAID1, bucket longhorn-backups), then re-sealed Longhorn’s S3 secret with kubeseal to point at the new endpoint:
# secret Longhorn re-scellé — endpoint hors-cluster
AWS_ENDPOINTS: http://10.0.0.30:9000
After the ArgoCD sync, the default BackupTarget goes back to available: true, and — the only test that counts — a real backup does show up in the NAS bucket, not in cluster storage. The old in-cluster backups stay around for a while as a safety net, but the circular dependency is broken. (If MinIO interests you in its own right, I laid the groundwork in MinIO on the cluster, and Longhorn storage is detailed in Longhorn on K3s.)
The secret seen in the clear: the near-miss
While setting up the off-site leg, an access token for the cloud provider appeared twice in the clear in configuration exchanges, before it had even been used. Nobody hostile saw it — but that is not the point.
The right reaction is not “I’ll be careful not to paste it again”. A secret that has been exposed in the clear, even once, must be considered compromised: it must be revoked and regenerated, full stop. The procedure applied:
- First finish the operation in progress, so as not to break a half-completed transfer.
- Revoke the third-party access on the provider side (remove the application’s authorisation).
- Generate a fresh credential and re-authorise the tool with it.
It sounds obvious written this way, and yet the temptation to “just not reuse it” is real when everything is working. The rule I set myself: the only question to ask about a secret seen in the clear is when to rotate it, not whether.
The 1 Mbit/s bottleneck
First real off-site backup of the small critical scope (dumps, configs, etcd — 432 MiB): flawless, a few seconds. Then comes the big one, the 93 GB of Nextcloud data. And there, throughput collapses to ~1 Mbit/s. At that rate, the initial seed was measured in days, a transfer that the slightest interruption would have restarted from far too early.
The diagnosis was not on the network side, nor on the NAS side. The rclone client was using, by default, shared application credentials — pooled across all users of the tool, and as such heavily rate-limited by the provider. The pipe was not clogged: I had simply been assigned the smallest tap.
The fix: create a dedicated OAuth application credential on the provider side, with a scope reduced to only the files created by the application. One trap along the way, worth knowing about: this “application files” scope is partitioned per application — a new credential does not see the files already uploaded by the old one, so the existing restic repository appeared “empty”. The scope had to be temporarily widened to full access for the new client to find the repository already in place. No data migration, just the right visibility setting.
The result, measured:
| Before (shared credentials) | After (dedicated credential) | |
|---|---|---|
| Throughput | ~1 Mbit/s | 90–150 Mbit/s |
| 91 GB seed | several days (projected) | 1 h 16 |
A factor of ~100 for a configuration change, without touching the hardware or the subscription. The 4,977 files of the seed (91.3 GiB) went through in an hour and a quarter, and thanks to restic’s deduplication, the following daily run had almost nothing to re-upload.
Immutability: the lesson still missing
Here is the blind spot I have not closed, and that it would be dishonest to present as solved. Today, my off-site backup is encrypted and decoupled — but the account that writes it can also erase it. Ransomware taking control of the NAS could, with the same credentials, delete the cloud copy right after encrypting the original.
And a backup that the compromised source can destroy is not really an offline backup. That is the whole meaning of the extra “1” in 3-2-1-1-0: an immutable, or offline, copy that survives the compromise of the source. Two concrete leads are on my list:
- An append-only target, or a token without delete rights, so that an attacker on the NAS can write but not erase.
- Eventually, a target with object locking (object-lock) — a few euros a month — for T0 only, guaranteeing that a version cannot be deleted before its expiry.
I document it here deliberately as acknowledged debt rather than hiding it. The strategy is operational; strict immutability is the next step.
Verify, or it is not a backup
A backup you do not monitor is a Schrödinger backup: you only know whether it works at the moment you desperately need it. Two disciplines close that loop.
The heartbeat. Every backup job writes a “last backup OK” timestamp. The existing monitoring (VictoriaMetrics + vmalert, described here) scrapes that metric, and a rule fires a Telegram alert if the last successful backup is more than 26 hours old. A missed backup thus becomes an event that notifies me, instead of a silence I discover three weeks later. The principle is the same as for the disk temperatures I monitor on the NAS: measure continuously, alert on a threshold.
Restore tests. This is the part everybody skips and yet the only one that proves the “0 errors”. The cadence I set myself:
- Monthly: restore a random file and a random service, run
restic check. - Quarterly: a more complete restore drill (one volume, one service end to end).
A backup never restored is only a hypothesis. The first monthly test has already served to validate that the break-glass chain held: the master keys that decrypt the secrets, the cluster token and the Nextcloud config live in a dedicated secrets vault (break-glass), whose root of trust — the paper emergency kit — is stored off-site. Without it, no recovery is possible, however many copies there are.
The numbers
Two weeks, from a brutal finding to a complete and verified chain. The cadences in place:
| Frequency | Action |
|---|---|
| Continuous | Nextcloud sync (PCs, phones) |
| 4×/day | Longhorn snapshots (cluster) |
| Daily (night) | cluster pg_dump → NAS · restic NAS → local disk · restic NAS → cloud · etcd snapshot → NAS |
| Weekly | restic check · media metadata export · Longhorn weekly backup |
| Monthly | restic prune · restore test (1 file + 1 service) |
| Quarterly | full restore drill |
And the number I like most: total cost ≈ €0. Everything reuses what was already there — Nextcloud already self-hosted, the family cloud plan already paid for, restic/rclone free, the NAS already there, the monitoring already in place. The only expenses considered are optional: a second external drive (€80) for a rotating physical copy, and object-lock storage (€5/month) on the day I harden T0 immutability.
The real shift was not technical. restic and rclone install in ten minutes. The click was to stop calling “backup” what was only availability — and to accept that the only backup that counts is the one you have already restored for real.