Context: how GitOps was set up on the cluster is covered in GitOps with ArgoCD, storage in Distributed block storage with Longhorn.
The constraint: switch everything off, lose nothing
In mid-September 2026, the homelab rack has to move. Not one node to reboot: all six. Three Raspberry Pi 4 masters (8 GB), two Pi 5 workers (16 GB), a small Pi 4 worker (2 GB), 15 Longhorn volumes replicated across them, and 32 ArgoCD Applications whose whole job is to make sure nothing moves. The NAS next to it holds MinIO and the backups; it is the first thing to come back on.
The scope is deliberately narrow: same machines, same router, same switch, DHCP reservations kept. IP addresses must not change. The problem is that a k3s cluster is not designed to be switched off entirely. etcd wants a quorum, Longhorn wants reachable replicas, ArgoCD wants the cluster to match git. Unplugging all of that cleanly takes an order. Bringing it back takes another one — and that is the one the runbook underestimated.
What follows: the runbook written on 17 September, its real execution on the 20th (a 60-minute restart, four surprises), and its re-run on the 26th (10 minutes).
The escape hatch: backups that were actually restored
Before touching a single replica, one rule: a backup that has never been restored is a hypothesis. On 17 September the escape hatch is prepared, then tested:
- a full forced Longhorn backup to MinIO on the NAS, triggered from the weekly CronJob;
- an etcd snapshot copied to the NAS;
- fresh database dumps, including Gitea’s PostgreSQL, with gzip integrity checked;
- a MAC ↔ IP ↔ node table, in case a DHCP reservation misbehaves;
- the GitHub mirror of the GitOps repository, checked at the same SHA as Gitea.
Then the tests: Gitea’s PostgreSQL dump restored into a temporary database — 39 repositories found — and a Longhorn volume restored from MinIO in 25 seconds. The runbook itself has a readable copy outside the cluster: during the outage Gitea is off, and a runbook you can only read in Gitea loses much of its point.
One detail matters more than it seems: the etcd snapshot is taken before any scale-down. It is the one that describes the “everything running” state. A snapshot taken afterwards would describe a cluster with every app at 0 replicas; restoring it would give you a perfectly healthy cluster that serves absolutely nothing.
The shutdown: disarm GitOps before contradicting it
The selfHeal trap
The naive reflex: kubectl scale --replicas=0 everywhere, then shutdown. On a GitOps cluster with selfHeal, that does not last three minutes. ArgoCD sees a Deployment declaring 1 replica in git and running 0 in the cluster, treats it as drift, and brings everything back. That is exactly its job.
Rather than editing 32 Applications, or worse, git, the controller is switched off:
K="sudo k3s kubectl"
$K -n argocd scale statefulset argocd-application-controller --replicas=0
No controller, no reconciliation. The repo-server and the UI stay up, but nobody compares anything anymore. For a while, the cluster is back to being an ordinary Kubernetes that does what kubectl says.
Record the CronJobs, then suspend them
Even before ArgoCD is switched off, the active CronJobs are written to a file and then suspended. A backup that fires in the middle of the shutdown is a volume reattaching at the worst possible moment. The list will be needed at restart — we will see why it is essential.
$K get cronjobs -A -o json | python3 -c "
import sys, json
for c in json.load(sys.stdin)['items']:
if not c['spec'].get('suspend'):
print(c['metadata']['namespace'] + '/' + c['metadata']['name'])" > cronjobs-actifs.txt
while IFS=/ read ns name; do
$K -n "$ns" patch cronjob "$name" -p '{"spec":{"suspend":true}}' </dev/null
done < cronjobs-actifs.txt
Apps, then databases, then the tunnel
The scale-down order follows dependencies in reverse:
- the application namespaces, then monitoring (Grafana, collectors, alerting);
- the applications backed by a database, Gitea included;
- a 45-second pause, so the apps can shut down cleanly;
- the databases: PostgreSQL, MySQL, the time-series store;
- the Cloudflare tunnel, last of all — as long as it runs, there is still a remote view of the cluster.
Databases last, because an app that loses its database mid-flight logs errors, retries, and sometimes leaves a broken local state. A database stopped when nobody is talking to it anymore is a non-event.
The gate: 15 volumes detached, not 14
No shutdown while a single Longhorn volume is attached. An attached volume means an engine that may be writing; powering off the node under it buys you a replica rebuild on return, in the best case.
watch -n5 "sudo k3s kubectl -n longhorn-system get volumes.longhorn.io \
-o custom-columns=N:.metadata.name,S:.status.state,R:.status.robustness | grep -v detached"
The list must empty out. If a volume stays attached, find the pod holding it and scale it to 0. The double quotes and the fully spelled-out command are not a whim: more on that below.
The nodes: workers first, primary master last
The shutdown commands are sent from the PC, over SSH directly to each node — not bounced through the first master, which has to go down last and would otherwise saw off the branch you are sitting on. The three workers first, then the masters, 20 seconds apart, the first master last. Wait for the disk LEDs to go dark before unplugging anything: the USB-SATA SSDs are the most fragile component in the rack.
The physical move is nothing glamorous: photos of the cabling, per-node labels on cables and power supplies, and a check of the power supply at the new spot. A Pi 5 wants 5 V / 5 A; undervoltage does not produce an error message, it produces silent corruption.
The restart: the reverse order, and what it did not anticipate
20 September, 15:05: the NAS is powered on. 15:09: the three masters are Ready and etcd has quorum.
sudo k3s kubectl get nodes
sudo k3s kubectl get --raw '/readyz?verbose' | grep etcd
# [+]etcd ok
# [+]etcd-readiness ok
Masters first, because without etcd quorum nothing else makes sense: the workers’ kubelets would be hitting an API that does not answer. The workers follow, Ready at 15:27. At 15:29 Longhorn reports 15 volumes out of 15 detached, none faulted. The hard part seems done. At this point the runbook has a single step left: scale the ArgoCD controller back to 1 and let git rebuild the cluster.
The final green would come at 16:05. In between, four surprises.
Surprise #1: the NAS does not restart its containers
Before the cluster, the NAS. Its Docker containers use restart: unless-stopped, and they had been stopped by hand with docker stop before the shutdown. unless-stopped means precisely “always restart, unless someone stopped me” — and Docker remembers that across a reboot. Result: a NAS that is up, and not a single container. MinIO included.
The fix: docker start the databases first, then everything else. 38 containers out of 38, by hand, in about forty seconds. Nothing serious, but worth knowing: a “just in case” docker stop is a decision that survives a reboot.
Surprise #2: ArgoCD needs Gitea, which only ArgoCD knows how to start
The controller comes back and reconciles nothing. Obvious in hindsight: ArgoCD reads its manifests from Gitea, and Gitea is at 0 replicas — ArgoCD was the one supposed to bring it back. A perfect circular dependency.
The way out is manual, and safe: bring up the database and then Gitea by hand, in the exact state git describes (1 replica each). When the controller takes over, it finds nothing to correct.
$K -n gitea scale deploy postgres --replicas=1 # wait for Ready (~20 s)
$K -n gitea scale deploy gitea --replicas=1 # wait for Ready (~30 s)
$K -n argocd scale statefulset argocd-application-controller --replicas=1
This is the most structural of the four lessons: on a cluster that hosts its own git, the cold-start path is not the nominal path. You need to know it before you need it.
Surprise #3: a ghost repo-server
Even with Gitea online, nothing syncs. The culprit: an argocd-repo-server pod stuck in Unknown since the shutdown. The ReplicaSet still believed it was alive and so did not create another one; without a repo-server, the controller cannot render any manifest, hence no comparison. Ten minutes of blockage before figuring it out.
$K get pods -A | grep Unknown
$K -n argocd delete pod <ghost-pod-name> --force --grace-period=0
Once the ghost is deleted, the ReplicaSet creates a real one and reconciliation starts. In the corrected runbook, this hunt for Unknown pods comes before bootstrapping Gitea: clean up the leftovers of the shutdown before relying on anything.
Surprise #4: selfHeal does not unsuspend CronJobs
The Applications turn green one by one. But no CronJob resumes. The original runbook claimed selfHeal would take care of it. Wrong.
The CronJob manifests in git do not declare the suspend field at all. ArgoCD compares what git declares; an absent field is not a declared value, so the suspend: true set by hand in the cluster produces no diff. As far as selfHeal is concerned, there is nothing to fix.
This is where cronjobs-actifs.txt earns its keep: re-enable exactly what was running, no more — a CronJob deliberately suspended before the outage stays suspended — and no less.
while IFS=/ read ns name; do
$K -n "$ns" patch cronjob "$name" -p '{"spec":{"suspend":false}}' </dev/null
done < cronjobs-actifs.txt
16:05: 32 Applications out of 32 Synced and Healthy, tunnel online, CronJobs re-enabled.
The bonus: three expired tokens hidden by the cache
Rescheduling every pod flushed out a problem entirely unrelated to the move: three image-registry pull secrets carried an expired token. Nobody had noticed, because the images were cached on the nodes. As long as a pod restarts where its image is already present, no pull happens and no token is checked. The three secrets were resealed with a valid token. Since then, any operation that reschedules pods is followed by an authentication test on every pull secret.
The actual timeline: 60 minutes, then 10
| Time (20.09) | Step |
|---|---|
| 15:05 | NAS powered on (containers started by hand) |
| 15:09 | 3 masters Ready, etcd ok |
| 15:27 | workers Ready |
| 15:29 | 15/15 volumes detached, 0 faulted |
| 16:05 | 32/32 Applications, tunnel, CronJobs re-enabled |
Zero data loss, same IP addresses, a one-hour restart against an estimate of 30 to 45 minutes. The surprises account for a good part of the gap — the ghost alone cost ten minutes.
On 26 September, round two: another full cluster shutdown, during which a worker’s disk was replaced (a different story, which will get its own post). This time the corrected runbook is applied as written. Full restart in 10 minutes.
The two shell bugs nobody had spotted
While fixing the runbook on the 26th, two shell bugs surfaced, the kind that survives every review.
The first one comes down to two quotes:
K=sudo k3s kubectl # K is "sudo" only for the "k3s kubectl" command
K="sudo k3s kubectl" # what was intended
Without quotes, the shell reads VAR=value command: an assignment that only applies to the environment of the command that follows. K only gets sudo, and k3s kubectl runs straight away. Every subsequent $K … then goes nowhere.
The second is nastier. The volume gate was written as watch -n5 '$K …'. watch runs its string in a new shell, where $K — a plain shell variable, not an exported environment variable — does not exist. So the command could not check anything. A gate that checks nothing is worse than no gate: you trust it. The fix: double quotes and the full command spelled out, as shown above.
The runbook, reusable version
What carries over to any GitOps cluster you have to switch off entirely:
- Restore, don’t just back up. A dump restored into a temporary database, a volume restored from object storage. And a copy of the runbook readable outside the cluster.
- etcd snapshot before any scale-down, to capture the “everything running” state.
- Record, then suspend, the active CronJobs. The list is the only source of truth at restart.
- ArgoCD controller to 0, otherwise selfHeal undoes the scale-downs in about three minutes.
- Apps, then databases, then the tunnel.
- Gate: every volume detached. A gate you have seen fail at least once.
- Workers, then masters, primary master last, from a machine outside the cluster.
- Restart: masters, etcd quorum (
/readyz?verbose), then workers. - Hunt down
Unknownpods before relying on reconciliation. - Bootstrap Gitea by hand (database, then app) before handing control back to the controller.
- Controller to 1, then unsuspend the recorded CronJobs.
- Outside the cluster, check restart policies:
unless-stoppedplus a manual stop is a service that does not come back.
The limits, honestly: all of this assumes unchanged IP addresses, guaranteed here by DHCP reservations. With a change of addressing, etcd would have been a very different story. And a runbook is only validated once executed: the one from 17 September had been reviewed, its backups tested, and it still contained two bugs and one wrong step.
The rack has moved. The cluster did not notice — give or take four details.