What the tutorials do not teach
Kubernetes tutorials show you how to deploy an application that works. They rarely show what happens when it breaks — and how to reason your way to the cause.
This homelab has been, above all, a school of debugging and resilience. Here are the most important lessons.
1. Hardware constraints are a teaching advantage
On AWS, a pod that eats too much memory causes, at worst, a higher bill. On an 8 GB RPi shared between etcd, the kube-apiserver and a dozen pods, an OOM-killed pod triggers a cascade of errors.
That constraint forces a discipline the cloud lets you skip: actually understanding what each component consumes. After this project, resources.requests and resources.limits are no longer optional fields — they are the first things I configure.
2. GitOps is not a tool, it is a discipline
ArgoCD can be set up in 20 minutes. Understanding what “Git is the source of truth” means takes considerably longer.
The real value of GitOps is not automation — it is auditability. When something breaks, the Git history answers “what changed and when?”. That question is often the first one asked during a production incident.
I learned never to manually modify a resource managed by ArgoCD. Not because it is forbidden, but because it breaks the mental model: once the cluster’s reality diverges from Git, nothing is trustworthy any more.
3. Distributed storage is the hardest problem
Longhorn has been the source of more incidents than anything else in this homelab. Degraded volumes, replicas that will not rebuild, unexpected I/O performance — these problems have no simple fix.
The main lesson: always test recovery, not just resilience. It is not enough for Longhorn to promise 2 replicas. You have to simulate the loss of a node and verify that the volumes actually rebuild on another node within an acceptable delay.
4. Security has to be built in from the start
Sealed Secrets and Cloudflare Tunnel are two examples of security “by design” — not bolted on afterwards.
Trying to retrofit secret management after committing credentials in plaintext is an expensive operation (rotating every credential, auditing the Git history). Starting with kubeseal on day one costs 30 minutes of setup. The cost difference is asymmetric.
5. Operational complexity has a real cost
This homelab gave me a deep appreciation of operational complexity. Maintaining 6 nodes, 15 services, backups, monitoring and security outside working hours has consumed entire weekends more than once.
The real skill is not deploying those services — it is making trade-offs: when automation is worth the investment, when simplicity beats functionality, when accepting technical debt is reasonable.
6. What transfers directly to professional work
These 5 months exposed me to concrete challenges you find in professional cloud environments:
- Incident response: diagnosing and resolving outages under pressure, with imperfect logs and metrics
- Infrastructure as Code: version everything, do nothing by hand
- HA and resilience: design for failure, not just for the happy path
- Observability: metrics, logs, alerting — without them, you operate blind
- Security: secrets management, zero-trust networking, minimal RBAC
No certificate teaches these skills as effectively as solving them on real hardware, alone, at 11pm, with a cluster that has stopped responding.
What I would do differently
A load balancer for the control plane from the start: pointing KUBECONFIG at a single master is fragile. A keepalived VIP or a MetalLB L2 Advertisement in front of the 3 masters would be more robust.
Documenting in real time: the most instructive incidents were partly lost because I resolved them without writing down the root cause. This blog is in part a correction of that mistake.
Starting with monitoring: install VictoriaMetrics and Grafana before the applications. Debugging startup problems without metrics is needlessly hard. Metrics would have diagnosed several problems in minutes rather than hours.