Context: the log pipeline (Vector → VictoriaLogs) and the alerting are described in Observability v2; the daily digest mentioned below, in A daily healthcheck on Telegram.
The symptom: a “10/10 OK” digest and 4.5 million error lines
On 20 July 2026 at noon, the homelab’s daily digest lands on Telegram. One line stands out: a backend produced 4,539,734 error lines in 24 hours, and a pod was killed by the OOM killer at 06:15. At the bottom of the message, the overall score still reads 10/10 OK. The healthcheck saw the problem, wrote it down, then concluded everything was fine.
The backend in question is a self-hosted aggregator of public feeds: a Python API (FastAPI) that continuously ingests dozens of real-time sources, including a third-party WebSocket feed relayed by a small Node.js subprocess. In normal operation it sits around 4.8 GiB of memory and writes roughly 3,000 log lines per hour. Nothing spectacular.
That evening, at 21:10, the storm is still going:
| Metric | Before 20 July | 20 July, 21:10 |
|---|---|---|
| Pod log lines | ~3,000 / h | 1.9 M over the 20:00–21:00 hour |
| Node conntrack entries | a few thousand | 94,083 / 131,072 (72 %) |
| Pod TCP sockets | a few hundred | 28,988 |
| Backend restarts | 0 | 2 (8 GiB OOM at 06:15, liveness at 17:03) |
Worse: at 04:46, the node hosting the pod (a Pi 5) lost contact with the Kubernetes API for about ten minutes. Timeouts to the control plane, lost HTTP/2 watches, failed lease renewal. At 04:47, the kubelet restarted six unrelated containers on that node, including a storage CSI plugin. An application bug had just spilled over onto an entire machine’s networking.
The investigation: tracing the storm back
The trigger: external and mundane
The very first errors date from midnight UTC sharp: certificate has expired. The WebSocket provider’s TLS certificate expired at 02:00 local time. Nobody on the homelab side controls that, and it is not a disaster in itself: a feed goes down, you retry later.
The problem is how you retry.
The root cause: every failure schedules two reconnects
In the Node.js subprocess, the connect() function opens a socket and attaches its handlers. On a CERT_HAS_EXPIRED error, a fallback path opens a second socket (validated by public-key pinning) with its own handlers. But the first socket keeps its own. When both close, each one calls setTimeout(connect, 5000).
One failure therefore produces two future connect() calls. On the next cycle, each of those two produces two more. The number of parallel loops doubles every 5 seconds, with no backoff and no jitter, bounded only by the machine’s resources: ephemeral ports (EADDRNOTAVAIL), the conntrack table, memory.
Three details turned a bug into a node-level incident:
- Double logging. Every error was logged twice, in the handler and in
connect(). The observed ratio confirms it: 7.9 METIMEDOUTfor 4.2 MReconnecting. - An unbounded stderr relay. On the Python side, every line of the subprocess’s stderr became a
logger.warning, hence a line in Vector, hence a line in VictoriaLogs. Total: 12.9 million lines in 24 hours. - A blind watchdog. The Python supervisor did have exponential backoff… which only applied if the Node.js process died. It never died: it spun internally. Respawn counter frozen, telemetry silent.
Around 05:00, the upstream went from “certificate expired” to “unreachable” (connect ETIMEDOUT). Server outage, or the homelab’s IP address blocklisted after a few million attempts: from the same public IP, there is no way to tell. The second hypothesis would not be unfair.
The mitigation: cut it off without touching the code
At 21:30, the priority is to stop the bleeding. The faulty feed only starts if its API key is present. An explicit env variable takes precedence over an envFrom pointing at a Secret: one line in the Deployment is enough to disable it cleanly.
# Temporary mitigation: an explicit env wins over envFrom.secretRef
- name: FEED_API_KEY
value: ""
Commit, push, ArgoCD syncs within three minutes. No kubectl edit: selfHeal would have reverted it. The effect is immediate:
| Metric | Before | After |
|---|---|---|
| Node conntrack entries | 94,083 | 4,464 (−98.5 %) |
| TCP sockets | 28,988 | 363 |
| Error lines of the new pod | 8.7 M / 24 h for the old one | 28 |
Rolling back the mitigation is a git revert. That is the whole point of going through Git even in an emergency.
The fix: three independent layers
The same evening, around 23:00, the real fix is deployed. It does not just plug the certificate hole: it assumes any feed will eventually misbehave, and that protection must hold even if one layer fails.
Chaque échec programme 2 reconnexions · retry fixe 5 s Every failure schedules 2 reconnects · fixed 5 s retry
12,9 M lignes de logs en 24 h, conntrack à 72 %, OOM du pod. 12.9 M log lines in 24 h, conntrack at 72 %, pod OOM.
Une seule reconnexion en attente · délai ×2, plafond 10 min, jitter 20 % At most one pending reconnect · delay ×2, 10 min cap, 20 % jitter
Reset seulement après 60 s de connexion stable. Reset only after 60 s of stable connection.
≥ 600 lignes stderr/min → sous-processus arrêté, respawn suspendu 15 min ≥ 600 stderr lines/min → subprocess stopped, respawn held for 15 min
Relais limité à 30 lignes/min, le reste résumé. État exposé sur /api/health. Relay capped at 30 lines/min, the rest summarised. State exposed on /api/health.
1. A single-flight scheduler with backoff
The reconnect logic moves out of connect() into a small dedicated class. Its core rule fits on one line: if a reconnect is already scheduled, any new request is ignored.
schedule(connectFn) {
if (this.timer !== null) return null; // single-flight
const expo = Math.min(this.baseDelayMs * 2 ** this.attempts, this.maxDelayMs);
const delay = Math.round(expo + expo * this.jitterRatio * this.random());
this.attempts += 1;
this.timer = this.setTimeoutFn(() => { this.timer = null; connectFn(); }, delay);
return delay;
}
No matter how many close or error handlers fire, they all land on the same timer. The delay starts at 5 s, doubles on each failure, caps at 10 minutes, with 20 % jitter so that several clients do not resynchronise on the same second.
The detail that matters: reset() is only called after 60 seconds of stable connection, not on every socket open. Otherwise an upstream that accepts the connection then drops it immediately resets the counter every cycle, and the backoff is useless.
The fallback socket also marks the old one as “superseded”, so its close handler can no longer schedule anything. Clock and randomness are injectable, which makes it possible to test the exact 20 July scenario (CERT_HAS_EXPIRED followed by a double close) and check that only one reconnect comes out.
2. Logs per transition, not per attempt
A repeated identical error becomes one summary line per minute. The “degraded TLS” banner is only written on a state change. Double logging is gone. A reconnect that fails for an hour should cost about sixty lines, not a million.
3. A circuit breaker on the symptom, not on process death
On the Python side, the stderr relay goes through a sliding window: 30 lines per minute relayed as-is, the rest summarised in one line (“N similar lines suppressed in the last 60s”). The same window provides a rate, and that rate is the signal: at 600 lines per minute or more, the subprocess is stopped and its restart held for 15 minutes. The breaker state (circuit_open, seconds remaining) is exposed on /api/health.
This is the most transferable lesson of the incident. The old safeguard watched the presumed cause of a problem (the process dies); the new one watches its observable symptom (the process is screaming). A live, runaway process slipped past the first. It no longer slips past the second.
Platform safeguards: what should have rung
Throughout the incident, vmalert fired no alert at all. Three rules were added that same evening:
ContainerOOMKilled;ContainerMemNearLimit: working set above 85 % of the limit for 15 minutes (a cgroup OOM is invisible to a node memory-pressure alert);ConntrackNearFull: conntrack table above 80 % for 5 minutes.
The second one flapped all through the first night on VictoriaMetrics, which deliberately lives between 81 and 89 % of its limit (its cache is sized on it). Excluding that pod plus keep_firing_for: 1h, and the noise stopped. A new alert is always calibrated against reality, not intuition.
On the log side, Vector now applies a throttle of 1,000 lines per minute per pod, about 15 times the chattiest legitimate pod. It is not an alert, it is a ceiling: VictoriaLogs can no longer be flooded by a single container. And the daily healthcheck learned to count: a recent OOM or 50,000 error lines in 24 h costs one point, 500,000 costs two. Replayed on the 20 July data, it returns CRITICAL 8/10 instead of “OK 10/10”.
The quiet failures: what you find while looking for something else
A week later, on 27 July, the same backend enters a restart loop. While examining it, three defects unrelated to the storm surface. None of them blocks the main event loop, but all of them saturate the small two-thread pool that runs slow tasks.
The SQLite lock cascade
Part of the feeds goes through 8 to 10 scheduled HTTP collectors, all writing to the same SQLite database. The database ran in the default journal mode (DELETE) with a 5-second busy_timeout. When the collectors finished their network calls together, they fought over the exclusive write lock: database is locked, tasks dragging for 50 to 120 seconds until the hard timeout, overlapping the next cycle.
The worst part was elsewhere: each collector opened its connection before its 10-to-60-second network call. The lock window therefore covered the entire network wait. The fix comes down to four moves:
_DB_CONNECT_TIMEOUT_S = 30
def init_db():
conn = _connect() # shared timeout, a single constant
try:
conn.execute("PRAGMA journal_mode=WAL") # readers no longer block the writer
conn.execute("PRAGMA synchronous=NORMAL") # no fsync on every commit
...
finally:
conn.close()
def ingest(self):
rows = self.fetch_data() # network first…
conn = _connect() # …lock second, only for the write
WAL mode lets readers work while a writer writes. synchronous=NORMAL, safe under WAL, avoids an fsync on every commit, which counts double when the file lives on a replicated Longhorn volume: every fsync crosses the network to the replicas.
The zombie process
A subprocess supervisor replaced its Popen after a crash without ever calling wait() on the old one. Result: a zombie in the process table until the server shut down, observed live in the pod. The fix: reap the exited process, with a bounded timeout, on every respawn and every crash detection.
Dead feeds retried every cycle
Two sources had been returning a permanent HTTP 404 for weeks, and the collector retried them on every pass. One joins the dead-URL list the code already handled, the other is commented out of the schedule, its class kept in case the source comes back. A permanent 404 is not a transient failure: it deserves a decision, not an eternal retry.
What about memory?
The pod was already running with an 8 GiB limit. It dated from 12 June: at 4 GiB, RAM saturated, the event loop slowed down, and the /api/health probes ended in timeouts, restarting the pod for nothing. Doubling the limit (and the request from 1 to 2 GiB) fixed that particular case.
In hindsight, it is the least satisfying fix of the three. It buys headroom, it fixes no cause. 20 July proved it: a loop that doubles every 5 seconds fills 8 GiB just as it would have filled 4. On a Pi 5 with 16 GiB shared with other workloads, memory is not a lever you can pull forever. The real fixes are the ones that bound growth: one reconnect at a time, a delay that grows, a circuit breaker.
Lessons learned
1. Every external feed will eventually misbehave
Expired certificate, permanent 404, a server that accepts then drops: the question is not if but when. Every feed client needs a single-flight scheduler, capped backoff with jitter, and a reset conditioned on a genuinely stable connection.
2. Watch symptoms, not just deaths
A supervisor that only reacts to process death misses the most dangerous case: the live process spinning in circles. Log rate, socket count or conntrack entries are cheap and reliable signals.
3. Logs are a shared resource
A chatty pod does not just fill its own volume: it loads the collection agent, the log storage and the node’s network. A per-pod ceiling at collection time (here 1,000 lines per minute) protects everyone, including from bugs nobody has written yet.
4. Take the lock as late as possible
For SQLite as for any exclusive resource: do the slow work first (network, computation), take the lock afterwards, only to write. And switch to WAL as soon as there is more than one potential writer.
5. An alert that does not ring is a bug
Zero alerts during 19 hours of incident, and a “10/10 OK” digest in the middle of it: that is the most expensive blind spot of the whole story. After an incident, the first question is not only “why did it happen?” but also “why did nobody see it?”.