Context: the vmalert → Alertmanager → Telegram pipeline is described in Observability v2, and the daily digest that complements real-time alerting in A Daily Health Digest That Stopped Crying Wolf. This post is about real-time alerting itself — and what it does when a single cause hits the whole cluster.
The symptom: 48 alerts for 24 pods
On 20 September 2026, the cluster moves into a rack. A planned operation: every node is shut down, moved, powered back on. The pods restart with them. Nothing unexpected.
On Telegram, it is a different story. The next day, the alert channel looks like a news feed. Once counted, the finding is simple and slightly embarrassing: 48 alerts for 24 pods. Every restarted pod triggered two different rules. And behind the alerts came a second wave: the messages from the script that launches an automatic investigation on every new alert. On 21 September, its budget of 8 investigations per day is burnt in 20 minutes, all on the same rule, PodProbeFailingSustained, all for the same cause — a reboot I had planned myself.
The most uncomfortable part comes from reading the script’s state: the daily cap had been hit every single day since 9 September. The migration did not create the problem; it made it impossible to ignore.
Digging in, three independent causes stack up, each one multiplying the previous.
The context: two rules for “slow” restarts
The cluster already has a PodCrashLooping alert for outright crash loops. Two finer-grained rules complement it, added for what slips under its radar.
PodRestartingSlowly catches the slow-motion crash loop: a container that restarts three times in 24 hours, spaced out enough never to cross a 15-minute rate threshold.
PodProbeFailingSustained targets a nastier case: a container repeatedly killed by its liveness probe, showing Running and Ready=True between each kill. The kubelet in this cluster does not expose prober_probe_total, so the rule uses a proxy: kube_pod_container_status_last_terminated_reason, the reason for the container’s last termination, combined with the same restart counter.
A quick reminder about this kube-state-metrics metric: it is a state, not an event. It is 1 for the reason of the last termination and stays there until the container terminates again. The daily digest had already tripped over it with a “phantom OOM”; here the trap is different — it is about which reason you accept.
Here are both expressions as they stood before 21 September:
# PodRestartingSlowly — before
increase(kube_pod_container_status_restarts_total[24h]) >= 3
# PodProbeFailingSustained — before
(kube_pod_container_status_last_terminated_reason{reason!="OOMKilled",reason!="Completed"} == 1)
and on(namespace, pod, container)
(increase(kube_pod_container_status_restarts_total[24h]) >= 3)
Putting one under the other is enough to spot the first problem.
Cause #1 — The duplicate: one rule contained in the other
PodProbeFailingSustained requires exactly the same condition as PodRestartingSlowly (at least 3 restarts over 24 hours), plus a filter on the reason. It is not a neighbouring rule: it is a strict subset. Every pod that triggers it also triggers the other one. Each “probe” incident therefore produces two alerts, two notifications, and two lines to read for a single piece of information.
As long as restarts stayed rare, the duplicate looked like a coincidence. With 24 pods restarted at once, it becomes a very visible factor of 2.
The fix is a single operator. The two rules become mutually exclusive: anything that terminated with reason="Error" goes to PodProbeFailingSustained, everything else stays with PodRestartingSlowly.
- alert: PodRestartingSlowly
expr: |
increase(kube_pod_container_status_restarts_total[24h]) >= 3
unless on(namespace, pod, container)
(kube_pod_container_status_last_terminated_reason{reason="Error"} == 1)
for: 1m
labels: { severity: warning }
unless is PromQL’s set operator: it keeps the left-hand series that have no match on the right-hand side for the given labels. With on(namespace, pod, container), matching happens container by container.
A rewritten rule that “should” work proves nothing. The measurement was taken against vmsingle, with both expressions evaluated on real data:
| Series (both rules) | Total | |
|---|---|---|
| Before | 24 + 24 | 48 |
| After | 11 + 13 | 24 |
Exactly the 24 affected pods, each counted only once. That is the criterion that matters: not “fewer alerts”, but one alert per container actually affected.
Cause #2 — reason="Unknown": a reboot is not a probe kill
The duplicate explains the factor of 2. It does not explain why a rule meant to detect liveness-probe kills lit up on pods that had no probe problem at all.
The PodProbeFailingSustained filter was negative: any reason except OOMKilled and Completed. Yet the comment above the rule stated clearly what it was looking for: liveness kills, exit 137, reason="Error". Between the two, one reason slipped through the net: Unknown.
| Reason | Typical exit code | What it means |
|---|---|---|
Error | 137 | Container killed by SIGKILL, typically by the kubelet after failed liveness probes |
Error | 1 | Application crash |
OOMKilled | 137 | Killed by the OOM killer |
Completed | 0 | Normal exit |
Unknown | 255 | The kubelet lost track of the container — because the node went down |
Unknown is the signature of a node vanishing from under the container: a planned reboot, a power cut, a rack move. Combined with the restarts caused by that same reboot, the negative filter armed the rule on every pod of the node. A maintenance window therefore showed up as an epidemic of failing probes.
The numbers in the database leave no room for doubt: 87 Unknown series versus 18 Error series. The rule spent most of its time watching reboots.
The fix flips the logic: instead of excluding what you don’t want, name what you do want.
- alert: PodProbeFailingSustained
expr: |
(kube_pod_container_status_last_terminated_reason{reason="Error"} == 1)
and on(namespace, pod, container)
(increase(kube_pod_container_status_restarts_total[24h]) >= 3)
for: 5m
This filter covers exit 137 (liveness kill) and exit 1 (application crash), which stays consistent with the intent: a container that dies on its own or gets killed, repeatedly. A pod restarted three times because its node went down is not silenced either: it falls back into PodRestartingSlowly, the generic rule, as a warning. That is deliberate — three restarts in 24 hours is still information, just not that information.
Cause #3 — The amplification: one investigation per pod
The first two causes produce 48 alerts. The third turns them into an avalanche.
The cluster is watched by a cron-driven script that launches an automatic investigation on every new alert: it gathers context, writes a report and sends a summary to Telegram. An investigation can take up to 15 minutes, so they are rationed, with a global cap of 8 per day and a 60-minute debounce per alert.
The problem fits in one line, the one that computes an alert’s fingerprint:
fp="$(jq -r '[.alertname, .namespace // "-", .pod // .instance // "-"] | join("_")' <<<"$alert")"
The fingerprint includes the pod. A common cause hitting 24 pods therefore produces 24 distinct fingerprints, 24 “new” alerts as far as the debounce is concerned, 24 candidates for an investigation. The cap of 8 per day protects nothing: it limits the damage, but it limits it to 8 near-identical investigations on the same cause, which eat the whole day’s budget before a genuinely different alert shows up. That is exactly what happened on 21 September: 8 out of 8 in 20 minutes, all on PodProbeFailingSustained.
And a cap hit every day since 9 September is no longer a safeguard. It is a symptom.
Bonus: the script’s log. Every alert skipped by the cap produced its own line, on every cycle. Result: 2.2 MB of log, including 12,715 lines for 20 September alone.
unless 24 + 24 → 11 + 13 séries24 + 24 → 11 + 13 series reason="Error" un reboot ≠ un kill de probea reboot ≠ a probe kill group_by alertname + severityalertname + severity 2 / alertname debounce 60 min → 6 hdebounce 60 min → 6 h Four changes in the script:
- A per-alertname cap: at most 2 investigations per rule per day, on top of the global cap. A single cause can no longer monopolise the budget.
- Debounce raised from 60 minutes to 6 hours: an alert that lasts a day is not a new alert every hour.
- The number of targets announced: before investigating, the script counts how many alerts in the cycle share the same alertname, and the Telegram message says so up front — “WARNING: 24 targets hit by PodRestartingSlowly — probably a common cause, this is only a sample.” Whoever reads it knows immediately they are looking at one cause, not twenty-four incidents.
- A bounded log: skipped alerts are counted and summarised in a single line per cycle, and the log truncates itself beyond a threshold.
The core of the new safeguard:
MAX_PER_ALERT_PER_DAY="${MAX_PER_ALERT_PER_DAY:-2}"
NCOUNT_FILE="$STATE/count-${TODAY}-${nsafe}"
NCOUNT="$(cat "$NCOUNT_FILE" 2>/dev/null || echo 0)"
if [ "$NCOUNT" -ge "$MAX_PER_ALERT_PER_DAY" ]; then
SKIP_ALERT_CAP=$(( SKIP_ALERT_CAP + 1 )); continue
fi
The order of the checks matters: debounce first, then the per-alert cap, then the global cap. An alert investigated recently costs nothing; a rule that has already had its two investigations does not eat into the others’ budget.
The routing: group by alert, not by machine
That leaves Alertmanager. Two settings were amplifying the noise as well.
The first: group_by: ['alertname', 'instance']. Grouping by instance means an event hitting N machines produces N messages — seven nearly full hosts, seven separate DiskAlmostFull notifications, where a single message listing all seven is enough. The instance label leaves the grouping, replaced by severity.
The second: a single 4-hour repeat_interval. But the restart rules work on a 24-hour window: once armed, they mechanically take up to 24 hours to clear. With a reminder every 4 hours, the same warning rang six times a day. Reminders are now set per severity:
route:
receiver: telegram
group_by: ['alertname', 'severity']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
routes:
- matchers: [ 'severity="critical"' ]
receiver: telegram
repeat_interval: 4h
- matchers: [ 'severity="warning"' ]
receiver: telegram
repeat_interval: 24h
A critical keeps a tight reminder, every 4 hours. A warning rings once a day. If a warning deserves more attention than that, it probably ought to be critical.
The verification: 48 simulated alerts, 4 messages
Before pushing, three checks:
- Valid YAML: 8 groups, 31 rules.
- Bash syntax of the script checked.
- A test harness that injects 48 simulated alerts — the 20 September scenario — and counts what comes out on Telegram: 4 messages, instead of the 8 that went out every day until then.
In fairness: this is a harness, not a real reboot. Only an actual planned maintenance will confirm the result at full scale. And the fix does not make a reboot silent: pods that restart three times in 24 hours are still reported, as a warning, in one grouped message, once a day. That is the intended behaviour — knowing that something restarted, without reading it forty-eight times.
Lessons learned
1. Overlapping rules must be made mutually exclusive
When one rule is a refinement of another, one of them must exclude the other. unless on(...) does it cleanly in PromQL. And the proof is not re-reading the expression: it is a query against the database showing that each target appears only once (24 + 24 before, 11 + 13 after).
2. On an enumeration of reasons, filter positively
reason!="OOMKilled",reason!="Completed" looks precise, but it is a denylist: any forgotten reason gets through. reason="Error" is an allowlist that says exactly what the rule is looking for. If a rule’s comment describes a positive signal (“exit 137, reason=Error”) and the expression filters negatively, the gap between the two is a bug waiting for its moment.
3. A planned reboot is a free alerting test
A maintenance that restarts every node is the best test scenario for alerting: a single, known, dated cause. What the channel receives that day gives you the pipeline’s amplification factor directly. Here it was 2 at the rule level, and worse downstream.
4. Alert-triggered automation must reason per cause
Anything that reacts to alerts — investigation, ticket, remediation — inherits their granularity. If the fingerprint includes the pod, one common cause becomes N actions. Capping per alertname, and announcing the number of targets, gives the human the right reading back: one incident, N symptoms.
5. A cap that saturates every day is an alert in itself
The budget of 8 investigations had been exhausted daily for twelve days. A safeguard that triggers every day no longer protects anything; it hides. Watching how often a cap saturates matters as much as the cap itself.
6. Cron script logs must be bounded
One line per skipped alert, on every cycle, adds up to 12,715 lines on a busy day. A one-line summary per cycle and self-truncation cost about ten lines of bash, and spare you discovering a multi-megabyte log on the day you actually need it.
The Telegram channel is back to what it should be: a place where every message deserves to be read. Which, after two days of news feed, takes a little getting used to.