The alerting pipeline set up in Observability v2 answers one question well: is something breaking, right now? vmalert evaluates its rules every 30 seconds, Alertmanager routes to Telegram, and a crash-looping pod knows it before I do. But it answers none of these:
- Was everything fine today, or did nothing simply cross a threshold?
- That pod which restarted once yesterday, that RAM spike that only lasted three minutes — noise, or a weak signal building up?
- If the alerting pipeline itself is broken (see the “pending is not firing” lesson in the previous article), who notices?
A daily digest is not a new source of alerts — it is a second pair of eyes that re-reads the day and correlates what no individual rule crosses on its own.
Where to run it: the access constraint
The first obstacle was not a technical one: the agent that orchestrates this kind of automation runs from a remote sandbox environment, which can only reach the cluster’s public endpoints — in practice, only the publicly exposed node metrics, nothing on the logs, history or internal API side. Running a script there that needs full cluster access is impossible.
The simplest solution was already in place: the NAS gbnas, which already hosts the backup stack and the CI runner, has direct network access to everything — cluster included, over the LAN. So the healthcheck runs there, not in the cluster itself: one more bash script in ~/healthcheck/, next to the NAS’s other automations.
Reuse rather than duplicate
The script reinvents no alerting rule. For consistency, it queries the same metric series as the existing vmalert pipeline:
vmq 'ALERTS{alertstate="firing"}'
On top of that, a PromQL layer specific to the digest: CPU/RAM/disk per node, container restarts over 24h, recent OOMKilled events, network spikes. A LogsQL correlation on VictoriaLogs surfaces the pods that are noisiest in errors along with a sample of their lines. And a local look at the NAS’s Docker side: stopped containers, disks past 85%.
Two outputs: a concise Telegram digest, sent through the bot already in place for alerting — not a new channel, just one more message in the same chat — and a full Markdown report archived locally, one file per day.
Least privilege: the dedicated RBAC
The script runs with no direct human supervision, on a cron, from a machine outside the cluster. Giving it the admin kubeconfig would have been the path of least resistance and the worst idea. Instead: a dedicated, read-only ServiceAccount, with a ClusterRole scoped to exactly what the script needs — no more:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: healthcheck-readonly
rules:
- apiGroups: [""]
resources: ["nodes","pods","services","namespaces","events","persistentvolumeclaims","persistentvolumes"]
verbs: ["get","list","watch"]
- apiGroups: [""]
resources: ["pods/portforward"]
verbs: ["create"]
- apiGroups: ["longhorn.io"]
resources: ["volumes","nodes","replicas","engines","backups"]
verbs: ["get","list","watch"]
- apiGroups: ["argoproj.io"]
resources: ["applications"]
verbs: ["get","list","watch"]
Two details are worth highlighting. First, pods/portforward with create but nothing in delete/update anywhere — just enough to open a tunnel to VictoriaMetrics and VictoriaLogs for the duration of a query, nothing that could modify anything. Second, access to the Telegram bot secret is not in the global ClusterRole: it lives in a separate namespaced Role, scoped by resourceNames to that one specific secret:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: healthcheck-telegram-secret
namespace: monitoring
rules:
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["alertmanager-telegram"]
verbs: ["get"]
A compromise of that token therefore does not give access to all the namespace’s secrets, just to that one. The ServiceAccount token itself is long-lived, deliberately: a cron with no human interaction cannot handle an OAuth refresh. The trade-off — a token that does not rotate — is documented, not hidden.
The hosting constraint that changes the implementation
An infrastructure detail with a real design consequence: on the NAS, the automation user does not have general sudo — only one precise hardware command (tied to SMART monitoring) is allowed under NOPASSWD. No sudo, no system systemd unit possible. Scheduling therefore goes through a classic user crontab, not a .service/.timer:
0 12 * * * ~/healthcheck/healthcheck.sh >> ~/healthcheck/cron.log 2>&1
Nothing exotic, but it is the kind of constraint you only discover by trying to install the “clean” version and hitting the wall of the missing sudo.
The false positives a “working” script was hiding
The script ran from day one without crashing. It took several runs to see that it “worked” in the wrong sense of the word: it produced a digest every day, but part of its content was noise dressed up as signal.
24h+ 24h window restarts récentsrecent restarts ≥5min≥5min threshold creationTimestamp warningscapped if warnings score ≤ 9 Phantom OOM. kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} is a state, not an event — it stays at 1 for as long as the pod lives, even if the OOM was three weeks ago and the pod has been running perfectly since. First digest: one pod flagged OOMKilled. Second digest, identical. The fix is an and on actual recent restarts:
sum by(pod)(
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}
and on(namespace,pod,container)
(increase(kube_pod_container_status_restarts_total[24h]) > 0)
)
Without the second term, that pod would have remained an “incident” indefinitely — until its next node failover, months later.
Pending has no age. A pod goes through the Pending state every time it is scheduled — normally for a few seconds. A CronJob firing every five minutes therefore generated one “incident” per run, purely because the script scanned the state at the wrong instant. Fix: only report a Pending if it is more than 5 minutes old, computed from creationTimestamp, not just from the current state.
The score that contradicted the text. The worst of the three: a digest could announce a score of 10/10 with, two lines below, the list of active warnings. The score and the status were computed independently — one ignored the other. A reader in a hurry who only looks at the number misses the problem. The fix, two guard lines after computing the raw score:
# Cohérence score/statut : jamais 10/10 avec des warnings, ni >8 avec un critique
(( WARN_N > 0 && SCORE > 9 )) && SCORE=9
(( CRIT_N > 0 && SCORE > 8 )) && SCORE=8
Why this is the most interesting part
Monitoring that alerts correctly on fake problems is worse than no monitoring at all: it trains the human to ignore notifications, exactly the reflex you cannot afford on the day the incident is real. None of the three bugs above is spectacular taken alone — but added together, they would have produced a digest that “cries wolf” almost every day, until it became background noise you scroll past without reading.
The validation that followed was nothing sophisticated either: replaying the script against the three precise scenarios (pod with an old OOM but stable, freshly scheduled pod, situation with active warnings) before considering the digest reliable. No formal test suite — just making sure each of the three identified traps could no longer happen before letting it run unsupervised.
Result
The digest fits in a single Telegram message (automatically split beyond ~3800 characters, just in case): overall status, score, incidents, spikes, log correlation, a glance at the NAS. The full Markdown report goes out in parallel to the local archives, one file per day, with a link to the previous day’s report.
Since the three false positives were fixed, the cron runs every midday, silent when everything is fine — and on the day it isn’t, it means there really is something to read.