❯ monitoring · Running

Relire ses propres tableaux Grafana : cinq angles morts qu'aucune alerte ne voyait Re-reading Your Own Grafana Dashboards: Five Blind Spots No Alert Could See

Une nuit à relire chaque tableau, panneau par panneau, a fait remonter une base de métriques coincée à côté d'etcd, un tmpfs rempli de bibliothèques orphelines, un binaire amd64 dans une image arm64 et des certificats K3s qu'un redémarrage complet du cluster n'avait pas renouvelés. Aucune alerte n'avait sonné, parce que rien ne les mesurait. One night spent reading every dashboard panel by panel turned up a metrics database squeezed next to etcd, a tmpfs full of orphaned libraries, an amd64 binary inside an arm64 image and K3s certificates that a full cluster restart had not renewed. None of it had triggered an alert, because nothing was measuring it.

· 10 min de lecture 10 min read ·
monitoringgrafanavictoriametricsvmalertk3slonghorn
🇫🇷 Lire cet article en français

Context: the monitoring stack (VictoriaMetrics, vmalert, Alertmanager, Telegram) is described in Observability v2, and the last time alerting made the news it was for saying far too much. This post is about the opposite: what it did not say at all.

The starting point: silent alerts prove nothing

On 10 October 2026, the alert channel is quiet. So everything is fine.

Except that a quiet alert channel says only one thing: none of the rules that were written is true. It says nothing about what no rule looks at. So I did the exercise I had been putting off for months: open every Grafana dashboard and read every panel, asking the same question each time: “is this number normal, and who would tell me if it stopped being normal?”

The night produced five findings. None of them was an outage. All of them were outages in the making.

Blind spot #1 — The metrics database did not monitor itself

First panel that stood out: the memory of the VictoriaMetrics pod. Between 80 and 92% of its 512 Mi limit, continuously. Nothing abnormal in itself, its caches are sized on the limit. What was more worrying was the node: a master, next to etcd, with about 24 GB of Longhorn writes per day. Had that pod grown, the cluster’s quorum was the one at risk.

While trying to put an alert on it, a second surprise: there was no vm_* series at all. VictoriaMetrics was collecting the whole cluster except itself. A database that saturates, rejects writes or stops ingesting was invisible. That is the nastiest kind of failure: dashboards freeze on their last value, and the absence of alerts comes from the absence of data.

Along the way, three more gaps of the same kind:

  • no kubelet_volume_stats_* series: PVC usage was not measured anywhere;
  • node-exporter ran with a 200m CPU limit and was throttled in 37% of CFS periods, for an average usage of about 0.004 cores. The bursts happen at scrape time (0.5 s per scrape on the Pi 4s): the limit was clipping precisely the measurements;
  • the “Kubernetes / Views / Pods” dashboard was empty: its datasource variable pointed at another project’s database.

The fixes are mundane, and that is exactly the problem: they could have been made on day one.

# vmsingle: some headroom, preferably on the Pi 5 workers, away from etcd
resources:
  requests: { cpu: 100m, memory: 512Mi }
  limits:   { cpu: 1000m, memory: 1Gi }
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      preference: { ... }   # the two Pi 5 workers

# vmagent: the kubelet, but only what measures volumes
- job_name: kubelet
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: "kubelet_volume_stats_(capacity|available|used)_bytes|kubelet_volume_stats_inodes(_free)?"
    action: keep

A victoriametrics job scraping vmsingle and vmagent, node-exporter without a CPU limit, and a capacite vmalert group complete the picture: VMSingleIngestionStopped (fewer than one row inserted per second for 10 minutes, critical), PVCAlmostFull below 15% free, NodeSwapHigh, TmpfsFilling, LonghornBackupError.

Blind spot #2 — 994 ghost libraries in a tmpfs

The new memory rules did not have to wait long. On the NAS, swap had been full for five days. Nobody had noticed: the NAS answered, the containers ran, just more slowly.

The culprit was not a process but a directory. On that machine, /tmp is an 8 GB tmpfs, in other words RAM. It held 994 files named .<hash>-00000000.so, 4.3 MB each, created between 29 September and 9 October, 4.15 GiB in total. None of them was open by any process.

Tracing the chain back took longer than the cleanup:

  1. The files are copies of libopentui.so, a library that opencode (a Bun binary launched from the npx cache) extracts to /tmp at startup and deletes on exit… when it exits cleanly.
  2. vibe-kanban, a systemd user service, starts opencode serve every time it starts, then kills it with SIGKILL. No chance to delete anything.
  3. vibe-kanban was restarting up to 300 times a day. Without linger, a user’s systemd manager stops when their last session closes. And a scheduled task on the PC (an ssh and scp sync every two minutes) kept opening and closing sessions. Every close stopped the user services, every open restarted them, and every restart left another 4.3 MB in RAM.

The fix takes three steps: delete the files, loginctl enable-linger (allowed without sudo by polkit), and an hourly cron job that deletes any older than one hour, as a safety net.

The annoying detail: TmpfsFilling would not have fired anyway. Usage peaked at 64%, and the rule triggers below 35% free. It was NodeSwapHigh that spoke up. An alert on a specific symptom easily misses its target; an alert on the final resource, here memory, eventually catches it.

Blind spot #3 — A SMART exporter that read no disk at all

The SMART dashboard for the six Raspberry Pis showed empty panels. Not zero values: nothing. I had always read that as “nothing to report”.

The exporter logs, however, were explicit: every call to smartctl ended in exec format error. The pinned arm64 image (tag master) shipped a smartctl binary built for amd64. The container started, the exporter answered on /metrics, the probe was green, and every read failed. Result: 0 disks monitored across 6 nodes, with a Running pod.

Switching to the upstream v0.15.0 image (built on Alpine arm64, smartctl 7.5 aarch64) was not enough. Automatic disk scanning caused two problems: it labelled the USB-attached root SSD as a scsi disk without ATA attributes, and it picked up Longhorn volumes, which the node sees as iSCSI disks. The fix is to name the disk explicitly:

args:
  - --smartctl.device=/dev/sda;sat   # USB root SSD, behind a SAT bridge

Five SSDs now report temperature, health and power-on hours. The sixth node boots from a USB stick, which simply has no SMART. At least it is now a known absence.

While at it, I enabled PSI on the six Pis (psi=1 on the kernel command line; the Raspberry Pi kernel ships with it disabled). node-exporter now exposes CPU, memory and I/O stall times, far more telling than a utilisation percentage.

Blind spot #4 — Certificates a full restart had not renewed

The most serious finding came from a visual detail. On the “Benchmarks” dashboard, the orange “K8s warnings” annotations formed a regular row. I had taken them for background noise.

It was the CertificateExpirationWarning event, emitted every 30 minutes by each master: 585 occurrences in four days. k3s certificate check confirmed it, in WARNING: client-auth-proxy, client-k3s-cloud-controller, etcd-client and friends, all expiring on 2 February 2027.

Just enough context. K3s issues its certificates for 365 days. It renews them on its own, under two conditions: at service startup, and only if expiry is less than 120 days away. With the cluster created in early February 2026, the window opened on 6 October.

But the whole rack had restarted in late September, when it was moved. A few days too early: at that point no certificate was inside its window, and nothing was renewed. After that, no restart was planned before… expiry, which takes the cluster API down.

The rotation itself is uneventful, provided you are patient:

  1. etcd snapshot, as a safety net;
  2. systemctl restart k3s on each master, one at a time, waiting until all three are Ready and etcd is healthy before the next one;
  3. systemctl restart k3s-agent on each worker, waiting for the 17 Longhorn volumes to be healthy again in between.

Six nodes in five minutes. Client certificates are now valid until 10 October 2027.

But re-reading k3s certificate check after the rotation, column by column, some lines had not moved. The serving certificates (apiserver, etcd server and peer) have their own issue date. They were not inside their window on 10 October, so K3s left them as they were. One master expires on 18 June 2027, the other two in September. The certificate served by the API itself is a separate case again: dynamic, shared by the three masters, it renews itself without a restart, and k3s certificate check does not even list it.

So a restart does not reset every counter. It resets some of them, depending on their date. The only lasting defence is to measure.

The measurement: a script, a text file, a few rules

node-exporter can read metrics from text files (the textfile collector). A script, run hourly by cron on all six nodes (and at boot), reads the expiry date of each K3s certificate with openssl x509 -enddate and writes it in Prometheus format:

k3s_certificate_not_after_seconds{source="file",name="server/tls/client-admin.crt"} 1823189700
k3s_certificate_metrics_last_run_seconds 1791667020

Then a certificats vmalert group (simplified excerpt):

- alert: K3sCertificateRenewalWindow     # a restart is enough
  expr: (min by (instance) (k3s_certificate_not_after_seconds{source="file"}) - time()) < 120 * 86400
- alert: K3sCertificateExpiringSoon      # critical, replaces the previous one
  expr: ... < 21 * 86400
- alert: K3sCertificateMetricsStale      # the script no longer runs
  expr: time() - k3s_certificate_metrics_last_run_seconds > 26 * 3600

The first version had three flaws, found by re-reading the fix with fresh eyes an hour later:

  • A single line without a value invalidates the whole file for node-exporter. One unreadable date would therefore have made all of the node’s measurements disappear, silently. The script now skips the offending line.
  • A heartbeat without a measurement: if openssl broke, the script still wrote a fresh last_run with no certificate at all, and MetricsStale stayed silent. Now, if no certificate is read, the old file is kept: the measurement ages and the alert eventually fires. A MetricsMissing rule also covers a live node with an incomplete measurement.
  • The API’s dynamic certificate, which renews itself, would have triggered RenewalWindow in spring 2027 with a wrong instruction (“restart”). It is excluded from that rule and has its own: alert only if it drops below 60 days without having renewed.

Finally, the window warning is routed once a week: about 17 reminders between February and June 2027, instead of 114 with the default repeat interval. No point recreating the spam we had just escaped.

Blind spot #5 — Snapshots holding 19 GiB of dead data

Last panel, on the Longhorn side: the actual size of the VictoriaMetrics and VictoriaLogs volumes was far larger than their content. Both were part of the recurring daily-snapshot job, which makes little sense for a time-series database that rewrites its blocks constantly: each snapshot freezes data that will be compacted the next day.

Removing them from the job was not enough, and the purge was instructive:

Step (10 October)vmsinglevictorialogs
Before23.3 GiB4.35 GiB
14 daily-snapshot snapshots deleted21.7 GiB3.7 GiB
That evening, at 22.0 and 3.8 GiB: last 2 snapshots deleted (one manual, one weekly) + Trim Filesystem8.3 GiB0.3 GiB

Deleting a Longhorn snapshot frees almost nothing: its data is merged into the next snapshot. The space only comes back once the chain is empty and the filesystem trim has run (with unmapMarkSnapChainRemoved enabled on the volume). Each volume’s S3 backups stayed intact. It is a cousin of the Longhorn memory leak: resources consumed by an internal mechanism, invisible until you compare actual size with content.

Lessons learned

  1. A quiet alert channel only proves that no rule is true. Schedule a full review of the dashboards, panel by panel, with one fixed question: “who would tell me if this number changed?” Here, one night for five problems.
  2. The measurement system must measure itself: database ingestion, disappearing measurements, scripts that stopped running. One “I have no data any more” alert is worth more than ten alerts on the data.
  3. An empty panel is not a zero panel. A Running exporter with a green probe can measure nothing at all. Check that a series exists before trusting it.
  4. Automatic renewals come with conditions. For K3s: at startup, under 120 days, certificate by certificate. A full restart at the wrong time renews nothing, and at the right time, only part of it. Measure expiry instead of assuming renewal.
  5. Prefer alerts on the final resource. TmpfsFilling would have missed the very problem it targeted; memory pressure found it.

Next deadline noted: restart the whole cluster between 13 and 17 June 2027. Until then, if I forget, an alert will remind me, rather than an orange mark I mistake for noise.

monitoringgrafanavictoriametricsvmalertk3slonghorn