Context: the Raspberry Pis’ SMART exporter was not reading any disk until the Grafana dashboard review, and the cluster’s last hard drive was replaced by an SSD at the end of September. Both changes were good news. Together, they produced one alert per night.
The symptom: 52 °C at two in the morning
The alert is called DiskTempHighHDD. It says a disk has been above 50 °C for 15 minutes. It fires at night, almost every night, and always on the same node: a Raspberry Pi worker whose system disk is a SanDisk SSD PLUS attached over USB.
Each alert also triggers the automatic investigation script, which collects context and writes a report. So the report became a daily one too. And every time it said the same thing: nothing is happening on this node.
The numbers, from the metrics database history:
| Observation | |
|---|---|
| Peak temperature | 52 to 53 °C |
| Peak duration | about 20 minutes |
| Host I/O during the peak | none |
| Scheduled job at that time | none |
| Peak time on October 11 | 02:15 UTC |
| Peak time on October 25 | 01:45 UTC |
A disk that heats up without doing any work is already odd. A disk that heats up without doing any work at a time that keeps moving earlier is a clue.
The investigation: a drifting time
The first naive hypothesis is a scheduled job: a weekly fstrim, a backup, a compaction. It does not hold. A cron job or a systemd timer fires at a fixed wall-clock time, and here the time moves back 30 minutes in 14 days, about 2 minutes per day. Nothing on the host runs on a 23 h 58 period.
The second hypothesis is load. It does not hold either: the node’s I/O metrics are flat during the 20-minute peak. The disk heats up without the system asking it for anything.
What remains is what happens inside the disk. An SSD has a life of its own: garbage collection of freed blocks, wear levelling, background scans of its cells. These are triggered by the firmware, on its own clock, which knows nothing about NTP or time zones. An internal period that is not exactly 24 hours produces exactly what we see: a regular peak, of constant length, slowly sliding along the wall clock.
I have no vendor documentation to confirm it. It is the only candidate that fits all three observations: no I/O, stable duration, steady drift. And above all, it is not a problem: 53 °C is far from the 70 °C this SSD is rated for in operation.
The problem was not the disk. It was the rule.
The cause: a rule written for spinning disks
Here is the expression as it was:
smartctl_device_temperature{temperature_type="current", device=~"sd.*"} > 50
The device=~"sd.*" selector meant “SATA disk”, and when the rule was written, a SATA disk in this cluster was a hard drive. The thresholds follow that logic: 50 °C as warning, 58 °C as critical, sensible values for an HDD, whose mechanics age badly in the heat.
Two things changed since then, without anyone re-reading the rule:
- The cluster’s last hard drive left on September 26, replaced by an SSD. Every
sd*on the Pis is now a USB-attached SSD. - The SMART exporter started reading disks again once its image was fixed. The Pis’ five SSDs began reporting their temperature.
The rule stayed the same. What it was watching did not. A 50 °C threshold applied to an SSD doing its nightly housekeeping at 52 °C means one alert per night, guaranteed.
The criterion: rotation rate
The temptation at this point is to exclude the node by name, or to raise the threshold to 55 °C. Both just hide the symptom. A node’s name says nothing about the disk it carries, and a single threshold cannot be right for both an HDD and an SSD.
What is needed is a criterion that describes the hardware itself. smartctl provides one: rotation rate. A spinning disk reports its RPM; an SSD has none. In this homelab, the exporter only exposes the smartctl_device_rotation_rate series for the NAS’s two WD Red drives (5400 RPM). The Pis’ SSDs do not have it at all.
Writing the filter as > 0 rather than relying on the series being absent covers both ways an exporter can behave: a missing series, or a series at zero for an SSD. Either way, the disk is classified as an SSD.
sd* > 50 °C pendant 15 minsd* > 50 °C for 15 min
Écrite pour des disques à plateaux. Le sélecteur sd.* attrape aussi les SSD USB.Written for spinning disks. The sd.* selector also catches USB SSDs.
52-53 °C, ~20 min, sans E/S52-53 °C, ~20 min, no I/O
Un cycle interne du firmware du SSD. L'alerte sonne, une investigation automatique part.An internal firmware cycle of the SSD. The alert fires, an automatic investigation starts.
rotation_rate > 0rotation_rate > 0
Seuls les disques à plateaux exposent une vitesse de rotation : 2 HDD d'un côté, 5 SSD de l'autre.Only spinning disks expose a rotation rate: 2 HDDs on one side, 5 SSDs on the other.
HDD 50/58 °C · SSD 60/68 °CHDD 50/58 °C · SSD 60/68 °C
and on(…) pour les HDD, unless on(…) pour les SSD : exclusives. Le pic nocturne reste sous le seuil.and on(…) for HDDs, unless on(…) for SSDs: mutually exclusive. The nightly peak stays below the threshold.
The HDD rules keep their thresholds, but only for disks that spin:
- alert: DiskTempHighHDD
expr: |
smartctl_device_temperature{temperature_type="current", device=~"sd.*"} > 50
and on(instance, device) smartctl_device_rotation_rate > 0
for: 15m
labels: { severity: warning }
And two new rules take care of the SSDs, with thresholds designed for them:
- alert: DiskTempHighSSD
expr: |
smartctl_device_temperature{temperature_type="current", device=~"sd.*"} > 60
unless on(instance, device) smartctl_device_rotation_rate > 0
for: 15m
labels: { severity: warning }
- alert: DiskTempCriticalSSD
expr: |
smartctl_device_temperature{temperature_type="current", device=~"sd.*"} > 68
unless on(instance, device) smartctl_device_rotation_rate > 0
for: 5m
labels: { severity: critical }
and on(instance, device) keeps the series that have a rotation rate; unless on(instance, device) keeps those that do not. It is the same pair of operators that made two restart rules mutually exclusive in 48 Alerts for 24 Pods, for the same reason: a disk falls into exactly one family, never two, never none.
The SSD thresholds start from the operating limit: 60 °C leaves 10 °C of headroom below the vendor’s 70 °C and sits 7 °C above the observed nightly peak; 68 °C is the point where someone needs to look right away. NVMe drives already had their own rule, at 70 °C; it does not change.
The check: replaying last night’s peak
A rewritten rule that “should” work proves nothing. The metrics database, however, still holds the previous night’s peak. On October 25 at 01:58 UTC, the worker’s SSD was at 52 °C. Evaluated at that exact moment:
| Expression | Result at the peak |
|---|---|
| Old HDD rule | the worker’s SSD |
| New HDD rule | nothing |
| New SSD rules | nothing |
smartctl_device_rotation_rate > 0 | the NAS’s 2 WD Red drives |
sd* disks without a rotation rate | the Pis’ 5 SSDs |
The criterion puts every disk in the homelab on the right side, and the alert that fired that night no longer exists under the new rules. One last check before pushing: vmalert -dryRun on the rules file, with no errors.
In fairness: a 52 °C peak does not exercise the “SSD genuinely too hot” path. The SSD rules have not yet been put to the test by a real breach. The day one of them fires, it will be for a good reason, or for a new lesson.
Lessons learned
1. A threshold encodes an assumption about the hardware
> 50 °C is not a truth about “disks”, it is a truth about hard drives. The day the hardware changes, the rule goes wrong silently, then very loudly. A hardware migration (here, HDD to SSD) should always trigger a review of the rules that cover it.
2. A drifting time is not a cron job
An event that comes back every day at a slightly different time does not come from a scheduled job on the host: those are pinned to the wall clock. A steady drift of a few minutes a day points to a clock that does not follow NTP, typically a firmware. Recording the exact time of the symptom over two weeks costs one query and rules out a whole family of hypotheses.
3. Classify by what the hardware is, not where it lives
Excluding a node by name would have worked until the next migration. smartctl_device_rotation_rate describes the disk itself and will follow the next SSD, the next HDD, the next node, with no changes.
4. and on / unless on: two families, no gap
When a population splits in two, the same condition used once with and and once with unless guarantees that each series lands in exactly one rule. No duplicates, nothing left out.
5. Replay the symptom on history before pushing
The metrics database keeps yesterday’s peak. Evaluating the old and new expressions at that moment proves both that the fix silences the false positive and that the criterion classifies every disk correctly. That is more convincing than re-reading YAML, and it does not require waiting for the next night.
The known limit
A hard drive behind a USB bridge that hides its rotation rate would land in the SSD family, with thresholds too generous for it. That is not the case today. If an HDD ever comes back to the cluster, the first thing to check will be that it reports its RPM.
The alert channel no longer rings at night. The SSD, for its part, keeps doing its housekeeping at 52 °C, somewhere between 1 and 2 a.m., a little earlier every day.