Context: the cluster’s storage architecture is described in Distributed block storage with Longhorn, and the full shutdown during which this migration took place in Powering down a whole GitOps k3s cluster.
The symptom: 47 IOPS in a cluster of SSDs
On 20 September 2026, on the side of the rack move, I finally measured the nodes’ disks one by one. A direct test with O_DIRECT, 4 KB random accesses, 400 samples per node. Nothing sophisticated, just enough to compare.
| Node | Median latency | Minimum latency | Random IOPS | Sequential throughput |
|---|---|---|---|---|
| worker-1 (SanDisk SSD) | 0.199 ms | 0.081 ms | 5,448 | 379 MB/s |
| worker-2 | 20.17 ms | 4.03 ms | 47 | 102 MB/s |
| master-1 (Intenso SSD) | 0.243 ms | 0.095 ms | 2,382 | 219 MB/s |
A hundred times slower than its neighbour. And this is not a worn-out SSD: the minimum latency gives the culprit away. An SSD bottoms out around 0.08 ms whatever you ask of it. 4.03 ms is the time a read head needs to move. worker-2, a 16 GB Pi 5, had been booting from day one off a Toshiba MQ01UBD050: a 500 GB 2.5-inch hard drive, 5400 rpm, over USB. The only spinning disk in the cluster.
Production metrics, averaged over seven days by node_exporter, tell the same story more quietly: 3.753 ms write latency on worker-2 against 0.643 ms on worker-1, 5.097 ms for reads against 0.094 ms. The most telling figure is the peak: 1,742 IOPS requested at peak from a disk that can handle 47. Roughly 37 times its capacity.
One might think: a slow node slows down the pods on that node, full stop. Except that with Longhorn, that is not how it works.
The mechanism: the slowest replica sets the pace
A Longhorn volume is an engine that receives the pod’s writes and forwards them to each of its replicas. The write is synchronous: the engine acknowledges to the pod only once every replica has been written. That is what lets you lose a node without losing data — and it is also what ties the whole volume to its slowest replica.
In this cluster, replicas live on only two nodes: worker-1 and worker-2. The masters hold none, and the third worker runs off a USB stick and stays out of the pool. As of 21 September, worker-1 held 13 replicas and worker-2 held 15. In other words, nearly every volume had one foot on the spinning disk.
The consequence is counter-intuitive: a pod running on worker-1, with a local replica on a perfectly healthy SSD, still pays the Toshiba’s latency on every fsync. This was not “a slow node”. It was the only measured hardware bottleneck in the cluster, and it applied to everyone.
The remedy was obvious: replace the disk. An Intenso 512 GB SSD was sitting in the NAS, never mounted, with a clean SMART record. What remained was the method.
The decision: clone rather than let Longhorn rebuild
There are two classic ways to change the root disk of a Kubernetes node.
Reinstall and let Longhorn do its job. A fresh system on the SSD, the node rejoins the cluster, Longhorn notices that 15 replicas are gone and rebuilds them from worker-1. It works, but it means copying some forty gigabytes of replicas over the network, during a window in which every affected volume has only one healthy copy left. Add reinstalling k3s, the node identity, and around 68 GB of image cache to download again.
Clone bit for bit. Copy the entire disk, sector by sector, onto the SSD. The copy carries everything: system, k3s, node identity, image cache, and the Longhorn replicas as they are. Above all, it reproduces the disk’s MBR signature. /etc/fstab and cmdline.txt refer to partitions by PARTUUID, which is derived from that signature: on the copy, they are identical. No file to edit. The Pi boots from the SSD as if nothing had happened.
Cloning won, with three rules set before touching any hardware:
- The source is never written to. Rollback, at any step: plug it back in on its own.
- Never boot the Pi with both source and copy attached. Same
PARTUUIDs: the system would pick one at random. - The source stays untouched for 14 days after the switch.
There was a cluster-side prerequisite too. Taking inventory of the volumes, two of them — a MySQL database and the VictoriaLogs storage — had only one replica, on worker-2 of all places. Both were moved to 2 replicas, and you wait for robustness=healthy, not just running: a replica that is still rebuilding is already running, which is not a sufficient signal. That detail would pay off sooner than expected.
Plan A: clone on the Pi itself (failed)
The first plan was simple: plug the SSD into worker-2’s second USB 3 port and run the copy from the Pi. It would test the cable and the USB-SATA bridge under real conditions, and avoid pulling the disk out.
On 26 September at around 15:57, the SSD was hot-plugged into worker-2 with the system running. The Toshiba dropped out at once. The system disk vanished from under the node: container engine stopped, SSH cut.
The cause is electrical. The USB ports on a Pi 5 share a common 5 V budget. A 2.5-inch hard drive over USB already draws a fair share of it; a second bus-powered device at plug-in time is enough to push the whole thing below the threshold. No explicit error message, just a disk that is no longer there.
The network-side symptom is worth knowing, because it misleads: the Pi answers ping, but SSH drops the connection at the banner exchange. A live kernel, a dead disk. Since OpenSSH 9.8, the daemon re-reads the sshd-session executable from disk on every new connection; no disk, no session. The only way out: cut the power and reboot with a single disk.
No data was lost. The two volumes moved to 2 replicas an hour earlier now had a copy on worker-1. Without that precaution, this would have been a different story.
The rule that came out of it now sits at the top of the cluster’s documentation: never two USB disks on the same Pi 5.
Plan B: the PC, WSL2, and a full cluster shutdown
If the Pi cannot carry two disks, the PC can: an 850 W power supply does not even notice two USB disks. And the copy happens cold: neither disk is mounted, no system is running on it while it is read.
The timing worked out: a full cluster shutdown was already planned that day to fit two fans. The migration was grafted onto it, using the shutdown runbook corrected after the rack move. No drain, no capacity question on worker-1: every volume detached, every node off.
On the PC side, WSL2 can hand a whole physical disk to the Linux VM with wsl --mount --bare. Three details cost time:
wsl --mountrefuses a disk “in use” (0x8007006c) as soon as Windows has assigned a letter to the Pi’s FAT32 boot partition. Both disks must first be taken offline in Windows. The mount script does that, and refuses any boot or system disk of the PC while it is at it.- WSL releases the disks if it goes idle. A
sleep infinityin the distribution, running in the background, keeps it awake. Same logic for the PC: sleep must be disabled on mains power for the whole operation. - Under WSL, a passed-through disk is called “Virtual Disk”. Model and serial number are gone. To tell source from target, only two things remain: the exact size in bytes, and the disk signature. That is exactly what the guardrails rely on.
The guardrails: making a reversal impossible
A dd in the wrong direction means the Toshiba overwritten with the contents of an empty SSD, and no way back. With two disks both named “Virtual Disk”, that mistake is anything but exotic. So the cloning script trusts nobody, least of all the operator:
[ "$s" = "$SRC_SIZE" ] || die "source size $s bytes, expected $SRC_SIZE"
[ "$d" = "$DST_SIZE" ] || die "target size $d bytes, expected $DST_SIZE"
[ "$d" -ge "$s" ] || die "target is smaller than source"
pt_src=$(blkid -p -s PTUUID -o value "$SRC" 2>/dev/null || true)
pt_dst=$(blkid -p -s PTUUID -o value "$DST" 2>/dev/null || true)
[ "$pt_src" = "$SIG" ] || die "source signature [$pt_src], expected $SIG"
[ "$pt_dst" != "$SIG" ] || die "target already carries $SIG: reversed, or already cloned"
The source must have exactly the expected size and signature. The target must have exactly its own size, and must not carry the source’s signature. Swapping the two arguments therefore fails twice over. Re-running the script after a successful clone fails too, since the target now carries the signature. The defaults are those of this migration; for another disk, unless they are passed as environment variables, the script refuses — safely. On top of that: a refusal if either disk is mounted, and a DRY_RUN=1 mode that runs every check without writing anything.
These guardrails were tested before touching the real disks, on fake ones: sparse files at the real sizes, attached as loop devices, with a minimal MBR table carrying the right or the wrong signature. Seven cases: the correct configuration, source and target swapped, source equal to target, target already carrying the signature, target of the wrong size, source of the wrong size, non-existent device. Expected: one OK, six refusals. Result: 7 out of 7.
Two more details in the script, each born from a real trap:
wipefs -a "$DST"
dd if="$SRC" bs=4M iflag=fullblock status=progress \
| tee >(sha256sum | awk '{print $1}' > "$SHA") \
| dd of="$DST" bs=4M iflag=fullblock oflag=direct
First, wipefs. If the target has ever carried a GPT table, that table has a backup copy at the end of the disk. Since the SSD is larger than the Toshiba, that end is not covered by the copy: you end up with a brand-new MBR table at the start and a ghost GPT at the end, which some tools take seriously. So everything is wiped first.
Then tee: the source’s SHA-256 fingerprint is computed during the copy. No need to re-read 500 GB from a 5400 rpm disk just to get a checksum.
Verification: SHA-256 first, e2fsck second
The verifier runs four checks: same signature on both sides, identical partition tables read from the disks, SHA-256 of the target’s first bytes (over the exact size of the source) compared with the one computed during the copy, then a read-only e2fsck -fn of the root partition. It only concludes CLONE VALIDE if all four pass.
The fourth one failed.
e2fsck: unsupported feature(s): FEATURE_C12
False alarm. The Pi’s ext4 was created by Debian 13, which enables the orphan_file feature. The e2fsck 1.46 shipped with Ubuntu 22.04 under WSL does not know it, and refuses to go further. Proof that the clone was not at fault: the same e2fsck on the perfectly healthy Toshiba returns exactly the same error.
The verifier was fixed on the spot: when e2fsck declares itself too old, it falls back to dumpe2fs -h, which reads the header without interpreting anything. It requires a clean state and no needs_recovery in the features — in other words, an empty journal. But to be honest: the decisive proof is the SHA-256. If both fingerprints match, the target is the source, bit for bit; the filesystem check really only verifies the state in which the source was shut down.
Two more subtleties in the verifier. The root partition is read from the source’s partition table rather than hard-coded. And it is accessed through a loop device with an offset, because after a dd the kernel’s view of the target’s partitions is stale.
The timeline: 1 h 49 of copying, 41 minutes of proof
| Step | Duration |
|---|---|
| Copy (17:24 → 19:13) | 1 h 49, averaging 76 MB/s — 112 MB/s at the start of the disk, less towards the end of the platter |
| SHA-256 verification | 41 min — re-reading the SSD at ~200 MB/s through USB and WSL |
| Cluster restart, masters on → 32/32 applications | 10 min |
The copy’s throughput curve is that of a hard drive, not of a cable: the outer tracks of the platters pass under the head faster than the inner ones, hence 112 MB/s at the start and a 76 MB/s average.
Switch-over: the SSD alone on worker-2, USB 3 port. At boot, three checks: lsblk, findmnt / to make sure the root is on the copy, blkid to confirm the same PARTUUID. Then the cluster restart, with the corrected runbook.
The result: 76× on latency, zero rebuilds
At 22:36 the migration was closed. On return, Longhorn found its replicas exactly where it had left them: 15 of 15 volumes healthy, without a single rebuild. Not one byte copied over the network. That was the whole point of the clone.
| worker-2 | Toshiba (20.09) | Intenso SSD (26.09) | Gain |
|---|---|---|---|
| Median latency | 20.17 ms | 0.264 ms | 76× |
| Minimum latency | 4.03 ms | 0.081 ms | SSD floor, same as worker-1 |
| Random IOPS | 47 | 3,727 | 79× |
| Sequential throughput | 102 MB/s | 376 MB/s | 3.7×, the USB 3 ceiling |
One amusing detail to finish: lsblk still reports ROTA=1 for the SSD. As far as the kernel is concerned, it is still a rotating disk. The USB-SATA adapter passes on neither the rotation rate nor SMART, and Linux assumes the worst. The 0.081 ms minimum latency does not lie. A udev rule forcing queue/rotational=0 would set things straight.
The other traps, in no particular order
fsck.fat“Writing changes” on first boot. The target, back online under Windows before being unplugged, had been given a drive letter — and the boot partition bears the trace. Nothing serious, but exactly what we wanted to avoid: keep the disks offline until they are unplugged.- A check reporting “ABSENT” while the file exists.
set -o pipefailplusgrep -qat the end of a pipe:grepstops at the first match, closes the pipe, the upstream command gets SIGPIPE, andpipefailturns that success into a failure. Fix:grep -q … <<< "$var", or go through a file. - Shell scripts in CRLF. The Windows machine was converting line endings, which is fatal under WSL. A
.gitattributesnow forces*.shto LF. - Variables swallowed by
wsl.exe -- bash -c '…'. WSL scripts are run from files, never inline.
Takeaways
- A synchronous Longhorn write is as slow as its slowest replica. A single poor disk in the pool affects every volume with a replica on it, including for pods on other nodes. Measuring each disk’s minimum latency is enough to spot the odd one out.
- Clone rather than rebuild. A bit-for-bit clone keeps replicas, node identity, image cache and
PARTUUIDs. Longhorn finds everything as it was; there is no window in which a volume is down to a single copy. - Before touching hardware, no volume may have its only replica on the node concerned, and you wait for
healthy, notrunning. - Never two USB disks on a Pi 5. Clone elsewhere, cold.
- Identify disks by exact size and signature, and write guardrails that make a reversal impossible. Test them on fake disks before the real ones.
wipefsthe target when it is larger than the source: the backup GPT lives at the end.- The fingerprint is the proof. SHA-256 computed during the copy, compared on the target. An outdated
e2fsckcan cry wolf. - Keep the source untouched for two weeks. Rollback is one cable away.
The limits, for completeness: all of this assumes a target at least as large as the source, and a window during which the node is down — here, a full shutdown that was already planned. The roughly 11 GiB of free space at the end of the SSD has not been reclaimed yet. And the NVMe question remains open, without urgency: with a single PCIe 2.0 lane on the Pi 5, the throughput gain would top out around 19%. The only real argument would be to remove the USB layer: a reliability gain, not a performance one.
It took two disks, a PC and an afternoon for an fsync on worker-1 to stop waiting for a read head on worker-2.