Stockage bloc distribué avec Longhorn Distributed Block Storage with Longhorn

Why Longhorn?

10 volumes going Degraded all at once — that is the price of a design assumption that looked reasonable on paper (tagging the masters’ SSDs as critical) and that collapsed the moment scheduling was disabled on those very masters. This post covers the two-tier storage architecture that followed.

Persistent storage is the hardest problem to solve in Kubernetes. Cloud-native solutions (EBS, GCE PD) are trivial in the cloud, but on bare metal the options are limited: NFS (single point of failure), Ceph (too heavy for RPis), or Longhorn.

Longhorn is a distributed block storage system developed by Rancher (now CNCF). It replicates data across several nodes, exposes native Kubernetes PersistentVolumes, and is managed entirely through a web interface. Its relative lightness makes it the ideal choice for an RPi cluster.

A two-tier architecture

The first version of the cluster used a single StorageClass. With experience, I moved to a two-tier strategy:

Tier	StorageClass	diskSelector	numberOfReplicas	Use case
Critical	critical-storage	critical	3	Databases, secrets, irreplaceable data
Standard	standard-storage	standard	2	Caches, artifacts, rebuildable data

The diskSelector is the key: Longhorn only schedules a volume’s replicas onto disks carrying the matching tag.

StorageClass configuration

# critical-storage
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: critical-storage
parameters:
  diskSelector: critical
  numberOfReplicas: '3'
  dataLocality: disabled
  replicaAutoBalance: best-effort
  fsType: ext4
  staleReplicaTimeout: '30'
provisioner: driver.longhorn.io
reclaimPolicy: Retain
volumeBindingMode: Immediate
allowVolumeExpansion: true

reclaimPolicy: Retain is crucial for critical data — the volume is not deleted when the PVC is deleted. An explicit manual action is required.

Disk tagging

Tagging is handled through the Longhorn interface (Settings → Nodes). Each disk on each node can carry one or more tags.

Initial design (mistake): the masters’ SSDs were tagged critical, the workers’ standard. The assumption was that the masters, being more stable, should host the critical data.

Problem discovered: after scheduling was disabled on the masters to relieve the control-plane, critical volumes could no longer create replicas — no critical disk was schedulable. Result: 10 volumes in Degraded state.

Fix: add the critical tag to the SSDs of workers worker-1 and worker-2, then migrate the replicas through sequential eviction.

Lesson: critical data belongs on the most powerful nodes (the RPi 5 workers), not on the masters whose primary mission is the control-plane.

Replica eviction and migration

To migrate replicas from the masters to the workers without service interruption:

For each master (sequentially, not in parallel):
1. In the Longhorn UI → Node → Disable scheduling
2. In the Longhorn UI → Node → Request eviction
3. Wait for all replicas to be migrated (watch for "Healthy" status)
4. Move on to the next master

Sequential eviction is mandatory. Evicting two masters simultaneously with numberOfReplicas: 2 means neither target node has enough replicas to maintain quorum — the volumes go Degraded with no way to rebuild.

longhorn-system Migration des répliques : masters → workers, un à la fois Replica migration : masters → workers, one at a time volume critical · 3 répliques critical volume · 3 replicas

① Évincer master-1 Evict master-1

master-1scheduling off worker-1tag critical

✓ attendre Healthy wait for Healthy

Disable scheduling → Request eviction

réplique reconstruite sur le worker ; quorum maintenu (2/3 restantes) replica rebuilt on the worker; quorum maintained (2/3 remaining)

② Évincer master-2 Evict master-2

master-2scheduling off worker-2tag critical

✓ attendre Healthy wait for Healthy

Disable scheduling → Request eviction

réplique reconstruite sur le worker ; quorum maintenu (2/3 restantes) replica rebuilt on the worker; quorum maintained (2/3 remaining)

③ Évincer master-3 Evict master-3

master-3scheduling off worker-3tag critical

✓ attendre Healthy wait for Healthy

Disable scheduling → Request eviction

réplique reconstruite sur le worker ; quorum maintenu (2/3 restantes) replica rebuilt on the worker; quorum maintained (2/3 remaining)

⚠ piège ⚠ pitfall Évincer deux masters en parallèle retire trop de répliques d’un coup : plus de quorum, plus de reconstruction possible → volume Degraded. Evicting two masters in parallel removes too many replicas at once : no more quorum, no more rebuild possible → Degraded volume.

Managing the replica count

With numberOfReplicas: 3 and critical-tagged workers on worker-1, worker-2 and worker-3, all 3 replicas can be scheduled. At the time of this article, the configuration used 3 replicas for critical-storage and 2 for standard-storage, matching the number of workers able to host each tier.

Update (2026-07): the StorageClass architecture has evolved since — the cluster now uses 3 classes (critical, longhorn, standard) with different replica rules. See the k3s-homelab repo for the current state.

# Check the state of every volume
for vol in $(kubectl get volumes.longhorn.io -n longhorn-system -o name); do
  kubectl get $vol -n longhorn-system \
    -o custom-columns='NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness,REPLICAS:.spec.numberOfReplicas'
done

All volumes in Healthy state with 2 or 3 replicas depending on their StorageClass.

Monitoring volume state

Longhorn exposes Prometheus metrics. The critical alerts to watch:

• longhorn_volume_state ≠ 1 (1 = Healthy) → degraded volume
• longhorn_volume_robustness = 3 (3 = Faulted) → unusable volume
• longhorn_node_status → state of the storage nodes

# Quick check from kubectl
kubectl get volumes.longhorn.io -n longhorn-system \
  -o custom-columns='NAME:.metadata.name,STATE:.status.state,ROBUSTNESS:.status.robustness'

Backing up to MinIO

Longhorn supports backups to an S3 endpoint. With MinIO deployed on the cluster, you get a complete backup system with no external dependency.

Configuration in Longhorn Settings:

• Backup Target: s3://longhorn-backups@us-east-1/
• Backup Target Credential Secret: secret holding the MinIO credentials
• S3 Endpoint: http://minio.minio.svc.cluster.local:9000

Backups are scheduled through Longhorn RecurringJobs. For critical volumes: daily backup at 2am, 7-day retention.

Things to watch out for

Longhorn is memory-hungry on the nodes hosting active replicas. On worker-2, longhorn-engine processes could consume up to 1.5 GB of RAM with multiple active volumes. Watching worker memory is essential.

Never delete a Retain PVC without checking: the underlying Longhorn volume persists and keeps consuming space. Clean up orphaned volumes regularly through the Longhorn interface.

dataLocality: disabled is the right choice for an RPi with Gigabit network access. dataLocality: best-effort or strict-local may look better for performance, but it creates scheduling problems when pods migrate to nodes without a local replica.

This two-tier architecture broadly held up — until an upstream regression in the instance-manager leaked several gigabytes of RAM on the storage nodes. See the incident and its runbook.