Context: the two-tier storage architecture (critical/standard tiers, replicas, disk tagging) is detailed in Distributed block storage with Longhorn.
The symptom: 12 GiB for a pod that “does nothing”
On 25 April 2026, a monitoring dashboard shows something abnormal on worker-1: available RAM has dropped below 4 GiB out of 16 GiB. Nothing unusual on the application workload side. Digging in with kubectl top pods -n longhorn-system, the culprit is immediately obvious.
NAME CPU(cores) MEMORY(bytes)
instance-manager-e3a8b2c1d4f6... 12m 12426Mi
longhorn-manager-xk9p2 8m 87Mi
longhorn-csi-plugin-r4t7n 3m 42Mi
12.4 GiB in an instance-manager pod. A pod whose job is to manage Longhorn’s engine and replica processes on that node — nothing that justifies approaching the RPi 5’s RAM limit. The pod had been alive for 32 days.
The pod is restarted manually, memory falls back to about thirty megabytes. Incident closed… for now.
On 11 May, the same scenario: worker-1 again, 9.6 GiB after 15 days. On 17 May, three pods at once during a routine check:
| Node | RAM consumed | Pod uptime | Load |
|---|---|---|---|
| worker-1 | 3.0 Gi | 6 days | active replicas |
| master-3 | 2.4 Gi | 6 days | vmsingle engine (heavy I/O) |
| worker-3 | 311 Mi | 70 days | idle, zero replicas |
The last case is the telling one: worker-3 hosts no active replica and no I/O traffic — and it still leaks, just more slowly. This is not a matter of load, it is structural.
The context: what is an instance-manager?
Longhorn is a distributed block storage system for Kubernetes. Its architecture rests on three layers of processes that interact to present a PersistentVolume to an application pod.
The longhorn-manager runs as a DaemonSet on every node. It is the central brain: it orchestrates volume creation, manages StorageClasses, and watches the global cluster state. It never touches the data directly.
The engine and replica processes are the real storage engines. For every mounted volume:
- engine: exposes the iSCSI/NVMe block device to the application pod, runs on the node where the pod is scheduled
- replica: writes and reads the actual data on the local disk, one process per replication node
With numberOfReplicas: 2, a volume has one engine and two replicas running on two different nodes. Data is synchronised in real time between replicas over the network.
These processes are independent Go binaries, but they do not run directly as Kubernetes pods. Longhorn drives them through one coordinator pod per node: the instance-manager.
The instance-manager acts as the local supervisor. It starts the engine and replica processes when a volume has to be served on the node, monitors their health, and talks to them over gRPC to report metrics and events back to the longhorn-manager. It is also responsible for restarting them if one of them crashes.
On a typical cluster with 15 volumes and 3 worker nodes, each instance-manager therefore potentially manages 10 to 30 child processes at once, with constant gRPC traffic.
On a heterogeneous RPi homelab — 16 GiB on worker-1 and worker-2, but only 2 GiB on worker-3 — with RAM shared between the system, the application pods and the Longhorn processes, a multi-GiB leak in that single coordinator pod directly threatens the stability of the whole node. The danger is proportionally worse on worker-3, where a few hundred MiB already weigh heavily. An OOM killer that kills an application pod is annoying. An OOM killer that kills engine or replica processes mid-write is potential data corruption.
The investigation: isolating the leak
Faced with abnormal memory consumption in the longhorn-system namespace, there are three natural suspects: the longhorn-manager itself, the child engine/replica processes, or the instance-manager binary. The method for telling them apart is reproducible and worth documenting.
Step 1 — Confirm it is the IM, not the children
The naive first hypothesis: the engine or replica processes are the ones consuming. We enter the pod to check.
# Identify the PID of the main instance-manager process
kubectl -n longhorn-system exec -it instance-manager-e3a8b2c1d4f6 -- \
ps aux --sort=-%mem | head -20
PID USER %CPU %MEM VSZ RSS COMMAND
1 root 0.1 77.4 13107200 12654592 /usr/local/bin/longhorn-instance-manager
142 root 0.0 0.2 458240 32768 longhorn-engine --listen ...
143 root 0.0 0.1 368640 28672 longhorn-engine --listen ...
201 root 0.0 0.2 442368 30720 longhorn-replica --sync-agent ...
Clear enough: the main binary (longhorn-instance-manager, PID 1) absorbs all the RAM. The engine and replica children stay stable at ~30 MiB each.
Step 2 — Distinguish RSS from VmData
RSS (Resident Set Size) can include shared pages. To confirm this is really a heap leak in the Go binary itself, we read /proc/1/status directly:
kubectl -n longhorn-system exec -it instance-manager-e3a8b2c1d4f6 -- \
cat /proc/1/status | grep -E "VmRSS|VmData|VmHeap"
VmRSS: 12654592 kB
VmData: 12587008 kB
VmData represents the process’s data segment — Go heap included. With 12.3 GiB in there and only ~60 MiB of anything else, the conclusion is beyond doubt: the Go garbage collector is not reclaiming that memory. This is a leak in the instance-manager binary itself.
Step 3 — Look for the upstream issue
The pattern is specific enough (Go binary, gRPC, grows with time and I/O) to search the Longhorn tracker directly. Searching instance-manager memory leak leads within minutes to issue #12573:
[BUG] Very high memory consumption in longhorn-instance-manager Severity: 1 — Regression introduced in v1.11.0
The confirmed cause: a regression in the instance-manager v1.11.0 gRPC layer causes unreleased structures to pile up on the Go heap. The leak rate is proportional to the volume of gRPC traffic (so to the node’s I/O activity), but present even at rest — which explains the 311 MiB on an idle worker-3 after 70 days.
The fix was merged and backported into v1.11.1, then shipped in v1.11.2.
The workaround (historical, v1.11.0 only)
The workaround is surgical: delete the bloated pod. Longhorn-manager recreates it in 15–30 seconds with fresh memory. The on-disk data does not move — the new instance-manager remounts the existing replica directories from /var/lib/longhorn with no loss.
But deleting an instance-manager pod is not a free action. You first have to make sure every volume hosted on that node has at least one healthy replica elsewhere.
Mandatory pre-flight checks
# 1. List the active engines on the node in question
sudo kubectl -n longhorn-system get engines.longhorn.io \
-o custom-columns="NAME:.metadata.name,NODE:.spec.nodeID,STATE:.status.currentState" \
| grep worker-1
# Example of expected output:
# pvc-abc123-e-0 worker-1 running
# pvc-def456-e-0 worker-1 running
# 2. List the replicas on that node
sudo kubectl -n longhorn-system get replicas.longhorn.io \
-o custom-columns="NAME:.metadata.name,VOL:.spec.volumeName,NODE:.spec.nodeID,STATE:.status.currentState" \
| grep worker-1
# Example:
# pvc-abc123-r-abc pvc-abc123 worker-1 running
# pvc-abc123-r-def pvc-abc123 worker-2 running ← replica on another node OK
# pvc-def456-r-ghi pvc-def456 worker-1 running
# pvc-def456-r-jkl pvc-def456 worker-3 running ← replica on another node OK
# 3. For each volume, confirm ≥1 healthy replica on another node
sudo kubectl -n longhorn-system get volumes.longhorn.io pvc-abc123 \
-o jsonpath='{.status.robustness}'
# Expected: healthy or degraded (never faulted before continuing)
The essential check: every volume must have at least one replica in running on a node other than the one being restarted. If a volume has a single replica (on that node), deleting it would make the volume unavailable — that is the one situation where you must not proceed.
The action
# Identify the exact pod name
sudo kubectl -n longhorn-system get pod -l app=longhorn-instance-manager \
-o wide | grep worker-1
# Delete with a reasonable grace period
sudo kubectl -n longhorn-system delete pod instance-manager-e3a8b2c1d4f6 \
--grace-period=30
What happens next
t+0s Pod instance-manager-e3a8b2c1d4f6: Terminating
t+5s Affected volumes: transition to Degraded (engine loses the local gRPC)
t+15s Pod instance-manager-f7c9d1e2b4a8: Running (new pod recreated)
t+25s Volumes: back to Healthy (IM reconnected to the on-disk replicas)
t+30s instance-manager RAM: ~25 Mi (idle)
Applications see nothing during that window: I/O keeps flowing through the replicas on the other nodes. The Longhorn CSI driver keeps the block device mounted on the application side.
sur ~30 jours · heap Go jamais récupéré over ~30 days · Go heap never reclaimed
stables — données sur disque dans /var/lib/longhorn stable — data on disk in /var/lib/longhorn
t+0s
le pod instance-manager bloaté passe en Terminating the bloated instance-manager pod transitions to Terminating
t+5s
l’engine perd le gRPC local — I/O servies par les répliques distantes the engine loses its local gRPC — I/O served by the remote replicas
t+15s
longhorn-manager recrée l’instance-manager avec une mémoire fraîche longhorn-manager recreates the instance-manager with fresh memory
t+25s
l’IM se rattache aux répertoires replica déjà présents sur disque the IM reattaches to the replica directories already present on disk
t+30s
plan de contrôle stateless redémarré ; le plan de données n’a pas bougé stateless control plane restarted; the data plane hasn’t moved
/var/lib/longhorn et le nouvel instance-manager les remonte au démarrage.
Plan de contrôle stateless, plan de données intact.
Deleting the coordinator doesn't delete the replicas : the data lives in
/var/lib/longhorn and the new instance-manager remounts it at startup.
Stateless control plane, data plane untouched.
When not to apply the workaround
- Volume with a single replica AND the engine on that node → zero tolerance, the volume becomes unavailable
- Node currently being evicted by Longhorn → wait for it to finish before intervening
- Several nodes stuck at once → handle one node at a time, confirm the return to
Healthybefore the next
The permanent fix: upgrade to v1.11.2
On 18 May 2026, cluster upgrade through Helm:
# Check the current version
sudo helm list -n longhorn-system
# NAME NAMESPACE REVISION UPDATED STATUS CHART
# longhorn longhorn-system 7 2026-04-01 ... deployed longhorn-1.11.0
# Upgrade to v1.11.2
sudo -E helm upgrade longhorn longhorn/longhorn \
-n longhorn-system \
--version 1.11.2 \
--reset-then-reuse-values \
--wait --timeout 15m
The rolling upgrade replaces the instance-manager pods node by node. After each replacement, volumes pass briefly through Degraded then return to Healthy — expected behaviour, identical to the manual workaround.
Then trigger the automatic upgrade of the engine binaries (the child processes are still running the v1.11.0 binary until this step):
sudo kubectl -n longhorn-system patch setting \
concurrent-automatic-engine-upgrade-per-node-limit \
--type=merge -p '{"value":"3"}'
Post-upgrade metrics (observed on 18 May)
| Node | Role | instance-manager RAM |
|---|---|---|
| worker-3 | idle, zero replicas | 11 Mi |
| worker-1 | 12 active replicas | 250–330 Mi |
| worker-2 | 12 active replicas | 260–310 Mi |
| master-1 | 1–2 engines | 13–43 Mi |
| master-3 | vmsingle engine (heavy I/O) | 43 Mi |
The contrast is striking: worker-3, which topped out at 311 MiB after 70 days of slow leaking, is now stable at 11 MiB. master-3, which ran the vmsingle engine at 2.4 GiB, sits at 43 MiB.
Full incident timeline
| Date | Action | Result |
|---|---|---|
| 2026-04-25 | First documented bloat — worker-1 at 12.4 Gi after 32d — pod restart | 12.2 Gi reclaimed |
| 2026-05-11 | Recurrence — worker-1 at 9.6 Gi after 15d — pod restart | ~9.5 Gi reclaimed |
| 2026-05-17 | Cycle confirmed (6d after last fix) — worker-1 3 Gi, master-3 2.4 Gi, worker-3 311 Mi | Restart × 3 pods |
| 2026-05-18 | Upgrade v1.11.0 → v1.11.2 | Root cause eliminated upstream |
Lessons learned
1. Monitor system pods individually, not just the node
kubectl top nodes shows the node’s total RAM — but it does not distinguish a pod that is ballooning from legitimate application load. The alert that should have existed from day one:
# Prometheus alert to add
- alert: LonghornInstanceManagerMemoryHigh
expr: |
container_memory_working_set_bytes{
namespace="longhorn-system",
pod=~"instance-manager-.*"
} > 2 * 1024 * 1024 * 1024
for: 10m
labels:
severity: warning
annotations:
summary: "Instance-manager {{ $labels.pod }} > 2 GiB RAM"
2. On a homelab, latent Severity/1 bugs stay invisible for a long time
In a managed production environment, a Severity/1 triggers alerts, war rooms and emergency patches within hours. On a homelab, v1.11.0 was installed and forgotten. The leak took 32 days to become visible enough to raise an alarm — 32 days during which the node was silently creeping towards the OOM threshold.
Two reflexes worth acquiring before upgrading a storage component:
- Read the target version’s CHANGELOG and the open issues labelled
kind/bugorseverity/1 - Treat
.0releases of critical components (storage, network, PKI) with the same caution as a beta — wait at least for a.1, or a few weeks of community feedback
In this case, v1.11.1 was available shortly after the issue #12573 report. Staying on v1.11.0 more than a few weeks after v1.11.1 shipped was avoidable.
3. Data survives an instance-manager restart — that is by design
The most counter-intuitive behaviour of the incident: deleting the pod that “manages” the replicas does not delete the replicas. The data lives on the physical disk under /var/lib/longhorn. From the data’s point of view the instance-manager is a stateless coordinator — it knows where the replica directories are on disk, it remounts them at startup, it does not own them.
This strict separation between the control plane (instance-manager) and the data plane (files on disk) is a Longhorn design principle. It is what makes the workaround safe, and it also explains why an OOM crash of the instance-manager itself — without killing the children — does not corrupt the data. It is the engine and replica processes that hold the locks on the data blocks, not their supervisor.
4. A pre-written runbook changes everything
The difference between “panicking at 11pm with 500 MiB of free RAM on a worker” and “running the procedure in 3 minutes” comes down to having a runbook with the pre-flight checks already written. The workaround itself takes less than a minute to run. What takes time without preparation is the decision phase: is this safe? Do my volumes really have replicas elsewhere? Is the timing right?
A runbook answers those questions before they get asked under pressure. It also documents the trigger thresholds (“how many GiB before we intervene?”) and the failure criteria (“if the volume does not return to Healthy within 60 seconds, escalate to X”). The next person to read that runbook — or you yourself in six months — gets the same guarantees as today.
References
- Upstream issue: longhorn/longhorn#12573 — Very high memory consumption in longhorn-instance-manager (Severity/1, v1.11.0)
- Fix backported into v1.11.1, shipped in v1.11.2
- Internal runbook:
docs/runbooks/longhorn-instance-manager-bloat.md(marked resolved as of 2026-05-18)