❯ infra · Running

Donner un vrai MemoryPressure à k3s : eviction-hard et réserves système en rolling sur 6 nœuds hétérogènes Giving k3s a Real MemoryPressure: Eviction Thresholds and System Reserves, Rolled Out Across Six Mismatched Nodes

Par défaut, K3s ne pose aucun seuil d'éviction mémoire et ne réserve rien au système : MemoryPressure ne peut jamais se déclencher et c'est l'OOM-killer du noyau qui arbitre. L'audit, les trois profils de nœuds, le rolling avec canari — et le chmod récursif qui a cassé le SSH de Gitea au passage. Out of the box, K3s sets no memory eviction threshold and reserves nothing for the system, so MemoryPressure can never fire and the kernel OOM killer has the final word. Here is the audit, the three node profiles, the canary-first rolling rollout — and the recursive chmod that quietly broke Gitea's SSH along the way.

· 10 min de lecture 10 min read ·
k3s-clusterk8srpiinfraincidentgitea
🇫🇷 Lire cet article en français

The finding: a node that cannot go hungry

On 17 September 2026, I run a routine audit of the kubelet configuration on the cluster’s six nodes. No outage, no alert: just a question that had been lingering since the Longhorn instance-manager incident. Back then, a pod had grown to 12 GiB without Kubernetes lifting a finger. Why had no node ever reported MemoryPressure?

The cluster is heterogeneous on purpose: three Raspberry Pi 4 masters with 8 GB, two Pi 5 workers with 16 GB, and a small Pi 4 worker with 2 GB that mostly runs DaemonSets. One kubelet configuration for machines whose RAM varies by a factor of eight is already suspicious.

The configuration a kubelet has actually loaded is not read from a file: you query it through the API, via the node proxy.

kubectl get --raw "/api/v1/nodes/<node>/proxy/configz" \
  | jq '.kubeletconfig | {evictionHard, mergeDefaultEvictionSettings,
        systemReserved, kubeReserved, enforceNodeAllocatable}'

In substance, the answer is the same on all six nodes:

{
  "evictionHard": { "imagefs.available": "5%", "nodefs.available": "5%" },
  "mergeDefaultEvictionSettings": false,
  "systemReserved": null,
  "kubeReserved": null,
  "enforceNodeAllocatable": ["pods"]
}

Three lines explain everything.

  1. No memory threshold. K3s defines its own eviction thresholds, for disk only (5% on imagefs and nodefs).
  2. The Kubernetes default is discarded. Upstream, the kubelet watches memory.available<100Mi. With mergeDefaultEvictionSettings=false, the K3s list replaces the Kubernetes one instead of being added to it, so the memory threshold disappears.
  3. Nothing is reserved. systemReserved and kubeReserved are null, so the scheduler believes all the RAM is its own.

Checking the node confirms it:

kubectl get node <node> -o jsonpath='{.status.capacity.memory} {.status.allocatable.memory}'
# 8007324Ki 8007324Ki

allocatable == capacity, down to the kilobyte. On a master this is even more awkward, because etcd and the k3s server process run outside pods: the scheduler hands out memory they already occupy.

The theory: why MemoryPressure could never become true

A node’s MemoryPressure condition is not a measurement, it is a verdict. The kubelet computes memory.available (capacity minus the node’s working set) and compares it with its eviction thresholds. Crossing an eviction-hard threshold sets the condition to True, adds a NoSchedule taint and starts an ordered eviction. Pods using more than their requests go first, typically Burstable ones, with priority taken into account.

Without a memory threshold, there is nothing to cross. The condition stays False until the last free page, and the kernel OOM killer decides, by oom_score. It knows nothing about QoS classes, requests or how important a pod is. On a worker carrying Longhorn replicas, killing the wrong process mid-write is more than an inconvenience.

The formula the kubelet applies once it is finally given numbers:

allocatable = capacity − kube-reserved − system-reserved − eviction-hard

The system reserve makes room for what lives outside pods. The eviction threshold keeps a margin between “the scheduler has handed everything out” and “the kernel starts killing”.

The settings: three profiles for six nodes

I go with three profiles rather than a single setting: a 200 Mi threshold does not mean the same thing on 16 GB as on 2 GB.

Profilememory eviction-hardminimum-reclaimsystem-reservedallocatable before → after
3 Pi 4 8 GB masters<200Mi100Mi512Mi8007324Ki → 7278236Ki (−712 Mi)
2 Pi 5 16 GB workers<200Mi100Mi256Mi−456 Mi
1 Pi 4 2 GB worker<100Mi50Mi128Mi1888960Ki → 1655488Ki (−228 Mi)

The masters get the largest reserve, 512 Mi, because etcd and the k3s server live there outside the pods cgroup. The Pi 5 workers only need 256 Mi for k3s-agent and systemd. The small worker’s real capacity is only 1.80 Gi, so everything is halved.

eviction-minimum-reclaim deserves a line of explanation. When an eviction fires, the kubelet does not stop right at the threshold: it reclaims at least this extra amount. Without it, a node hovering around 200 Mi would evict a pod, climb back to 201 Mi, then start again ten seconds later.

The trap: eviction-hard replaces, it does not add

K3s natively reads /etc/rancher/k3s/config.yaml, on the server and the agent alike, without touching the systemd units. The directory did not even exist: this is the first config.yaml in the cluster. Until then, everything ran on K3s defaults.

Here is the master profile:

# /etc/rancher/k3s/config.yaml — master profile (Pi 4 8 GB)
kubelet-arg:
  - "eviction-hard=memory.available<200Mi,imagefs.available<5%,nodefs.available<5%"
  - "eviction-minimum-reclaim=memory.available=100Mi,imagefs.available=10%,nodefs.available=10%"
  - "system-reserved=memory=512Mi"

The trap is in the first line. Passing eviction-hard as a kubelet-arg replaces the whole K3s default list. The naive version, eviction-hard=memory.available<200Mi on its own, adds memory protection and removes both disk thresholds at the same time. You gain one safety net and lose two, without a single warning. So imagefs.available<5% and nodefs.available<5% have to be listed again explicitly. The same logic applies to eviction-minimum-reclaim, where I put the 10% disk values back.

Second deliberate choice: system-reserved stays accounting-only. enforceNodeAllocatable keeps its ["pods"] value, and I do not add --system-reserved-cgroup. System processes are not locked into a capped cgroup, so there is no risk of etcd being killed for exceeding 512 Mi. The reserve exists so the scheduler stops handing out memory that does not exist. It is the recommended mode, and the only one I want to explain at 2 a.m.

The rollout: canary, masters, then workers

Changing the kubelet configuration means restarting k3s (or k3s-agent) on each node. containerd is not touched: containers keep running, and the node goes NotReady for roughly 10 to 30 seconds while the kubelet comes back. On a cluster with a three-member etcd and replicated volumes, order still matters.

  1. Canary: the small 2 GB Pi 4 worker. It holds no Longhorn replica, only DaemonSets. If it refuses to restart, nothing important goes down. I check two things: that the syntax is accepted (the kubelet starts) and that allocatable matches the calculation, 1655488Ki.

  2. The three masters, one at a time. Between nodes, a strict gate: 6/6 nodes Ready, and etcd healthy as seen by the API server.

    kubectl get --raw '/readyz?verbose' | grep etcd
# [+]etcd ok
# [+]etcd-readiness ok

    Restarting two masters at once is a coin toss with etcd quorum. One at a time, with this check in between, quorum is never at risk.

  3. The two Pi 5 workers, one after the other. Here the gate is about storage: node Ready, the node’s pods Running, and every Longhorn volume healthy before moving on to the second one.

    kubectl -n longhorn-system get volumes.longhorn.io \
  -o custom-columns=NAME:.metadata.name,ROBUSTNESS:.status.robustness

Per-node rollback is a one-liner: delete the file and restart the service.

sudo rm /etc/rancher/k3s/config.yaml && sudo systemctl restart k3s        # master
sudo rm /etc/rancher/k3s/config.yaml && sudo systemctl restart k3s-agent  # worker

It was not needed. Outcome: zero pods restarted, all six allocatables matching the calculation, and no pod left unschedulable. The busiest worker still keeps 6.8 Gi of headroom between its total requests and its new allocatable. Rollout done, all green… almost.

The incident: Permission denied (publickey), with 20 days of uptime

Shortly after the kubelet restart on the first Pi 5 worker, git pushes to Gitea start failing. So does CI, for the same reason:

git@<gitea-host>: Permission denied (publickey).
fatal: Could not read from remote repository.

The link with the rollout is far from obvious. The Gitea pod has not restarted, and neither has sshd: 20 days of uptime each. The key has not changed, and the web UI responds. Yet every key is refused.

The sshd logs inside the pod give the real lead: “bad ownership or modes”. With StrictModes enabled by default, OpenSSH refuses to read an authorized_keys file that anyone other than its owner can modify. It checks the parent directories too. The permissions on the volume:

  • /data/git/.ssh/authorized_keys at 660 instead of 600;
  • the /data/git home at 2775, group-writable with the setgid bit.

Someone ran a chmod -R g+w on the PVC, and that someone was the kubelet. The Gitea Deployment declares fsGroup: 1000 without setting fsGroupChangePolicy, so it uses the default Always policy. On every (re)attach, the kubelet’s volume manager then walks the entire volume to enforce the group and group write permission. When the kubelet restarted and rebuilt the state of mounted volumes, it re-applied that fsGroup recursively. For most applications this is harmless. For an sshd running with StrictModes, it amounts to locking the door.

The live fix is immediate, straight from inside the pod:

chmod 700 /data/git/.ssh
chmod 600 /data/git/.ssh/authorized_keys
chmod g-w /data/git

Pushes work again right away. What remains is making the fix durable, otherwise the next kubelet restart will replay exactly the same scene.

The durable fix: OnRootMismatch and an idempotent init container

Two changes in the Gitea manifest:

      initContainers:
      - name: fix-ssh-permissions
        image: busybox
        command: ['sh', '-c', 'mkdir -p /data/git/.ssh && chown -R 1000:1000 /data/git/.ssh && chmod 700 /data/git/.ssh && { [ ! -f /data/git/.ssh/authorized_keys ] || chmod 600 /data/git/.ssh/authorized_keys; } && chmod g-w /data/git']
        securityContext:
          runAsUser: 0
      # ...
      securityContext:
        fsGroup: 1000
        fsGroupChangePolicy: OnRootMismatch
  • fsGroupChangePolicy: OnRootMismatch: the kubelet only walks the volume if its root does not already have the right group. Once the volume is in place, there is no more recursive chmod on every attach. As a bonus, mounts are faster on a large volume.
  • The hardened init container already existed to create .ssh. It now also enforces 600 on authorized_keys (if present) and removes g+w from the home. It is idempotent: on every start, it restores the state sshd expects, whatever caused the drift.

The Deployment uses the Recreate strategy, so applying the fix costs about a minute of git and web downtime, accepted.

Lessons learned

  1. Audit configz, not your memory. /api/v1/nodes/<node>/proxy/configz shows what the kubelet actually loaded. Here, it revealed a cluster that had run without a memory safety net since day one. Also compare allocatable with capacity: if they are equal, nothing is reserved.
  2. On K3s, eviction-hard replaces the default list. Always include imagefs.available and nodefs.available again, and do the same for eviction-minimum-reclaim. The trap is silent: the kubelet starts without complaint, with one less disk protection.
  3. Size per machine profile. A threshold and reserve designed for 16 GB are oversized on 2 GB, and the reverse is too tight. Three profiles for six nodes means three files to maintain, not a factory.
  4. An accounting-only reserve is enough to start with. Keeping enforceNodeAllocatable=["pods"] gives the scheduler an honest view without risking etcd or systemd being killed by a badly calibrated cgroup cap.
  5. Order the rollout by risk, with a measurable gate. First the node that holds nothing, then the masters with etcd ok, then the storage nodes with Longhorn healthy. Have the one-line rollback ready before you start.
  6. A kubelet restart is not neutral for volumes. With fsGroup and the default Always policy, it can re-apply a recursive chmod on volumes that are already mounted. Anything that checks its own permissions (sshd, some databases, TLS keys) deserves fsGroupChangePolicy: OnRootMismatch and an init container that restores the expected state.
  7. Be wary of “20 days of uptime”. An intact process is no proof that its environment is intact. The symptom was on the SSH side, the cause on the kubelet side, and the trigger was a change that had nothing to do with Gitea.

Since then, memory saturation produces what it should have produced from the start: MemoryPressure=True, a taint, and an eviction ordered by QoS. The OOM killer is still in the kernel, but it is no longer the first to step in.

k3s-clusterk8srpiinfraincidentgitea