In early June, the blog becomes unreachable every other time. HTTP/2 502, server: cloudflare header, and the great classic among maddening bugs: “it works again after a refresh”. Then it comes back. Always.
This post documents four days of plausible dead ends — each of them with evidence that seemed to incriminate it — before the discovery of a root cause that was in none of my hypotheses. The title spoils it, but the road there is worth the detour.
The symptom, precisely
Methodical test: 20 requests per service, from the internet, through the tunnel.
| Service | Result | |
|---|---|---|
| argocd, code, portainer, claude | 20/20 OK | ✅ |
| shadowbroker (the initial suspect) | 20/20 OK | ✅ |
| grafana, homepage, n8n | OK | ✅ |
| blog | 8/20 OK — 12× 502 | ❌ |
The crucial detail that led me astray for a long time: the blog goes through exactly the same route as the healthy services (wildcard *.olig-homelab-2025.win → traefik.kube-system:80). Same tunnel, same Traefik, same path. So why this one?
The “302 = OK” trap
First diagnostic lesson: most of my services sit behind Cloudflare Access. An unauthenticated curl gets a 302 to the login page — served by the Cloudflare edge, without ever touching the connector or the origin. All my “20/20 OK” results on protected services were testing only the edge.
The blog is my only genuinely public service. It wasn’t specifically broken — it was the only one revealing the problem, because it was the only one whose requests crossed the tunnel all the way through. If you are diagnosing a Cloudflare tunnel with services behind Access: only public endpoints exercise the full path.
Dead ends (all plausible, all wrong)
The memory-hungry app. A backend OOMKilled three days earlier on the same worker as the tunnel pods. Verification: 20/20 through the tunnel, no eviction of other pods, 10 Gi free on the node. Cleared.
QUIC. The cloudflared manifest carried an inherited comment: “QUIC unstable on this network”. Reverse test — switch to QUIC with UDP buffers raised to 7.5 MB on all 6 nodes (sysctl DaemonSet): the connections register (protocol=quic, 4 conns/pod), but the edge routes no request at all. total_requests=0, 100% 502s. The UDP 7844 data path is dead somewhere inside the Swisscom box. Rollback to http2 — but the intermittent 502s themselves persist.
http2 connection ageing. A seductive theory: the box’s NAT silently kills idle TCP connections, the edge routes towards a dead connection, restarting the pods repairs everything… until the next time. I even deployed a CronJob doing a periodic rollout every 6 h. The restart did repair things — a clue that seemed to confirm the theory. In reality it was only reshuffling the deck, and sometimes it made things worse: after certain restarts, 100% 502s.
That last point should have tipped me off: if restarting my pods worsens the problem, then the edge prefers routing towards something more stable than my pods. Something else.
The root cause: a third connector
Cloudflare dashboard → homelab-k3s tunnel → Connectors tab:
- 2× k8s pods
cloudflared-*—linux_arm64, v2026.5.0 — healthy - 1×
picokvm—linux_arm32-bit, v2026.5.2 — and what exactly are you doing here?
A cloudflared installed on my PiKVM months earlier, wired by mistake to the same tunnel as the cluster, and forgotten.
cloudflared oublié sur le PiKVM reçoit sa part du trafic mais ne peut pas joindre
traefik.kube-system.svc : 502 intermittent, qui « remarche après refresh »
(retry sur un pod sain). Fix : un tunnel = un environnement. To the edge, a tunnel's connectors are interchangeable — it routes to any of them.
The cloudflared forgotten on the PiKVM gets its share of the traffic but can't reach
traefik.kube-system.svc : intermittent 502s, which "work again after a refresh"
(retry lands on a healthy pod). Fix : one tunnel = one environment. The mental model I was missing: inside a Cloudflare tunnel, connectors are interchangeable. The edge routes any request towards any registered connector — that is the very principle behind the product’s high availability. When a request landed on picokvm, it tried to resolve traefik.kube-system.svc.cluster.local… from a device that is not in the cluster. Failure, 502, “try again”.
Everything falls into place at once:
- ~⅓ to ½ of requests in 502: the share of traffic routed to the phantom connector
- “It works after a refresh”: the retry lands on a healthy pod
- Restarts made it worse: while my 2 pods were restarting, the edge shifted everything to the only stable connector left — the phantom. 100% 502s.
- Nothing in my logs: the offending requests never reached my pods
Fix, on the PiKVM:
systemctl stop cloudflared && systemctl disable cloudflared
The 502s disappeared instantly. The PiKVM lost nothing in the process — its pico.olig-homelab-2025.win route was already served by the k8s connectors towards its LAN IP.
The lessons
1. One tunnel = one environment. If a device needs its own cloudflared, it needs its own tunnel. Mixing connectors that don’t see the same origins on a single tunnel is a time bomb — the product is designed to treat them as equivalent.
2. The Connectors page first. Four days diagnosing transport, NAT and buffers, while the connector list — three entries, one of which had no business being there — was two clicks away. For any “intermittent 502 emitted by the edge” symptom, the connector inventory is the first reflex, not the last.
3. “A restart repairs it” doesn’t clear the thing you restarted. It is a clue about the ecosystem, not proof of local guilt. Here, the restart wasn’t repairing my pods — it was forcing the edge to redistribute traffic.
4. Document the devices that touch the infrastructure. The PiKVM’s cloudflared appeared in no manifest, no repo, no documentation — installed by hand, one evening, “just to test”. Anything that registers against a shared service must leave a written trace. Ghosts, by definition, are the ones you have forgotten.
For the details of the tunnel + Traefik configuration behind this routing, see Traefik & Cloudflare.