The finding: “Dashboard” from the wrong container
On 24 September 2026, I checked something I had taken for granted. The admin console of a small Python web app is published behind a Cloudflare tunnel, and its host is protected by Cloudflare Access: mandatory login screen, a policy restricted to my account. From the internet, there is no way in without a badge.
From the inside, it is another story. The application pod groups several containers, including the public part, the one that talks to the internet all day long. From that container, a request to the pod’s local interface:
curl -s http://127.0.0.1:<port>/
returns 200, with the title “Dashboard”. No badge, no login, nothing. And the neighbouring container is nothing special: any other pod in the cluster that can reach the Service gets the same answer.
The sentence that sums up the problem still sits at the top of the module that fixed it: Access protects a URL, not a port. The admin console had no authentication of its own. It delegated everything to the network edge, and the network edge only sees what goes through it.
The threat model: everything that does not go through Cloudflare
A Cloudflare tunnel combined with Access is a very good front door. The problem is the other doors:
- The neighbouring container. Containers in a pod share the same network interface. If the public part is compromised, the intruder is one
curlaway from the admin console. - The rest of the cluster. Without a restrictive NetworkPolicy, any pod can reach any Service. A flaw in another homelab application is enough.
- Any route that bypasses the tunnel. An Ingress reachable from the LAN, a forgotten
port-forward, a future routing change: Access sees none of it. - An Access misconfiguration. A policy that is too broad, or an Access application aimed at the wrong target. More on that below, because that is exactly what happened.
The textbook answer is defence in depth: the application does not trust the path, it demands proof that the request went through Access, and checks that proof itself.
The badge: a JWT the app can verify on its own
On every request it lets through, Access adds a Cf-Access-Jwt-Assertion header. It is a JWT signed with RS256 by the Zero Trust team’s private key. The matching public keys are served at https://<team>.cloudflareaccess.com/cdn-cgi/access/certs. Among other things, the token carries an audience (aud, the identifier of the Access application), an issuer (iss, the team domain) and an expiry (exp).
The important point: verifying this badge requires no secret. The team domain and the AUD are public values. They go into the manifest in plain text, unsealed.
/healthz échappe au contrôle,
pour les sondes du kubelet ; les WebSocket y passent comme le reste.
Access protects a URL, not a port. Only /healthz skips the check,
for the kubelet probes; WebSockets go through it like everything else.
All the cryptography is pyjwt’s (with the [crypto] extra, which pulls in cryptography); nothing is hand-written. The core of the check fits in a few lines:
def verifier(self, jeton: str) -> dict:
if not jeton:
raise AccesRefuse("en-tête absent")
try:
entete = jwt.get_unverified_header(jeton)
except Exception as exc:
raise AccesRefuse("jeton illisible") from exc
# Refusé AVANT de chercher la clé : un `alg: none` ou un HS256 à `kid`
# inventé ne doit même pas coûter un rechargement.
if entete.get("alg") != "RS256":
raise AccesRefuse("signature invalide")
cle = self._cle(entete.get("kid"))
try:
return jwt.decode(jeton, cle, algorithms=["RS256"],
audience=list(self.audiences),
issuer=self.domaine, leeway=60,
options={"require": ["exp", "iat", "iss", "aud"]})
except jwt.ExpiredSignatureError as exc:
raise AccesRefuse("jeton expiré") from exc
except jwt.InvalidAudienceError as exc:
raise AccesRefuse("audience inattendue") from exc
# … émetteur, signature, puis un `except Exception` final
(The codebase is in French: jeton is the token, AccesRefuse the refusal, cle the key.) A few choices matter here. The algorithms=["RS256"] list is what closes the algorithm-confusion hole (an HS256 token signed using the public key as a shared secret). The four claims are required, not merely checked when present. Clock leeway is 60 s. And every failure becomes an AccesRefuse with a short reason: no other exception can bubble up, or it would turn into a 500. Only that reason reaches the logs; never the token, the e-mail address or any claim.
The middleware: everything except the probe
The check is wired into a pure ASGI middleware, added last in create_app and therefore the outermost: it runs before everything else, static files and previews included.
async def __call__(self, scope, receive, send) -> None:
if scope["type"] == "lifespan" or (scope["type"] == "http"
and scope["path"] == CHEMIN_SONDE):
await self.app(scope, receive, send)
return
try:
await run_in_threadpool(self.verificateur.verifier,
Headers(scope=scope).get(EN_TETE, ""))
except AccesRefuse as refus:
...
A single exemption, /healthz, matched exactly: kubelet probes carry no badge. verifier is synchronous (loading the keys makes a blocking HTTP call), hence the run_in_threadpool. The middleware is only mounted on the admin app; the public part and the showcase site are left alone.
The rollout: log first, refuse later
Putting a 403 in front of a console you use yourself means risking locking yourself out over a mistyped AUD. Hence two modes, driven by one variable:
journal(log): an invalid badge is logged and the request still goes through. A valid badge is announced (“badge Cloudflare Access valide”).strict(the default): missing or invalid badge, 403.
On the GitOps side, three variables on the admin container only:
- name: ADMIN_ACCES_DOMAINE
value: https://<team>.cloudflareaccess.com
- name: ADMIN_ACCES_AUDIENCES
value: <aud-de-64-caractères-hexadécimaux>
- name: ADMIN_ACCES_MODE
value: journal
The timeline, as both repositories tell it:
| When | What |
|---|---|
| 30.09, 22:15 | Manifest: the three variables, in journal mode |
| 30.09, 22:18 – 22:26 | pyjwt dependency, verification module, tests, docs |
| 30.09, 22:49 – 23:19 | Fixes from a security review (two passes) |
| 01.10, 10:10 | AUD replaced with that of a dedicated Access application |
| 01.10, 10:26 – 10:27 | Merge on the GitOps side, then on the application side |
| 01.10, 10:55 | Switch to strict |
The merge order is deliberate: the new image refuses to start in a pod without these variables (more on that below). The manifest therefore had to carry them before the image arrived. Log mode lasted less than half an hour: long enough to see a real badge accepted in the logs. The manifest comment is blunt about it: go back to journal only to diagnose, never for long, since that mode lets the intruder in while flagging them.
The audience trap: a valid badge, but for whom?
The first AUD set in the manifest had been read from the login redirect. It belonged to the generic Access application, a wildcard covering every subdomain of the homelab. Grafana, n8n, the public part of the app: they all share that application, and therefore that AUD.
The badge does not say which host it came through. It says: “signed by the team, for application X, valid until such time”. If the admin console accepts the generic application’s AUD, a badge captured on any of those services is accepted by the admin console. And the public part receives a badge every time I visit it. A compromised public container only had to keep it and replay it against the admin for the whole length of the Access session.
In other words, the signature check was flawless and did not protect against the very scenario that motivated it. On 1 October, an Access application dedicated to the admin host alone was created, and its own AUD replaced the old one. A manifest test checks that the value is a well-formed AUD (64 hexadecimal characters), but no test can tell whether it is dedicated: that is spelled out in the README and in the manifest comment.
WebSockets: the route that did not exist yet
The first version of the middleware let through anything that was not an HTTP request. No WebSocket route existed, so there was no visible hole. But the day a live-tracking page appeared, it would be born open to the whole pod.
Now only lifespan goes through unchecked. A websocket scope is verified with the same header. In strict mode, a failure closes the connection with code 1008 (“policy violation”) before accept, which the server turns into a 403; in log mode, a warning and the connection proceeds. The /healthz exemption applies to HTTP only.
The logs: when the URL is the secret
The admin app also mounts the ordering flow, and several of its URLs are bearer secrets: a token in the path grants access to an order. Two places were writing them down.
The first is the middleware’s own refusal message. The logged path is cut down to its first segment (/a/…) and passed through %r to neutralise line breaks: an intruder replaying a stolen token must not get it written into the logs.
The second, and bigger one, is uvicorn’s access log, which writes every request with its full path, query string included. Those lines are shipped by Vector to VictoriaLogs and kept for 8 days. A logging filter rewrites them:
_SUITE_JETON = re.compile(r"[A-Za-z0-9_-]{16,}")
def chemin_sans_jeton(chemin: str) -> str:
return _SUITE_JETON.sub("…", chemin.split("?", 1)[0])
Tokens are HMAC-SHA256 in base64url, i.e. 43 characters. Any run of 16 or more characters from that alphabet, anywhere in the path, becomes …; the query string is dropped. The first version only masked whole segments; the second review pass showed that a token glued to a foreign character (a parenthesis, a full stop, a %20, a zero-width space from a link copied out of a sentence) went through in full. Accepted side effect: a somewhat long site address gets masked too. The filter never raises, never drops a line, and it is installed on all three applications in the repository, not just the admin one.
The public keys: a cache that must neither lie nor block
Fetching the keys on every request would be absurd. The cache has four constants:
DUREE_CACHE = 3600 # clés gardées une heure
INTERVALLE_RECHARGEMENT = 60 # un rechargement par minute au plus
CONSERVATION_MAX = 24 * 3600 # clés périmées servies 24 h au plus
ATTENTE_PREMIER_CHARGEMENT = 6 # au-dessus du délai d'httpx (5 s)
That is: keys cached for one hour, at most one reload per minute, stale keys served for at most 24 hours, and a 6-second wait on first load. Keys are loaded lazily, on the first request: an unreachable Cloudflare never prevents startup. An unknown kid (key rotation) triggers a reload, but never more than one attempt per minute, successful or not: an attacker forging tokens with made-up kid values must not be able to make the admin app hammer Cloudflare.
The security review then found three flaws, fixed the same evening.
Stale keys forever. If a reload failed, the known key set kept serving, with no limit. Cloudflare rotates its keys with a long overlap, so that was convenient; but a key revoked after a leak stayed valid for as long as the pod could not reach Cloudflare. Now: at most 24 hours beyond the cache hour, then “public keys unreachable”.
A lock held during the network call. One slow load blocked every other request. The rule became: decide under the lock, load outside it, swap under it. During a reload, other threads serve the existing cache without waiting. Along the way, /healthz became an async def: the probe, which has a one-second timeout, no longer waits for a free thread from the pool it shares with badge verification.
The very first load. With no cache at all, a thread arriving during the first load was refused immediately with “public keys unreachable”, while Cloudflare was answering just fine. Two tabs opened right after a restart thus produced a 403 and a misleading network lead. Now those threads wait for the load to finish (a threading.Event, 6 s at most) and then verify with the keys obtained. The wait is bounded: no thread piles up indefinitely behind Cloudflare.
Fail-closed: no badge, no startup
Locally, the admin app must run without any configuration. In production, it must never run without verification. The configuration encodes both:
- domain and audiences go together: one without the other means refusing to start (a half-done configuration is not an admin that is “somewhat” protected, it is a typo that silently opens it);
- the domain must start with
https://and loses its trailing slash, since it is compared for equality with the issuer; - both empty inside a pod: refuse to start.
elif os.environ.get("KUBERNETES_SERVICE_HOST"):
raise RuntimeError(
"administration SANS vérification du badge Cloudflare Access "
"dans un pod Kubernetes ... : refus de démarrer.")
Kubernetes always sets KUBERNETES_SERVICE_HOST in a pod. The scenario in mind: a manifest that gets reorganised, synced by ArgoCD without the application’s CI running, and loses the three variables. Rather than an admin app restarted wide open, an admin app that goes down. Outside a pod, just a warning at startup.
The flip side showed up immediately: my code editor itself runs in the cluster, so the variable is set there and an admin app started from its terminal refuses to start. The README documents the workaround, scoped to that single process: env -u KUBERNETES_SERVICE_HOST uv run uvicorn …. The pytest configuration removes the variable before importing the application, for the same reason: a test suite run from inside the cluster is not a deployment.
The dependency: a floor for a CVE
The dependency reads pyjwt[crypto]>=2.10.1, and the floor is not decorative. CVE-2024-53861 affects pyjwt 2.10.0: when issuer is passed as a string, the issuer is compared as a substring. That is precisely the form used here (issuer=self.domaine). The locked version (2.15.1) was not affected; the floor simply prevents ever sliding back to it, through a regenerated lock file or an unlucky resolution.
The tests: pinning down who carries what
The verification module has 55 test functions, the log filter 10. They cover the classics (alg: none, HS256 confusion with the public key, foreign audience and issuer, expired token, one minute of leeway), the cache (one hour, one reload per minute, 24-hour retention, wait bounded at 6 s), the 1008 close for WebSockets and the refusal to start inside a pod.
Two tests run against the GitOps repository’s manifests, and those are the ones I find most useful:
def test_seule_l_admin_porte_les_reglages_du_badge():
for conteneur in _containers():
if conteneur["name"] == "admin":
continue
noms = {e["name"] for e in conteneur.get("env", [])}
assert not noms & set(VARIABLES_ACCES)
The first requires the admin container to carry all three variables, with a well-formed AUD and a known mode. The second forbids those variables on the other containers: giving them a badge to check would protect nothing and only muddy the manifest. The deployment checks thus go from 41 to 43.
What I keep
- Access protects a URL, not a port. Anything that reaches the Service without going through Cloudflare (neighbouring container, another pod, a forgotten route) bypasses Access. A sensitive application checks the badge itself.
- Check the audience, and make it specific. A valid signature only proves the badge comes from the team. An AUD shared by several hosts makes a badge captured elsewhere replayable on the admin app: one Access application per sensitive host.
- Log mode, then strict, quickly. Log mode is for seeing real badges accepted, not for living with.
- Every protocol. WebSockets require the badge, even when no WebSocket route exists yet.
- Keep tokens out of the logs. If a URL is a secret, the access log is a leak; mask by pattern, not by segment.
- A bounded, non-blocking cache. Stale keys for 24 hours at most, no lock held during a network call, a bounded wait on first load.
- Fail closed where it matters. Inside a pod, no configuration means no startup; locally, a warning.
- A version floor when a CVE hits exactly the call you make.
The limit is clear: none of this replaces a NetworkPolicy that would stop the rest of the cluster from reaching the admin app. It is one more layer, not the only one. But since 1 October, the side entrance answers 403, and so does a badge borrowed from another service.