The starting point: a command you type a hundred times
On 11 September 2026, an application is about to go to production for the first time. Its Deployment references thirteen keys through secretKeyRef: an LLM API key, SMTP credentials, a publishing token, a payment provider’s secrets, a hashing salt. The repository is clean: an exposure audit the day before had already pulled a token out of a git clone URL and hardened the error messages. All that is left is sealing.
The method documented everywhere, including in my own Sealed Secrets article, fits in two lines:
kubectl create secret generic app-secrets \
--from-literal=API_KEY='sk-…' --dry-run=client -o yaml > secret.yaml
kubeseal --format yaml < secret.yaml > sealedsecret.yaml && rm secret.yaml
It works. It also leaves three traces. The value is passed as an argument: it is visible in ps while the command runs, and it stays in the shell history. The secret.yaml file exists in plaintext on disk, and an rm erases nothing on an SSD. With thirteen keys, that is thirteen chances to get it wrong — not counting the day you forget the rm.
The principle: the value only lives in memory
The goal is easy to state: between the password vault and the Git repository, the value must never exist anywhere other than in a process’s memory or in an encrypted channel.
op://…/cleop://…/key
le dépôt ne contient que des référencesthe repo only holds references
op run -- python …op run -- python …
en clair, mais seulement en mémoireplaintext, but only in memory
tube, pas de fichierpipe, no file
rien dans ps, rien dans l’historiquenothing in ps, nothing in history
stdin → --format yamlstdin → --format yaml
chiffré pour la clé du contrôleurencrypted for the controller key
git commitgit commit
testé : chaque clé > 100 caractères de chiffrétested: every key > 100 chars of ciphertext
ps. Ce qui redescend du cluster est déjà chiffré.
The value only exists in plaintext in memory (steps ② and ③): never in a file,
never in an argument visible to ps. What comes back from the cluster is already encrypted.
Three tools are enough:
- The vault (here 1Password and its
opCLI). The repository only holds a file of references,API_KEY=op://Vault/App/api-key. That file can be committed: it says nothing. op run, which resolves the references and starts a process with the values in its environment. A process’s environment is only readable by its owner and root, and it disappears with it.- A small script that reads that environment and writes the Kubernetes
Secretto its standard output. Not to a file.
The output goes into an SSH pipe to a control-plane node, where kubeseal reads its standard input:
op run --env-file=refs.env -- python secret_pour_kubeseal.py app \
| ssh <admin>@<control-plane> \
"sudo -E KUBECONFIG=/etc/rancher/k3s/k3s.yaml kubeseal \
--controller-namespace kube-system --format yaml > /tmp/sealedsecret-app.yaml"
What lands on the node’s disk is already encrypted for the controller’s public key. So is what you bring back and commit. The plaintext form of the Secret only ever existed in a pipe buffer.
Here, kubeseal runs on the node, where the kubeconfig already lives. Offline sealing with --cert on the workstation works too; the principle is the same as long as the value comes in through stdin.
The script: forty lines and two traps
The core of the script is mundane: a tuple of expected keys, a base64 encoding, YAML written by hand.
CLES_APP = ("API_KEY", "SMTP_HOST", "SMTP_PORT", "SMTP_USER", "SMTP_PASSWORD", ...)
def secret_app(env: dict[str, str]) -> str:
valeurs = {c: env.get(c, "") for c in CLES_APP}
if not valeurs["CONTACT_HASH_SECRET"]:
valeurs["CONTACT_HASH_SECRET"] = secrets.token_hex(32)
# Quotes are mandatory: the base64 of an empty string is an empty string,
# which YAML would read as null.
lignes = [f' {cle}: "{_b64(val)}"' for cle, val in valeurs.items()]
return "apiVersion: v1\nkind: Secret\n...data:\n" + "\n".join(lignes) + "\n"
Both traps are in the comments.
Trap #1: an empty key is not a null key
Some keys are legitimately empty on the first deployment: the payment provider is not configured yet, and the feature that uses it stays disabled. The base64 of "" is "". Written without quotes, PAYMENT_SECRET: is read by YAML as null, and kubeseal seals a null value instead of an empty string. With quotes, PAYMENT_SECRET: "" stays a string. The difference only shows up when the pod starts.
Trap #2: a missing key blocks the pod
The temptation is to leave empty keys out. Bad idea: the Deployment references all of them, and a secretKeyRef to a key that does not exist gives a CreateContainerConfigError. The pod does not start, for a feature you are not even using yet. So the script always writes all thirteen keys, empty if need be. An empty key is a disabled setting, perfectly valid; a missing key is an outage. (The clean alternative, optional: true on the secretKeyRef, only makes sense if the application can genuinely live without it.)
A third point, subtler, concerns the CONTACT_HASH_SECRET salt. The script generates a random one if it is missing, which is convenient. But links sent by e-mail are derived from it via HMAC: resealing with a different salt invalidates every link already sent. After the first sealing, the generated value must therefore go back into the vault. The convenience of a generated value is paid for with a value you now have to keep.
The tests: the one that blocked the only correct state
The repository had tests on the manifests. Two of them checked that the committed SealedSecrets contained only the à sceller (“to be sealed”) marker: a reasonable guarantee as long as the files were only templates, to make sure no real value was lying around.
On sealing day, both tests went red. They were blocking exactly the only correct production state. The useful guarantee is the opposite: every key present, and every one encrypted.
def _chiffre(valeur) -> bool:
"""A sealed secret is RSA+AES ciphertext encoded in base64: several hundred
characters. Anything short, empty or carrying a template marker is either a
forgotten template or a plaintext value."""
return isinstance(valeur, str) and len(valeur) > 100 and "sceller" not in valeur
def test_sealedsecret_app_porte_les_treize_cles_toutes_chiffrees():
chiffrees = sealedsecret["spec"]["encryptedData"]
assert set(chiffrees) == CLES_SENSIBLES
assert not [c for c, v in chiffrees.items() if not _chiffre(v)]
The hundred-character threshold has nothing cryptographic about it. It catches the two real mistakes: a forgotten template, and a plaintext value pasted into encryptedData by accident (an API key is rarely longer than a hundred characters, a sealed ciphertext never shorter). The image registry secret has its own test: type kubernetes.io/dockerconfigjson, otherwise the kubelet ignores it and the pod stays in ErrImagePull.
The backlash: when the pipe itself damages the secret
On 9 October 2026, another application on the cluster (an e-mail sorting agent) needs four SealedSecrets, one of them an LLM API key. Same principle, different workstation: this time the sealing is run from PowerShell. Four incidents in half an hour.
The vanished line breaks. The SealedSecret is written with Out-File -NoNewline. The YAML loses its line breaks: the file is no longer a valid document. Fixed by restoring the line breaks, no resealing needed.
The BOM and the CRLF. The PowerShell pipe slipped a UTF-8 BOM in front of the value and a \r\n after it. The decrypted Secret therefore holds three invisible bytes at the start and a carriage return at the end. The application puts the key in an HTTP header: invalid header, every call fails.
The leak through the log. Worse: the exception raised (UnicodeEncodeError) copies the offending string into its message, and the pod log printed the full exception. The key thus ended up in plaintext in the pod log. This is no longer an outage, it is a leak. There is only one answer: the key has to be revoked and a new one issued.
The truncation. Resealed through the same pipe, the new key arrives six characters short. Resealed a third time from Git Bash, this time comparing a fingerprint of the value before the pipe and after decryption in the pod.
The application-side fixes are modest and final: every value read from a Secret is cleaned (BOM, whitespace and line endings stripped) before use, and errors are now logged only by their type and their first 120 characters. A secret must never be able to surface in an error message, even if it was carried badly.
What I keep
- No value as an argument, no plaintext file. Vault → environment → stdout →
kubesealstdin. If a step needs a file, that is the step to replace. - All keys, always. Empty rather than missing, and quoted so YAML does not turn them into
null. - Test the ciphertext, not the template. A test that requires a template marker blocks production on sealing day. Requiring long, opaque values covers both the forgotten template and the plaintext value.
- Verify by fingerprint. Compare a hash of the value before the pipe and after decryption in the pod (
kubectl exec … | sha256sum), rather than “it seems to work”. A shell that adds a BOM does not warn you. - Clean on input, redact on output. The application cleans what it reads from a Secret and never logs a raw exception that could contain a value.
- A generated value is a value to keep. A randomly drawn salt that links or tokens depend on must go back to the vault immediately.
The first sealing was done with a script and tests written for the occasion; it only broke one test. The second, with a less cooperative shell, cost a revoked key. Sealed Secrets’ mechanics were not at fault in either case: what breaks is carrying the value all the way to kubeseal.