❯ infra · Running

Zéro secret en clair dans Git avec Sealed Secrets Zero-Secret-in-Git with Sealed Secrets

Chiffrement des secrets Kubernetes pour un stockage GitOps sécurisé via Bitnami Sealed Secrets — fonctionnement, workflow kubeseal et considérations opérationnelles. Encrypting Kubernetes secrets for safe GitOps storage using Bitnami Sealed Secrets — how it works, the kubeseal workflow, and key operational considerations.

· 8 min de lecture 8 min read ·
k3s-clustersecuritysecretsgitopsk8s
🇫🇷 Lire cet article en français

The fundamental problem of GitOps and secrets

A rebuilt cluster, a lost Sealed Secrets private key: every secret versioned in Git becomes unreadable in one stroke. That is the kind of scenario that forces you to really understand how secret encryption works in GitOps.

GitOps requires the entire state of the cluster to be versioned in Git. But Kubernetes secrets (passwords, tokens, API keys) obviously cannot be committed in cleartext. A base64-encoded Kubernetes Secret is worth nothing — base64 is not encryption.

The alternatives:

  • Vault (HashiCorp): excellent, but oversized for a homelab. High operational complexity.
  • External Secrets: pulls secrets from external stores (AWS SSM, GCP Secret Manager). Requires an external service.
  • Sealed Secrets: asymmetric encryption, a controller inside the cluster, zero external dependency.

Sealed Secrets is the natural choice for a self-contained homelab.

How it works

Sealed Secrets rests on an asymmetric key pair:

  1. The controller runs in the cluster and holds the private key. It can decrypt SealedSecrets.
  2. kubeseal (CLI) fetches the controller’s public key to encrypt a Secret into a SealedSecret.

The encrypted SealedSecret is perfectly safe to commit to Git — without the controller’s private key, it is unusable.

Secret YAML (cleartext)
        │
        ▼  kubeseal --cert <public key>
SealedSecret YAML (encrypted) ──► Git ──► ArgoCD ──► Cluster
                                                        │
                                            Controller decrypts
                                                        │
                                                        ▼
                                             Kubernetes Secret (cleartext)

Installation

The Sealed Secrets controller is installed from a static manifest, in the kube-system namespace:

kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml

It matters that the controller is named sealed-secrets-controller in kube-system — that is the default name kubeseal looks for.

Workflow: creating a SealedSecret

1. Prepare the source Secret

kubectl create secret generic gitea-credentials \
  --namespace gitea \
  --from-literal=DB_PASSWORD='$(openssl rand -base64 32)' \
  --dry-run=client \
  -o yaml > /tmp/secret.yaml

The --dry-run=client -o yaml option generates the YAML without creating the resource in the cluster.

2. Encrypt with kubeseal

kubeseal \
  --controller-name sealed-secrets-controller \
  --controller-namespace kube-system \
  --format yaml \
  < /tmp/secret.yaml \
  > gitea/sealedsecret-credentials.yaml

3. Clean up the cleartext secret

rm /tmp/secret.yaml  # Never commit this file

4. Commit the SealedSecret

git add gitea/sealedsecret-credentials.yaml
git commit -m "feat(gitea): add sealed credentials"
git push

ArgoCD syncs the SealedSecret, the controller decrypts it, and the Kubernetes Secret shows up in the gitea namespace.

Anatomy of a SealedSecret

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: gitea-credentials
  namespace: gitea
spec:
  encryptedData:
    DB_PASSWORD: AgB3x9K2m...  # RSA-OAEP encrypted
  template:
    metadata:
      name: gitea-credentials
      namespace: gitea
    type: Opaque

The encryptedData field holds the encrypted value for each key. The template defines the Secret the controller will create.

Encryption scope

By default, kubeseal encrypts with the strict scope: the SealedSecret can be decrypted only in the specified namespace, and only for the specified name. There is no way to decrypt a gitea/gitea-credentials SealedSecret in a different namespace.

That is the desired behaviour — strong isolation between namespaces.

Sealing offline with the public key

The kubeseal workflow above assumes that the sealing machine has access to the cluster (to fetch the public key through kubectl). Yet there are plenty of cases where you want to seal without that access — from a GitHub Actions runner, from a dev environment that does not carry the cluster’s kubectl config, or simply while pair-programming with a colleague who is not an admin.

The way around it: commit the controller’s public key into the GitOps repo (docs/sealed-secrets/sealed-secrets-pub.pem in my case). It is a public key — disclosing it is risk-free by construction. With it, anyone can encrypt a secret for this cluster, but nobody can decrypt it apart from the controller that holds the private key.

Extracting the public key (done once, from a machine with kubectl):

kubeseal --fetch-cert \
  --controller-name sealed-secrets-controller \
  --controller-namespace kube-system \
  > docs/sealed-secrets/sealed-secrets-pub.pem
git add docs/sealed-secrets/sealed-secrets-pub.pem
git commit -m "docs: pin sealed-secrets public cert for offline sealing"

Usage from anywhere, with the repo cloned:

kubectl create secret docker-registry ghcr-pull \
  --docker-server=ghcr.io \
  --docker-username=YellowBeanie \
  --docker-password=<PAT> \
  --namespace=tapaz \
  --dry-run=client -o yaml \
| kubeseal --cert docs/sealed-secrets/sealed-secrets-pub.pem \
           --format yaml \
           --scope strict \
  > tapaz/sealedsecret-ghcr-pull.yaml

Not a single kubectl touching the cluster. No need for the workstation to hold a valid kubeconfig. The resulting SealedSecret is strictly equivalent to the one an online workflow would have sealed — same cert, same algorithm, same scope protection.

Rotation: if the controller regenerates its key (manual rotation or reinstallation), the committed .pem has to be refreshed. Otherwise new SealedSecrets keep working (the controller retains the old keys), but you are still encrypting with a key older than necessary — harmless, just not optimal.

Backing up the private key

The controller’s private key is stored in a Secret in kube-system:

kubectl get secret -n kube-system \
  -l sealedsecrets.bitnami.com/sealed-secrets-key \
  -o yaml > sealed-secrets-master-key-BACKUP.yaml

That backup is CRITICAL. If the cluster is rebuilt and the private key is lost, every existing SealedSecret is undecryptable. The backup has to be stored outside the cluster (password vault, encrypted storage).

Key rotation

Sealed Secrets automatically generates new encryption keys on a regular basis (every 30 days by default). Old SealedSecrets stay decryptable (the old keys are retained), but new ones are encrypted with the most recent key.

To force re-encryption of a SealedSecret with the current key:

kubeseal --re-encrypt < old-sealedsecret.yaml > new-sealedsecret.yaml

Lessons learned

Always specify --controller-namespace: if kubeseal cannot find the controller, it fails silently or uses the wrong one. Being explicit avoids surprises.

Version SealedSecrets alongside the manifests: keep sealedsecret-credentials.yaml in the same directory as the Deployment that consumes it. Logical organisation makes operations simpler.

Never regenerate an existing secret without updating the application: if a database’s DB_PASSWORD is regenerated, the database has to be updated at the same time. A changed SealedSecret without the matching update in the DB means an authentication error at the pod’s next restart.

Once secrets are encrypted and versioned in Git, the next step is to automate their deployment with ArgoCD.

k3s-clustersecuritysecretsgitopsk8s