Context: the move of the source of truth to a self-hosted Gitea is told in Migrating a GitOps homelab from GitHub to self-hosted Gitea. This incident is what happens three months later, when you forget that a migration doesn’t end on switchover day.
The symptom: two refs that don’t match
On August 13, 2026, after a routine push to the GitOps source of truth, a reflex check: did the backup actually receive the copy? The question sounds rhetorical. The push mirror is configured with sync_on_commit, plus a fallback every 8 hours. There is nothing to watch.
git ls-remote <source-of-truth> refs/heads/actual-state
# a3f9c21… refs/heads/actual-state
git ls-remote <backup> refs/heads/actual-state
# 7d14e08… refs/heads/actual-state
Two different SHAs. First reasonable reaction: latency. The mirror hasn’t run yet; the copy will land in a few seconds.
Except no. Commit 7d14e08 on the backup side is not an ancestor of a3f9c21. The backup isn’t behind the source: it has diverged. It contains content the source never had.
The context: one direction, in theory
On paper, the architecture fits in one sentence. The self-hosted Gitea repository is the source of truth. The GitOps deployer reads it continuously, with selfHeal and prune enabled — whatever is in the repository exists in the cluster, whatever is removed from it gets deleted from the cluster. A push mirror copies the source to an off-site backup repository, in that direction only.
One direction, one writer, one reader. It’s exactly the kind of invariant you stop checking after a few weeks, because it is true by construction.
It was. Until it stopped being true, with nothing to signal it.
The investigation: qualify the divergence before touching anything
A git ls-remote tells you two refs differ. It doesn’t tell you how. The question that matters, before any action, is: does content exist on only one side — and if so, which one?
That is what merge-base answers. Fetch both refs into a local clone, then count on each side of the divergence point:
git fetch source actual-state
git fetch backup actual-state
git merge-base source/actual-state backup/actual-state
# c8b2f45… ← the last common ancestor
git rev-list --left-right --count source/actual-state...backup/actual-state
# 3 2
Three commits present only on the source side. Two commits present only on the backup side. The divergence is bilateral, and the second number is the one that hurts: two commits the source of truth never saw, therefore the deployer never read, therefore they were never applied to the cluster.
git log --oneline source/actual-state..backup/actual-state
# 7d14e08 chore: remove static site (2)
# 9a02f31 chore: remove static site (1)
Two static-site removals. Dated five days earlier. Author: a service account.
The sites in question were still running in the cluster. In fact, they were running remarkably well: the deployer’s selfHeal, reading a source of truth where they were still declared, had been diligently keeping them alive for five days. Someone — something — had asked for their removal, and the system had conscientiously done the opposite.
Cause 1: a writer nobody expected anymore
The service account pointed to a small home-grown admin webapp, written well before the migration. Its job: manage a catalog of static sites, and commit changes directly to the GitOps repository. It did so through an SSH deploy key configured back when the backup repository was still the source of truth.
The migration had moved the source of truth. It had not moved this webapp. Nobody had thought of it, because nobody was using it — except, precisely, five days earlier.
The outcome is nastier than a mere configuration oversight: the webapp worked perfectly. It authenticated, it committed, it pushed, it got an acknowledgment back. From its point of view, the removal was done. Nothing in its interface could indicate it was now writing to a repository that no longer has a single reader. A residual writer after a migration doesn’t break: it keeps succeeding, into the void.
Cause 2: a token that expired in silence
That leaves the mirror. Had it been working, it should have overwritten those two commits long ago. Gitea keeps the state of every push mirror in its database, in a dedicated table. One query shows the whole fleet:
SELECT repo_id, remote_name, sync_on_commit,
to_timestamp(last_update) AS last_update,
last_error
FROM push_mirror
ORDER BY last_update ASC;
Out of 18 configured push mirrors, 16 had synced within the previous minutes. Two showed a last_update frozen on the same day — five days earlier — and an unambiguous last_error:
last_error: Invalid username or token. ...
The credential embedded in those two mirrors had expired. No notification, no alert: a failing push mirror writes the error to its table row and retries on the next cycle. The backup had been frozen for five days, and the only trace of that state lived in a column nobody looks at.
The token page on the backup side confirmed the diagnosis and, in passing, explained the contrast: the token used by the 16 healthy mirrors had no expiration date; the one behind the two broken mirrors had been created with a lifetime, and had reached it.
poussés après le gel du mirror — jamais recopiés vers le backup pushed after the mirror froze — never copied over to the backup
écrits par le writer résiduel — jamais lus par le déployeur written by the residual writer — never read by the deployer
And here lies the irony of this incident. The two failures were compensating for each other. A healthy mirror would have force-pushed the source’s content over the backup on every commit, erasing the webapp’s commits as they came. The sites would have kept running exactly the same, the webapp would have kept failing silently, and nothing would ever have revealed that a second writer existed. It is the mirror’s failure that preserved the evidence.
Remediation: order above all
Two independent problems, two obvious fixes. The trap is that they don’t commute.
Fixing the mirror first feels natural — it’s the “technical” failure, the one with an error message. It would have been data destruction. A push mirror doesn’t merge: it forces the remote ref to match the local one. The very first sync of a repaired mirror would have overwritten the backup with the source’s content, and the two orphaned commits would have vanished — this time for good, beyond the reach of any merge-base.
réconcilier reconcile
les 2 commits orphelins sont rapatriés dans la source de vérité — le backup n’est pas touché the 2 orphaned commits are brought back into the source of truth — the backup isn’t touched
~10 s plus tard ~10s later
l’intention vieille de 5 jours devient enfin réelle : les 2 sites sont supprimés du cluster the 5-day-old intent finally becomes real: the 2 sites are removed from the cluster
delete + recreate delete + recreate
le credential d’un mirror existant n’est pas éditable — il faut le supprimer puis le recréer, avec un token sans expiration an existing mirror’s credential can’t be edited — it has to be deleted then recreated, with a token that never expires
en conditions réelles under real conditions
un commit de test, puis comparaison des refs des deux côtés — pas seulement un code retour d’API a test commit, then comparing the refs on both sides — not just an API return code
un seul writer a single writer
deploy key révoquée, puis décommission complète de l’application qui l’utilisait deploy key revoked, then full decommissioning of the app that used it
So the applied order started with reconciliation:
1. Bring the orphaned commits back into the source of truth. A cherry-pick of the two removal commits from the backup ref onto the branch the deployer follows, then a push to the source. The backup is not touched at this stage.
git cherry-pick 9a02f31 7d14e08
git push source actual-state
2. Let the deployer do its job. About ten seconds later, the five-day-old intent finally became real: the two static sites were removed from the cluster. It is the only moment in the whole incident where the system did exactly what it was asked.
3. Recreate the push mirror. An implementation detail that costs five minutes of searching: Gitea does not let you edit the credential of an existing push mirror. You have to delete the entry and recreate it — this time with a dedicated token that has no expiration date.
4. Verify under real conditions. A test commit on the source, then a git ls-remote on both sides to compare the refs. Not an API return code, not a status displayed in some UI: the SHAs themselves. That lesson was already known around here — during the initial migration, a mirror-sync endpoint would happily return 200 while copying nothing.
5. Cut off the second writer. Revoke the webapp’s deploy key, then fully decommission the application itself — it had served no purpose since the migration, and keeping it meant keeping a write door open onto a critical repository.
Lessons learned
1. A mirror breaks silently, by design
A push mirror is a piece of infrastructure with no users. Nobody complains when it goes down, because nobody ever reads its output — being useless until the day it isn’t is precisely its job. The only protection is to monitor its state at the source. The last_error field of the mirrors table is exportable; an alert on “at least one mirror in error for more than an hour” would have turned five days of invisible incident into a notification on the first evening.
2. An expiring token is a time bomb, not a best practice
On a workstation or in CI, letting a credential expire is healthy: someone will renew it, because someone will notice the breakage. On a component nobody watches, expiration produces the exact opposite of the intended security — a silently dead backup. Two defensible options: a non-expiring token, inventoried and rotated deliberately; or a deliberate expiration paired with an end-of-life alert. The third option — a default expiration set once and forgotten — is the only one that protects nothing.
3. One repository, one writer
The invariant that actually broke isn’t the mirror; it’s this one. As soon as a second actor can write to a repository that participates in a synchronization chain, there exists a scenario where its work gets overwritten or ignored. The rule is easy to state and easy to let erode: everything that writes, writes to the source of truth. A service that must commit points at the source — never at a mirror, never at a backup.
4. A migration doesn’t end on switchover day
Repointing the readers is the visible part: it fails loudly if you forget it. Writers, on the other hand, never complain. After migrating a source of truth, you need an explicit inventory of everything holding write access — deploy keys, CI tokens, webhooks, service accounts, home-grown apps — and to repoint or revoke them one by one. A repository’s deploy-key list is a good starting point: every entry you can’t explain today is a residual writer in waiting.
5. Reconcile before repairing the channel
This is the lesson that transposes beyond git. Faced with two diverged replicas and a broken synchronization channel, the temptation is to fix the channel — it’s the failure with an error message. But a restored synchronization channel propagates; it does not arbitrate: it turns divergence into overwrite, in whichever direction its configuration dictates. As long as content exists only on the wrong side, repairing the sync is a disguised deletion. Qualify the divergence, reconcile, and only then put the channel back in service.
Runbook: checking a mirror chain
Do it once, then automate it:
git ls-remoteevery remote in the chain, compare the SHAs — not API statuses.- If they diverge:
git merge-basethengit rev-list --left-right --countto learn which side holds orphaned content. Repair nothing before you have that answer. - Query the mirrors’ state at the source (
last_update,last_error) and check none is frozen. - List each repository’s write credentials and confirm every one has a known owner and a known expiration date.
- After any repair of a sync channel, make a test commit and compare the refs on both sides.