Mesurer son audience sans traceur : quand un import horodaté multiplie le trafic par 8 Counting Visitors Without a Tracker: How a Timestamped Import Multiplied My Traffic by Eight

Context: the monitoring stack (VictoriaMetrics, vmalert, Alertmanager, Telegram) is described in Observability v2, and the last review of the Grafana dashboards already got a post of its own. This one is about a dashboard that was missing nothing: it simply showed numbers eight times too large.

The context: an audience without a single line of JavaScript

The blog and the homelab’s public services all go through a Cloudflare tunnel. Cloudflare therefore already counts every request before it even reaches the cluster. Rather than adding a measurement script to the pages, I wrote a small Python collector at the end of July: a CronJob that queries Cloudflare’s GraphQL Analytics API (free plan) every 15 minutes, flattens the aggregates and pushes them into VictoriaMetrics in the Prometheus exposition format, with an explicit timestamp on every point.

Two families of series come out of it:

• hourly zone aggregates (cfa_requests, cfa_bytes, cfa_threats…), from httpRequests1hGroups;
• the fine breakdown by host, country and status (cfa_host_requests), plus subdomain probing (cfa_scan_requests), from httpRequestsAdaptiveGroups.

Each run re-reads the last two closed hours. That is by design: a failed run loses nothing, the next one re-reads the same range. The export function’s comment summed up the philosophy with confidence:

# L'horodatage rend l'import IDEMPOTENT : rejouer la meme fenetre ecrit les
# memes points aux memes instants, sans doublon.

(“The timestamp makes the import IDEMPOTENT: replaying the same window writes the same points at the same instants, without duplicates.”) That sentence is false. It took ten weeks to notice.

The symptom: 47 requests at Cloudflare, 420 in Grafana

On 11 October 2026, checking one specific hour of the “Audience” dashboard against the source, the two numbers refuse to match. Nextcloud, status 302, hour 19:00Z on 9 October:

Source	Requests
Cloudflare GraphQL API	47
VictoriaMetrics (sum_over_time over the hour)	372 to 420

A factor of roughly 8. Not a drift, not rounding: a multiple. And the hourly zone aggregates, computed along an entirely different path, showed the same factor.

An audience dashboard raises no alert when it is too optimistic. Nobody complains about having eight times more visitors than expected.

The investigation: read the raw export, not the dashboard

Grafana aggregates, so it hides exactly what you are looking for. VictoriaMetrics’ /api/v1/export endpoint returns every stored sample, timestamp included. For a single hourly point of cfa_host_requests, there were 12 samples. And 12 is exactly 3 HTTP paths × 4 runs: each sample equalled one path’s total over a 2-hour window.

Three causes stacked up.

1. The 2-hour window was dated at its end. The 20:15 run read 18:00-20:00 and wrote everything at 20:00. The 21:15 run read 19:00-21:00 and wrote at 21:00. Hour 19:00 therefore counted in two consecutive points: ×2.
2. VictoriaMetrics did not deduplicate. The four runs of a given hour wrote the same point four times at the same timestamp, and the database kept all four copies. sum_over_time added them all up: ×4. For the hourly aggregates, correctly dated at the start of the hour but re-read in two windows, that meant 8 copies of the same point.
3. One point per path, with identical labels. The path only matters for probing; on legitimate traffic it was read but not exposed as a label. Three paths of the same host produced three indistinguishable points, which the lack of deduplication added up… by accident, and correctly.

victoriametrics Une heure, huit copiesOne hour, eight copies import horodatétimestamped import

① chez Cloudflareat Cloudflare

une heure, un hôte, un statutone hour, one host, one status

Le compte de référence : 47 requêtes.The reference count: 47 requests.

② la collectethe collection

toutes les 15 min, 2 h reluesevery 15 min, 2 h re-read

Chaque heure close est relue par 8 passages. Voulu : rien ne se perd si un passage échoue.Each closed hour is re-read by 8 runs. Intended: nothing is lost if one run fails.

③ estampille de finend-of-window stamp

fenêtre de 2 h datée à sa fin2 h window dated at its end

La même heure tombe dans deux points consécutifs : ×2.The same hour lands in two consecutive points: ×2.

④ aucune déduplicationno deduplication

4 passages, même horodatage4 runs, same timestamp

La base garde les 4 copies, sum_over_time les additionne : 372 à 420 au lieu de 47.The database keeps all 4 copies, sum_over_time adds them up: 372 to 420 instead of 47.

⑤ le correctifthe fix

datetimeHour + dedup 1msdatetimeHour + 1ms dedup

Un point par heure du trafic et par série ; la base n'en garde qu'un. Retour à 47.One point per traffic hour and per series; the database keeps only one. Back to 47.

The third cause is the sneakiest. As long as the database does not deduplicate, it cancels itself out. The day you turn deduplication on to fix the second cause, it becomes a bug in turn: the database keeps only one of the three paths.

Just enough context: what “deduplicate” means here

By default, VictoriaMetrics accepts several samples of the same series at the same timestamp. For scraped series the question never arises: vmagent dates each point at scrape time. For a timestamped import that deliberately rewrites the past, it is a trap.

The -dedup.minScrapeInterval setting merges a series’ samples that fall within the same interval. With 1 ms, it only merges points with the exact same timestamp: no effect on scraped series, decisive for this import. The merge rule is not “last write wins” but “largest value wins”, both at query time and during on-disk merges.

# monitoring/vm/vmsingle-values.yaml
server:
  retentionPeriod: "15"        # no unit: 15 months, not 15 days
  extraArgs:
    dedup.minScrapeInterval: 1ms

Incidentally, the repository’s documentation claimed 15 days of retention. retentionPeriod without a unit means months. A mistake in the right direction, for once.

The fix: one point per traffic hour and per series

On the collector side, three changes:

• each adaptive row requests the datetimeHour dimension and is dated at the traffic hour, no longer at the end of the window;
• paths are summed per series before export: a run never emits two points of the same series at the same instant;
• the database deduplicates, and the docstring finally tells the truth: the import is idempotent only because VictoriaMetrics deduplicates.

cle = (name, tuple(sorted(labels.items())), _ts_ms(heure))
totaux[cle] = totaux.get(cle, 0.0) + float(row.get("count") or 0)

The tests came first, with two fakes: a fake Cloudflare that genuinely groups by the requested dimensions, and a fake VictoriaMetrics that deduplicates like the real one. 38 tests, then 12 deliberate mutations of the code, all caught. Without the deduplicating fake, the third cause would have slipped through: the tests would have summed the paths just as the database did.

The trap within the trap: a 500-row page

Re-reading the query surfaced a second defect. The adaptive query had a limit of 500 groups, sized on 29 July from a measurement of 71 groups over 2 hours, “7× headroom”. One burst of subdomain probing later, the worst measured hour had 9,021 groups (on 22 September). The response was silently cut, largest groups first: the rarest probes vanished.

The free plan’s limits, read from the API settings, are: a maxPageSize of 10,000, 31 days of history for adaptive data, 73 hours for hourly aggregates, 365 days for daily ones. The limit therefore goes up to 10,000, and a full page is split in two, recursively, down to a single hour. If a single hour still overflows, a cfa_query_truncated sentinel says so: no silent ceiling.

The repair: when “largest value wins” forbids correction

Turning deduplication on fixes the hourly aggregates in one go: their eight copies shared the same timestamp, the database keeps the largest, which is also the most complete. The whole hourly history becomes accurate again.

For cfa_host_requests and cfa_scan_requests, it is the opposite. Their old points, dated at the end of the 2-hour window, land on the same full hours as the new ones. Once deduplicated, an old point is worth the largest path of its 2-hour window, for example 70 instead of 50 for Nextcloud 302. It is larger than the correct value, so it wins. Neither collection nor a backfill can correct it.

The only way out: delete those series and rebuild them. And VictoriaMetrics only deletes entire series, not a time range. The procedure, in order:

1. Raw backup first, via /api/v1/export: after merges, the copies are gone for good. 2,361 series, 589,906 samples, re-importable as is.
2. vmsingle restarted with deduplication, fixed collector deployed, one run of the new code completed.
3. Exact deletion of the two series names, preview included. Never cfa_.*, which would take the hourly aggregates with it, and Cloudflare only keeps those for 73 hours:

M='match[]={__name__=~"cfa_host_requests|cfa_scan_requests"}'
curl -s "$VM/api/v1/series" --data-urlencode "$M" --data-urlencode 'start=2026-01-01T00:00:00Z' | jq '.data | length'
curl -s -X POST "$VM/api/v1/admin/tsdb/delete_series" --data-urlencode "$M"

4. A backfill (MODE=backfill) that replays the last 30 days of adaptive data, closed hour by closed hour, exactly like live collection.
5. A check against the source.

Result: 2,072 series deleted, the other 289 cfa_* series untouched, rebuilt from 11 September onwards. Hour-by-hour check against GraphQL: 3,346 host × hour pairs identical out of 3,346 (145,192 requests on both sides), and 72 hours out of 72 for the zone total. Before 11 September, the per-host panels are empty. An honest gap beats a false history.

The review also fixed the classification: about ten real hosts, including the NAS media services, served with 200 and 304 for weeks, were counted as “probed subdomains” because they were missing from the list of legitimate hosts.

The day after: monitor the collector, then monitor the alerting

A failing collector was only visible on the dashboard. The vmalert group cf-analytics fixes that with two rules:

• CfAnalyticsCollectorDown (warning): a single rule for both failure modes, failed runs or nothing published at all. It fires if and only if the third consecutive run fails, 55 minutes after the last success. Since each hour is re-read for 2 hours, nothing is lost as long as a collection succeeds within an hour and 45 minutes of the previous one: firing earlier would be noise.
• CfAnalyticsQueryTruncated (info): a single hour still overflows the 10,000-group page.

Before deploying it, the rule was replayed over the history, from 29 July to 11 October. It would never have fired for a collector failure (16 minutes of consecutive failures at most), only during outages of VictoriaMetrics itself, when nothing would have fired live anyway. Along the way, KubeJobFailed now ignores this CronJob’s Jobs: each failed attempt produced a FIRING then a RESOLVED, even when the third attempt had published.

Reviewing these rules surfaced four older blind spots, in the alerting chain itself:

1. Alertmanager was not scraped. A rejected Telegram delivery left no trace. It now has its own job and an alert on alertmanager_notifications_failed_total.
2. Beyond 4,096 characters, Alertmanager replaces the entire Telegram message with a generic text, without counting a failure. One day, 82 alerts fired together: 12,915 characters. The template now lists 15 and gives the total.
3. rate() returns an empty result, not zero, when ingestion stops completely. The “less than one inserted row per second” rule added during the previous review therefore could not fire in the worst case. An absent_over_time branch closes the gap.
4. vmalert lost the state of its alerts on restart: an ordinary restart produced a RESOLVED then a FIRING. The -remoteRead.url flag restores the state from the database.

Lessons learned

1. A timestamped import is only idempotent if the database deduplicates. Check the database setting before writing “idempotent” in a comment. On VictoriaMetrics, -dedup.minScrapeInterval=1ms only affects points with identical timestamps.
2. Date an aggregate at the start of the period it measures, never at the end of the read window. If the API provides the time dimension (datetimeHour), ask for it.
3. Read the raw export when a number looks off. Counting samples per point immediately tells you how many writes are stacked up.
4. “Largest value wins” makes over-counting permanent. A value that is too small can be fixed by rewriting; a value that is too large forces you to delete the whole series. Back up the raw export first, delete the exact name, rebuild within the source’s retention limit.
5. Tests must simulate the database, not just the API. A fake VictoriaMetrics that deduplicates revealed a bug the real database was masking.
6. Monitor the alerting chain like a service: the notifier, message size, the “no data at all” case.

Today the “Audience” dashboard shows numbers eight times more modest. They have the advantage of being true.