Observabilité v2 — logs, alertes & annotations en une journée Observability v2 — Logs, Alerts & Annotations in One Day

On 11 June, while going through six hours of cluster metrics, I run into a load spike of 7.18 on worker-1 (16× its average) at 17:20. No pod created in the window. Nothing in the k3s-agent logs. No way to know which workload had eaten the CPU — because vmagent was not scraping cAdvisor, so there was no per-pod history at all.

That unattributable spike acted as a revealer. My v1 monitoring (VictoriaMetrics + Grafana) could say “the node is suffering”, but not “here is the culprit”. The audit listed four blind spots:

1. No per-pod/container metrics — cAdvisor not scraped, no way to attribute a spike after the fact
2. No centralized logs — correlation meant SSH by hand into every node, one at a time
3. No alerting — a backend’s readiness probe had been failing silently for days
4. No annotations — the Grafana graphs showed neither K8s events nor ArgoCD deployments

By the following evening, all four were closed. Here is how, and above all what I learned on the way.

① cAdvisor — the quick win (30 minutes)

cAdvisor is already in the kubelet, on every node, port 10250. All it takes is scraping it:

# monitoring/vm/vmagent-values.yaml — job added
- job_name: cadvisor
  scheme: https
  tls_config: { insecure_skip_verify: true }
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs: [{ role: node }]
  relabel_configs:
  - source_labels: [__address__]
    regex: '(.*):10250'
    target_label: __metrics_path__
    replacement: /metrics/cadvisor

vmagent’s existing ClusterRole already had nodes/metrics — one helm upgrade later: 6/6 targets up, 426 container_cpu_usage_seconds_total series. The K8s dashboard (15760) enriches itself. The next spike will have a name and a namespace.

② Centralized logs — VictoriaLogs + Vector, not Loki

Loki had been on my roadmap for months. I never deployed it, and in hindsight that was good procrastination: VictoriaLogs is lighter, ARM64-native, and consistent with my existing VictoriaMetrics stack (same vendor, same single-binary philosophy).

The architecture is two pieces:

• VictoriaLogs single-node (v1.50.0) — 10 Gi Longhorn PVC, 7-day retention
• Vector as a DaemonSet (0.56.0) on the 6 nodes — reads /var/log/pods, enriches by namespace/pod, ships through the Elasticsearch bulk API

# victorialogs/vector.yaml — the heart of the pipeline
[sources.k8s]
type = "kubernetes_logs"

[sinks.vlogs]
type = "elasticsearch"
endpoints = ["http://victorialogs:9428/insert/elasticsearch/"]
api_version = "v8"
[sinks.vlogs.query]
_msg_field = "message"
_time_field = "timestamp"
_stream_fields = "kubernetes.pod_namespace,kubernetes.pod_name"

Verification after deployment: ~360,000 lines ingested in 2 minutes, queryable in Grafana through the victoriametrics-logs-datasource plugin. The concrete win: metrics and logs in the same Grafana time range. A spike on a graph → one click → the logs from the same window. No more journalctl over SSH on six machines.

One detail from the field: the first Vector pods crash-looped during the rollout (5-6 restarts while VictoriaLogs and its PVC came up). That is exactly the kind of event the rest of this article makes visible.

③ Alerting — vmalert → Alertmanager → Telegram

The main course. Two deployments and one SealedSecret:

• vmalert (v1.140.0) evaluates the rules every 30 s against vmsingle, and writes the ALERTS state back into VictoriaMetrics (handy: the pending/firing history becomes queryable)
• Alertmanager (v0.32.2) groups and routes to Telegram — the bot token lives in a SealedSecret mounted as bot_token_file, never in the clear in Git

Six rules to start with, deliberately simple:

- alert: NodeDown          # up{job="kubernetes-nodes"} == 0, for: 3m
- alert: NodeHighLoad      # node_load1 > 8, for: 10m
- alert: DiskAlmostFull    # >90%, for: 15m
- alert: NodeMemoryPressure # >92%, for: 10m
- alert: PodCrashLooping   # restarts > 3 in 15m, for: 5m
- alert: PodNotReady       # not-ready in Running phase, for: 15m

First false positive on the very first evening: PodNotReady was firing for the pods of finished Jobs (Completed means ready false forever — Longhorn snapshots, metrics CronJobs…). The fix, a join on the phase:

- alert: PodNotReady
  expr: |
    kube_pod_status_ready{condition="true"} == 0
    and on(namespace, pod) kube_pod_status_phase{phase="Running"} == 1
  for: 15m

④ Annotations — K8s events on the graphs

Last piece: a grafana-annotator CronJob (every 5 min) pushing to Grafana’s annotations API:

• the Warning K8s events (tag k8s-warning) — probe failures, OOMKilled, FailedScheduling…
• the ArgoCD syncs (tag argocd) — every deployment becomes a vertical line on the graphs

It is the least spectacular and the most profitable tool of the day. The next morning, the dashboards were telling a story: every load spike had its dotted orange line next to it, with the event that explained it.

The next day’s lesson: pending is not firing

Immediate irony: the following day, the new annotations showed recurring probe failures on a backend (/api/health timeouts under load) — and Telegram stayed silent. Was the pipeline broken?

No. It was working exactly as configured:

alerting vmalert → porte for: → Alertmanager → Telegramvmalert → for: gate → Alertmanager → Telegram pending ≠ firing

vmalert eval 30s · 6 règleseval 30s · 6 rules

porte for:for: gate pending → firing

Alertmanager group_wait 30s · repeat 4h

Telegram @bot · SealedSecret

panne franche — la condition reste vraieclean outage — the condition stays true

🔴 FIRING

panne qui flappe — la condition s'interromptoutage that flaps — the condition keeps interrupting itself

🟡 pending…

The investigation, with numbers to back it up (the ALERTS state written back into VictoriaMetrics makes this trivial):

• PodNotReady{pod="...backend..."}: 3 pending samples in 24 h, zero firing — each not-ready episode lasted ~2 min, never the 15 min of the for:
• NodeHighLoad{instance="worker-2"}: 6× pending — the load went above 8 in spikes, never for 10 continuous minutes
• Meanwhile, on the Alertmanager side: alertmanager_notifications_total{integration="telegram"} = 8, zero failures — the alerts that had actually reached firing (the Vector crash-loops, precisely) had gone out just fine

Three takeaways:

1. The for: is a gate, not a delay. The condition has to stay true without interruption. A workload that flaps in short episodes lives in eternal pending — invisible if you only look at notifications.

2. Verify the pipeline end to end, with metrics. alertmanager_notifications_failed_total per integration tells you in a single query whether the problem is “nothing fired” or “delivery is failing”. In my case: nothing had fired for that pod, and the 8 real deliveries had gone to the configured chat — not the one I was watching. Check your chat_id and open the right chat.

3. Flapping deserves its own rule. Neither PodNotReady nor NodeHighLoad can catch it, by construction. The missing rule:

- alert: PodReadyFlapping
  expr: changes(kube_pod_status_ready{condition="true"}[30m]) > 4
  for: 0m
  labels: { severity: warning }
  annotations:
    summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is flapping ({{ $value }} ready transitions in 30m)"

changes() counts transitions — that is the signature of flapping, insensitive to how long the episodes last.

Outcome

Four blind spots closed in a day, in GitOps (two ArgoCD apps: victorialogs, alerting, plus the annotator CronJob), on Raspberry Pis that take it without flinching — VictoriaLogs and Vector add ~150 Mi of RAM in total.

The real victory is not the stack, it is the loop: instrumentation deployed one evening made visible, the very next day, a problem that had been going on for weeks — and supplied the metrics to understand why it was structurally unalertable. That is what observability is: not dashboards, but questions you can finally answer.