Update 2026-06: this stack has come a long way since — centralized logs, alerting that actually fires, and automatic annotations. The evolution is traced at the end of this article, and the detailed implementation in Observability v2.
Why VictoriaMetrics rather than Prometheus?
The first Prometheus I deployed on a Pi 4 8 GB master ended up eating its way to 2 GB of RAM — for two weeks of retention and barely fifty targets. On a node that already runs the API server and etcd, that is a quarter of the available capacity requisitioned by a single monitoring component. What was needed was a tool designed to run in tight quarters.
VictoriaMetrics offers a far better performance-to-resources ratio:
- 5-10x better data compression than Prometheus
- ~70% lower memory usage at equivalent configuration
- Full compatibility with Grafana’s Prometheus dashboards
- Identical PromQL API — zero changes to existing dashboards
For a homelab on ARM64, it is the better choice.
Stack architecture
vmagent (collection)
│ scrapes targets (pods, nodes, k8s API)
▼
VictoriaMetrics single-node (storage)
│ compressed TSDB, 30d retention
├──► Grafana (dashboards)
└──► VMAlert (alert rules)
│
▼
Alertmanager (notifications)
vmagent is the lightweight equivalent of Prometheus scraping — it collects metrics from the targets and pushes them to VictoriaMetrics through the remote write protocol. It uses ~50 MB of RAM.
Deployment
VictoriaMetrics
apiVersion: apps/v1
kind: Deployment
metadata:
name: victoriametrics
namespace: monitoring
spec:
template:
spec:
containers:
- name: victoriametrics
image: victoriametrics/victoria-metrics:stable
args:
- -storageDataPath=/storage
- -retentionPeriod=30d
- -httpListenAddr=:8428
resources:
requests:
memory: 256Mi
limits:
memory: 1Gi
volumeMounts:
- name: storage
mountPath: /storage
30 days of retention with ~50 targets generates roughly 3-4 GB of compressed data. A 10 GB PVC is more than enough.
vmagent — scraping Kubernetes
The scraping configuration follows the Prometheus format. The key targets:
# vmagent-config ConfigMap
scrape_configs:
# Node metrics through node-exporter
- job_name: 'node-exporter'
kubernetes_sd_configs:
- role: node
relabel_configs:
- source_labels: [__address__]
regex: '(.*):.*'
target_label: __address__
replacement: '${1}:9100'
# Kubelet metrics
- job_name: 'kubelet'
kubernetes_sd_configs:
- role: node
scheme: https
tls_config:
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
# Pods carrying the prometheus.io/scrape annotation
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
Grafana
Grafana connects to VictoriaMetrics as a standard Prometheus datasource:
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-datasources
namespace: monitoring
data:
datasources.yaml: |
apiVersion: 1
datasources:
- name: VictoriaMetrics
type: prometheus
url: http://victoriametrics.monitoring.svc.cluster.local:8428
isDefault: true
The standard Kubernetes dashboards (Grafana IDs: 315, 6417, 13332) work without modification.
Key dashboards
Cluster Overview (Grafana 315): global CPU/RAM/network view per node. The first dashboard to open for a general status check.
Kubernetes Pods (Grafana 6417): per-pod metrics — ideal for spotting OOM-killed pods or containers using too much.
Longhorn: the official Longhorn dashboard for volume health and storage usage.
Node Exporter: detailed system metrics per node (CPU temperature, disk I/O, memory saturation).
Alerting
⚠️ These rules existed on paper — real alerting (Alertmanager → Telegram) was only switched on months later. Details in the 2026-06 update.
VMAlert evaluates the alert rules and sends notifications through Alertmanager. The critical alerts configured:
groups:
- name: homelab-critical
rules:
- alert: NodeMemoryHigh
expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) < 0.1
for: 5m
labels:
severity: warning
annotations:
summary: "Node {{ $labels.instance }} — available memory < 10%"
- alert: LonghornVolumeDegraded
expr: longhorn_volume_robustness != 2
for: 10m
labels:
severity: critical
annotations:
summary: "Longhorn volume {{ $labels.volume }} degraded"
- alert: PodCrashLooping
expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
for: 5m
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is CrashLooping"
Update 2026-05-12 — RWO deadlock on Grafana
Grafana stores its SQLite database (custom dashboards, users, plugins) on a Longhorn PVC in RWO (ReadWriteOnce) access mode. The default Deployment uses the RollingUpdate strategy with 1 surge and 0 unavailable. During a rolling update, the new pod tried to mount the same RWO PVC as the old pod still in Terminating → AttachVolume blocked → the rollout dragged on for 5+ minutes, sometimes all the way to the timeout.
Two fixes applied (commits 4c02c9c + 917823a):
1. Recreate strategy
spec:
strategy:
type: Recreate
The new pod waits for the old one to terminate completely before starting. This introduces a brief Grafana outage (~30s), in exchange for a deterministic rollout with no blocking. For an internal homelab dashboard, the trade-off is obvious.
2. Memory limit 256Mi → 512Mi
Grafana with defaultDashboardsEnabled: true (Helm chart) loads the dashboards into memory at startup and keeps the plugins active. At a 256Mi limit, random OOMKills happened when opening the k8s-cluster dashboard (30+ panels at once). Bumped to 512Mi → stable.
resources:
requests:
memory: 100Mi
limits:
memory: 512Mi
The requests stays at 100Mi to avoid over-reserving on the nodes.
The transferable lesson: any stateful workload on an RWO PVC needs strategy: Recreate (or an RWX PVC if the storage allows it). The “RollingUpdate + RWO” combo is a silent trap — it works on the days you get lucky, and blocks on the others.
Lessons learned
node-exporter on every node: deploy node-exporter through a DaemonSet (not a Deployment) to guarantee one instance per node, including future ones. Without node-exporter, system metrics are missing — only Kubernetes metrics are available.
Retention vs storage: 30 days of retention is a good balance. Beyond that, PVC growth becomes significant. For long-term archival, configure VictoriaMetrics to remote write to MinIO (S3-compatible format).
insecure_skip_verify: true for kubelet: the kubelet certificate is self-signed. The correct alternative is to configure a CA trust chain — acceptable for a homelab, but to be fixed for a production environment.
Update 2026-06-12 — v2: logs, real alerting, annotations
Retrospective honesty: the “Alerting” section above described the target more than the reality. vmalert and Alertmanager stayed inactive for months — the rules existed on paper, nothing notified anyone. And actual retention settled at 15 days (not 30): enough for diagnosis, half the storage.
What finally unblocked everything: an unattributable load spike during an audit — no per-pod metrics, no centralized logs, no alert. The four gaps were filled in a single day:
~317 Mi RAM, ça tient sur des Pi
Mais : un pic de load reste inexpliqué — pas de métriques par pod, pas de logs, alerting inactif.
~360k lignes ingérées en 2 min
Corrélation logs ↔ métriques dans Grafana, même time range — fini le SSH sur chaque nœud.
le cluster me parle, enfin
Events K8s Warning et déploiements ArgoCD directement sur les graphes — chaque pic a son contexte.
The implementation details (VictoriaLogs + Vector, vmalert → Alertmanager → Telegram, automatic graph annotations, and the “pending is not firing” lesson about flapping probes): Observability v2 — logs, alerts & annotations in one day.