Running Local AI with Ollama + Open WebUI (decommissioned)

Update 2026-05-10 — Stack decommissioned. The ai ArgoCD app (Ollama + Open WebUI) has been removed from the cluster. Reason: 3 Longhorn replicas × 25 Gi (20 Gi of models + 5 Gi of config) saturated the distributed storage and stayed permanently degraded — the cluster only has 3 Pi workers, one of them on a 32 Gi USB drive and unable to host a 20 Gi replica. Since the models can be re-downloaded, deletion was accepted as the lesser evil. The rest of the article is kept as a field report.

Motivation: AI without a cloud dependency

LLMs reachable through cloud APIs (OpenAI, Anthropic, Google) are powerful but raise questions of privacy and cost. For a homelab, having a model running locally offers:

Total privacy: no data leaves the local network
Offline availability: works without internet
Free experimentation: try models, prompts and configurations without a token meter

The flip side: Raspberry Pis are not GPUs. Inference is slow. A 7B-parameter model takes 30-60 seconds to generate an answer on an RPi 5 in CPU-only mode. Acceptable for experimentation, not for a real-time assistant.

Ollama on ARM64

Ollama is an LLM inference runtime that automatically handles model downloads, quantization, and exposing an OpenAI-compatible API. It supports ARM64 natively.

Kubernetes deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ai
spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - containerPort: 11434
        resources:
          requests:
            memory: 2Gi
          limits:
            memory: 10Gi    # Models are loaded into RAM
        volumeMounts:
        - name: ollama-data
          mountPath: /root/.ollama
      nodeSelector:
        node-type: worker    # Only on the 16GB workers

strategy: Recreate: Ollama loads models into memory. Two simultaneous instances would cause conflicts on the data volume. Recreate guarantees that only one instance runs at a time.

10 Gi memory limit: 7B models in Q4 quantization take up ~4 GB of RAM; gemma4:e2b (7.2 GB on disk) consumes more once loaded. The high limit avoids an OOM kill during inference over a long context.

Models used

# Pre-pull the models through a Kubernetes Job
kubectl exec -n ai deployment/ollama -- ollama pull gemma4:e2b
kubectl exec -n ai deployment/ollama -- ollama pull llama3.2:1b
kubectl exec -n ai deployment/ollama -- ollama pull nomic-embed-text  # For RAG

gemma4:e2b (7.2 GB) is the main model — Google Gemma 4 in 2-bit quantization, a good balance between quality and memory footprint on an RPi 5. llama3.2:1b (1.3 GB) is used for light tasks that need low latency. nomic-embed-text is a lightweight embeddings model, used for Open WebUI’s RAG feature.

Open WebUI

Open WebUI is a ChatGPT-style web interface designed to work with Ollama (and other OpenAI-compatible backends). It supports conversations, documents, images and RAG.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: open-webui
  namespace: ai
spec:
  template:
    spec:
      containers:
      - name: open-webui
        image: ghcr.io/open-webui/open-webui:main
        env:
        - name: OLLAMA_BASE_URL
          value: http://ollama.ai.svc.cluster.local:11434
        - name: WEBUI_AUTH
          value: "true"
        resources:
          requests:
            memory: 512Mi
          limits:
            memory: 2Gi

RAG: avoiding HuggingFace downloads

By default, Open WebUI downloads embeddings models from HuggingFace for RAG. On an RPi with little RAM and a residential connection, those downloads (~500 MB) cause timeouts and OOM kills.

Solution: configure Open WebUI to use Ollama as its embeddings backend, pointing at nomic-embed-text, which is already present locally.

env:
- name: RAG_EMBEDDING_ENGINE
  value: ollama
- name: RAG_OLLAMA_BASE_URL
  value: http://ollama.ai.svc.cluster.local:11434
- name: RAG_EMBEDDING_MODEL
  value: nomic-embed-text

This change removes the HuggingFace downloads at startup and significantly reduces Open WebUI’s memory consumption.

Storage and persistence

Two PVCs are required:

# PVC for the Ollama models (~20 GB minimum for several models)
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ollama-data
  namespace: ai
spec:
  storageClassName: standard-storage
  resources:
    requests:
      storage: 20Gi

The LLM models are stored in standard-storage (1 replica) — they can be downloaded again from the Ollama Registry and are not irreplaceable data.

The Open WebUI data (conversations, users) uses critical-storage (2 replicas).

Performance in practice

On an RPi 5 (ARM Cortex-A76, 16 GB RAM):

Model	Size	Tokens/sec	First-token latency
llama3.2:1b	1.3 GB	~15 t/s	~2s
gemma4:e2b	7.2 GB	~4 t/s	~12s
nomic-embed-text	274 MB	n/a (embeddings)	~0.5s

This performance is good enough for summarising, simple code generation and exploration — not for a fluid interactive assistant.

Lessons learned

Pick Q4 quantization: models in Q8 or FP16 are too heavy for the RPis. Q4 offers a good quality/memory compromise.

Recreate rather than RollingUpdate: during an Ollama update, the old pod must terminate before the new one starts, otherwise both instances try to access the same model files — guaranteed corruption.

Patience: the first startup after pulling a model can take several minutes. Liveness probes need a generous initialDelaySeconds (120s minimum) to avoid premature restarts.

IA locale avec Ollama + Open WebUI (décommissionné) Running Local AI with Ollama + Open WebUI (decommissioned)