Performance

Memo is engineered for low-latency, high-throughput local AI. Every component is optimized for the constraints of consumer hardware — laptops, desktops, and workstations.

Token Streaming via SSE

Messages are delivered token-by-token using Server-Sent Events (SSE), providing immediate visual feedback:

HTTP GET /api/chat/stream
  │
  ├─▶ data: "Hello"
  ├─▶ data: ","
  ├─▶ data: " world"
  ├─▶ data: "!"
  └─▶ data: [DONE]

Streaming Characteristics

Metric	Local (llama.cpp)	External Provider
Time to First Token (TTFT)	50–200ms	300–800ms
Tokens per Second (TPS)	20–60 (GPU) / 5–15 (CPU)	30–100 (API-dependent)
Buffer Size	1 token	1 token
Connection	Persistent SSE	Persistent SSE

The Flutter frontend uses Riverpod state management with per-token granularity updates. There is no batching delay — each token is rendered as it arrives.

GPU Layer Configuration

The most impactful performance setting is GPU layer offloading:

Model: llama-3.1-8b-instruct (Q4_K_M, 4.9 GB)

n_gpu_layers = 0   →  CPU only        →  8–12 tok/s
n_gpu_layers = 16  →  Partial GPU     →  18–25 tok/s
n_gpu_layers = 33  →  Full GPU (8GB)  →  35–55 tok/s

Memo auto-configures n_gpu_layers based on detected VRAM. The formula:

func optimalLayers(modelSizeGB, vramGB float64) int {
    maxLayers := totalLayers(modelSizeGB)
    gpuRatio := vramGB / modelSizeGB
    if gpuRatio >= 0.9 {
        return maxLayers
    }
    return int(float64(maxLayers) * gpuRatio)
}

Users can override the auto-configuration in Settings → Performance.

RAM Management

Model Loading

GGUF models are memory-mapped (mmap) for efficient loading:

• Cold start (first load): Full file read into memory (~5–10s for 5 GB model)
• Warm start (cached): OS page cache serves the file (~1–3s)
• mmap advantage: Multiple models can share the same physical pages via the OS buffer cache

Context Window

The context window size (--ctx-size) directly affects RAM usage:

Context Size	RAM Overhead
2048	~512 MB
4096	~1 GB
8192	~2 GB
16384	~4 GB
32768	~8 GB

Reducing context size is the most effective way to fit a model on limited hardware.

Core Performance Metrics

Latency

Operation	p50	p95	p99
Model load (GPU)	3s	8s	15s
First token (GPU)	80ms	200ms	500ms
First token (CPU)	150ms	400ms	1s
Vector embedding (single chunk)	30ms	80ms	150ms
Vector search (10K docs)	3ms	8ms	15ms
Agent tool dispatch	5ms	20ms	50ms

Throughput

Workload	GPU (RTX 3060)	CPU (Ryzen 7)
Chat tokens/sec	40–55	10–15
Embeddings/sec (768-dim)	50–80	20–30
Ingest (pages/sec)	15–25	5–10

Optimization Tips

Strategy	Impact	Trade-off
Increase GPU layers	2–4× faster inference	Requires VRAM headroom
Use Q4_K_M quantization	50% less RAM vs Q8_0	Minimal quality loss
Reduce context window	Less RAM pressure	Shorter conversation memory
Disable unused providers	Faster startup	Must re-enable to use
Limit agent iterations	Faster task completion	Complex tasks may need more
Use local model for simple queries	Zero network latency	Complex reasoning may need cloud

Monitoring

Memo exposes runtime performance metrics via the API and diagnostic dashboard:

GET /api/diagnostics
{
  "llama_server": {
    "status": "running",
    "uptime_seconds": 12345,
    "tokens_generated": 458200,
    "avg_tps": 42.3,
    "vram_used_mb": 5800,
    "vram_total_mb": 8192
  },
  "embedding_server": {
    "status": "running",
    "uptime_seconds": 12340,
    "embeddings_generated": 12400,
    "avg_ms_per_embed": 45.2
  },
  "memory": {
    "total_chunks": 8420,
    "index_size_mb": 128,
    "last_vacuum": "2026-07-04T10:15:00Z"
  }
}

If you see avg_tps drop below expected levels, check that GPU layers are offloaded (Settings → Performance) and that no other GPU-intensive application is competing for VRAM.