Llama.cpp Integration

Memo bundles llama.cpp as its native inference engine. It is managed as a subprocess by the Go backend with automatic lifecycle handling, GPU offloading, and port coordination.

Subprocess Lifecycle

The Go backend manages llama.cpp as a child process with full lifecycle control:

Go Backend (main process)
  │
  ├─▶ llama.cpp Server    (port 8081) — Chat inference
  ├─▶ llama.cpp Embedding  (port 8082) — Vector embeddings
  └─▶ Health checker       (goroutine) — Periodic ping + restart

Lifecycle States

  [STOPPED] ──start()──▶ [STARTING] ──health_ok──▶ [RUNNING]
                              │                        │
                              └──timeout──▶ [DEAD]     │ crash
                                           restart()   │
                                           loop ───────┘

• STARTING: Process spawned, waiting for HTTP health endpoint to respond
• RUNNING: Health check passing, accepting inference requests
• DEAD: Process crashed or unresponsive. Automatic restart with exponential backoff (1s → 2s → 4s → max 30s)

GPU Offloading

llama.cpp is launched with GPU layer offloading configured per detected hardware:

Platform	Backend	Flag	Detection
NVIDIA	CUDA	--n-gpu-layers N	nvidia-smi + CUDA lib
AMD	ROCm	--n-gpu-layers N	rocm-smi + ROCm lib
Apple Silicon	Metal	--n-gpu-layers N	MPS device check
CPU-only	—	--n-gpu-layers 0	Fallback

The number of offloaded layers is calculated automatically based on available VRAM and model size. Users can override in Settings.

# Example: NVIDIA with 8GB VRAM, llama-3.1-8b Q4_K_M
llama-server \
  --model models/llama-3.1-8b-instruct-q4_k_m.gguf \
  --n-gpu-layers 33 \
  --host 127.0.0.1 \
  --port 8081 \
  --ctx-size 8192

Automatic Installation

llama.cpp binaries are bundled with the application package. On first launch:

1. The Go backend detects the platform (Windows/Linux/macOS)
2. Extracts the appropriate pre-built llama-server and llama-embedding binaries
3. Verifies the binary with a version check (--version)
4. Runs system diagnostics for GPU capability
5. Launches both servers with platform-optimized flags

No manual compilation or dependency installation is required.

Health Checks

A health goroutine polls the llama.cpp HTTP endpoints every 5 seconds:

GET /health → 200 OK  → RUNNING
GET /health → timeout → mark DEAD → restart
GET /health → 500     → log warning → retry after 15s

func (e *LlamaEngine) healthLoop() {
    ticker := time.NewTicker(5 * time.Second)
    for range ticker.C {
        if !e.isHealthy() {
            e.restart()
        }
    }
}

Port Management

Memo uses a fixed port allocation to avoid conflicts:

Service	Default Port	Config Key
Chat Inference	8081	llama.chat_port
Embedding Server	8082	llama.embed_port
API Server	8090	server.port

Ports are checked at startup. If a port is occupied, Memo attempts the next available port and logs a warning.

Dedicated Embedding Server

The embedding server runs as a separate llama.cpp instance with its own model (default: nomic-embed-text-v1.5):

• Isolation: Embedding requests never block chat inference
• Optimization: Embedding server uses smaller context window, no GPU offloading needed
• Concurrency: Both servers can process requests simultaneously

llama-embedding \
  --model models/nomic-embed-text-v1.5.gguf \
  --host 127.0.0.1 \
  --port 8082 \
  --embd-normalize 2 \
  --ctx-size 2048

Fallback Behavior

If llama.cpp fails to start or crashes irrecoverably:

• External providers become the fallback inference path
• Memory search continues via cached embeddings (read-only)
• A notification is shown in the UI with the error details
• Manual restart is available from Settings → Diagnostics

---

Running llama.cpp models larger than available RAM will cause severe performance degradation due to swap thrashing. Always check the hardware compatibility badge before downloading.