RAG Semantic Memory

Memo's Retrieval-Augmented Generation (RAG) system gives the model persistent, searchable memory across conversations. It is built on SQLite with vector extensions — no external vector database required.

Architecture Overview

┌──────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│  Ingest Doc   │────▶│  Chunk & Embed      │────▶│  SQLite + vec0    │
│  (MD/TXT/PDF) │     │  (768-dim vectors)  │     │  (ANN Index)      │
└──────────────┘     └─────────────────────┘     └────────┬─────────┘
                                                          │
┌──────────────┐     ┌─────────────────────┐              │
│  User Query   │────▶│  Embed Query        │────▶────────┘
│               │     │  (same model)       │     │  Cosine Search  │
└──────────────┘     └─────────────────────┘     │  Top-K Results  │
                                                  └────────┬────────┘
                                                           │
┌──────────────┐     ┌─────────────────────┐              │
│  LLM Response │◀────│  Inject Context     │◀─────────────┘
│  (augmented)  │     │  (prepend chunks)   │
└──────────────┘     └─────────────────────┘

Embedding Model

• Default model: nomic-embed-text-v1.5
• Dimension: 768
• Format: GGUF, loaded via llama.cpp embedding server
• Dedicated server: Runs on a separate port from the chat inference server

The embedding server is managed as a subprocess with health checks and automatic restart on failure.

Storage Layer

Vectors and metadata are stored in SQLite using the sqlite-vec extension with the vec0 virtual table for Approximate Nearest Neighbor (ANN) indexing:

CREATE VIRTUAL TABLE vec_memory USING vec0(
    embedding float[768],
    content text,
    source text,
    chunk_index integer,
    created_at datetime
);

Persistence Strategy

• Binary-atomic persistence: Write operations use SQLite's atomic commit. Partial writes are impossible.
• RAM indexing: The vec0 ANN index lives in memory for sub-millisecond search. The underlying table persists to disk.
• Lazy loading: Chunks are loaded from disk only when matched by a search query. The full index is checked first.

Ingestion Pipeline

When a document or conversation chunk is ingested:

1. Chunking: Content is split into overlapping chunks (default ~512 tokens with 10% overlap)
2. Deduplication: Content hash check against existing chunks to avoid duplicates
3. Embedding: Each chunk is sent to the embedding server, returning a 768-dim float32 vector
4. Storage: Vector + metadata written to the vec_memory table in a single transaction
5. Index Update: The vec0 ANN index is updated atomically

Semantic Search Flow

When a user query triggers memory recall:

1. Query Embedding: The query text is embedded using the same model
2. Cosine Similarity Search: The vector is compared against the ANN index
3. Top-K Retrieval: The K most similar chunks are fetched (default K=5)
4. Threshold Filter: Results below min_similarity (default 0.7) are discarded
5. Context Injection: Matching chunks are prepended to the system prompt:

[Relevant Memory]
- chunk_1 content (similarity: 0.92)
- chunk_2 content (similarity: 0.85)

[User Message]
...current query...

Cross-Mode Memory

Memo supports cross-mode memory where external chat providers (OpenAI, Claude, etc.) can still write to and read from your local vector store:

• Chat responses from external providers are embedded and stored locally
• Memory search runs locally regardless of which provider generated the response
• The embedding server is always local — even when the chat model is remote

This means your memory stays unified whether you're using a local GGUF model or an external API.

Concurrency & Safety

• Read/write separation: The vec_memory table supports concurrent reads; writes are serialized via sync.RWMutex
• Store initialization: Guarded by storeMu to prevent races during model reload
• Health checks: Embedding server health is verified before every write operation

Performance Characteristics

Operation	Typical Latency
Chunk embedding (512 tokens)	~50ms
Vector search (10K documents)	<5ms
Full ingestion (1MB text)	~2s
Context injection	<1ms

---

For best memory quality, use the same embedding model consistently. Switching embedding models requires re-indexing all stored documents.