MemoDocumentation
EN

System Architecture

Memo follows a two-process architecture: a headless Go backend and a Flutter desktop/mobile frontend. They communicate over REST + SSE on localhost:8090.

High-Level Architecture

┌──────────────────────────────────────────────────┐
│                  Flutter Frontend                  │
│  ┌─────────┐  ┌──────────┐  ┌──────────────────┐ │
│  │  Chat   │  │  Model   │  │  Settings (8 tabs)│ │
│  │  Screen │  │  Store   │  │  Provider, Memory, │ │
│  │         │  │          │  │  Agent, Orchestra  │ │
│  └────┬────┘  └────┬─────┘  └────────┬─────────┘ │
│       │            │                 │            │
│  ┌────┴────────────┴─────────────────┴──────────┐ │
│  │              Riverpod State Layer             │ │
│  │   chatProvider, memoryProvider, agentProvider  │ │
│  └───────────────────────┬──────────────────────┘ │
│                          │                        │
│  ┌───────────────────────┴──────────────────────┐ │
│  │           Dio HTTP / SSE Client              │ │
│  │       MemoApiClient (singleton)               │ │
│  └───────────────────────┬──────────────────────┘ │
└──────────────────────────┼────────────────────────┘
                           │ HTTP + SSE
                           │ localhost:8090
┌──────────────────────────┼────────────────────────┐
│                          ▼                         │
│                Go Backend (app.go)                 │
│  ┌──────────────────────────────────────────────┐ │
│  │           http.ServeMux Router               │ │
│  │        ~90 REST + SSE endpoints              │ │
│  └──────────────────────┬───────────────────────┘ │
│                         │                          │
│  ┌──────────────────────┴───────────────────────┐ │
│  │              AppBridge / FullBridge           │ │
│  │         Decouples handlers from App           │ │
│  └──────────────────────┬───────────────────────┘ │
│                         │                          │
│  ┌──────────────────────┴───────────────────────┐ │
│  │              Central App Struct               │ │
│  │  Orchestrates all subsystems                  │ │
│  └───┬───────┬───────┬──────┬──────┬────────────┘ │
│      │       │       │      │      │               │
│  ┌───┴─┐ ┌───┴─┐ ┌───┴─┐ ┌──┴──┐ ┌──┴──────────┐ │
│  │Memory│ │Agent│ │Prov.│ │Sync │ │Calendar...   │ │
│  │Store │ │Exec.│ │Rout.│ │Drive│ │29 packages   │ │
│  └─────┘ └─────┘ └─────┘ └─────┘ └──────────────┘ │
│                         │                          │
│  ┌──────────────────────┴───────────────────────┐ │
│  │              Data Layer (SQLite/FS)           │ │
│  │  data/memory/ sessions/ models/ providers.json│ │
│  └──────────────────────────────────────────────┘ │
└───────────────────────────────────────────────────┘

Process Design

Go Backend (main.go → internal/app/app.go)

The backend is a single Go binary that starts:

  1. Configuration — Loads config/config.yaml
  2. Database init — Opens SQLite databases, runs migrations, validates schemas
  3. Provider system — Loads provider configs, starts health-check goroutines
  4. llama.cpp manager — Optionally starts the bundled llama-server process
  5. Memory store — Initializes RAG store, prepares embedding model
  6. Web server — Registers ~90 routes on http.ServeMux, starts listening on :8090
  7. Background services — WhatsApp, cloud sync, proactive learning, calendar reminders, memory decay/consolidation, mood engine

All subsystems are initialized through the central App struct in internal/app/app.go.

Flutter Frontend (frontend/lib/main.dart)

The Flutter app renders the desktop UI and communicates with the backend:

  • Riverpod providers manage state and call MemoApiClient methods
  • Dio handles HTTP requests and SSE stream parsing
  • 8 Settings tabs configure every backend subsystem visually
  • Engine strip shows live status of active models and providers

Module Map — 29 Go Packages

Package Responsibility
internal/app/ Central orchestrator — chat, memory, LLM, agent flows
internal/webserver/ HTTP handlers, bridge pattern, SSE streaming
internal/memory/ RAG vector store — embedding, search, import/export
internal/provider/ LLM provider routing, API clients (10 providers), fallback
internal/orchestra/ Multi-model workflow — conductor, 8 specialist roles
internal/agent/ Tool-calling sandbox — executor, pipeline, permissions
internal/cloudsync/ Google Drive E2E sync — crypto, OAuth, transfer
internal/whatsapp/ WhatsApp bridge — client, store, message handling
internal/calendar/ Event store, reminders, intent-to-event bridge
internal/sessions/ Chat session persistence
internal/identity/ System prompt, persona styles
internal/config/ YAML configuration loading and validation
internal/skill/ opencode-compatible skill system
internal/mood/ Stochastic emotion engine
internal/database/ SQLite write queue, connection pooling
internal/encryption/ AES-256-GCM, PBKDF2, key management
internal/proactive/ Intent extraction, habit tracking, learning loop
internal/routine/ Routines — scheduled automations, per-device timezone
internal/swarm/ Memo Swarm (beta) — pooled-compute room host/worker
internal/anthropicapi/ Developer API Gateway — Anthropic-compatible local endpoint
internal/stats/ Usage Stats — persistent token/speed/model event store
internal/replcli/ Terminal CLI — REPL, @ file-mention, localized l10n

Data Flow by Feature

Chat Flow

User types message
  → Flutter ChatScreen
  → chatProvider.sendMessage(text)
  → MemoApiClient.postChatStream(body)   [POST /api/chat]
  → Go handler → App.Chat()
  → Build system prompt (identity + mood + skills)
  → Retrieve memories (RAG hybrid search)
  → Construct full prompt (system + memories + history + message)
  → Route to provider (local llama.cpp or cloud API)
  → Stream tokens via SSE
  → Flutter SSE parser updates chatProvider in real-time
  → After response: save session, store memory embedding

Memory Flow

New exchange complete
  → App.rememberExchange(userMsg, assistantReply)
  → Chunk message pair (300 words, 50-word overlap)
  → Compute embedding (nomic-embed-text-v1.5 via llama.cpp)
  → Write to memory.db (memories + memory_vec + memory_fts tables)
  → Atomic transaction across all three writes

Retrieval (before each response)
  → Compute embedding for user query
  → Vector search (vec0, cosine similarity)
  → FTS5 search (keyword match)
  → RRF merge (k=60)
  → Filter by importance threshold
  → Inject into system prompt as [Relevant Memories]

Agent Flow

User sends agent request
  → App.Chat() detects agent mode
  → agent.Executor runs pipeline
  → For each tool call:
      → Check permissions (6 policies: allow, deny, ask, allow-session, deny-session, timeout)
      → Show permission dialog if needed
      → Execute tool with 60s timeout
      → Validate paths (symlink protection)
      → Stream result back via tool_executing → tool_result events
  → After pipeline completes (or 20 iterations):
      → Final LLM call synthesizes all tool outputs
      → Return synthesized response

Orchestra Flow

User sends complex task
  → orchestra.Conductor plans sub-tasks
  → 8 specialist roles execute in parallel:
      planner, frontend, backend, bug_fixer,
      reviewer, security_auditor, devops, generalist
  → Each role uses its assigned provider + model
  → Progress events streamed as each specialist completes
  → Conductor synthesizes all outputs into final response

Proactive Learning Flow

User sends message
  → Keyword pre-filter (fast, no LLM call)
  → If intent-like keywords detected:
      → LLM classifies: plan / habit / none
      → Extracts: time, topic, recurrence
      → Calendar intent bridge creates event
      → Habits stored for future reference
  → Daily proactie loop:
      → Checks calendar for upcoming events
      → Checks habits for scheduled follow-ups
      → Generates suggestions via LLM
      → Sends to user via notification / chat

Cloud Sync Flow

Auto-sync trigger (N messages or manual)
  → cloudsync.Drive.Sync()
  → Read local data files (WAL checkpoint first)
  → Encrypt with AES-256-GCM (user passphrase or machine key)
  → Upload to Google Drive (OAuth2 token)
  → Store sync metadata

Restore / pull:
  → Download encrypted blobs from Drive
  → Decrypt with passphrase
  → Write to local data directory (atomic temp + rename)

Concurrency Model

The Go backend uses explicit concurrency patterns:

  • sync.RWMutex for shared state (memory store, provider configs, WhatsApp client)
  • Goroutines for background services (health checks, proactive loop, mood engine, decay cycle)
  • context.Context passed as function parameters — never stored in struct fields
  • database.DB.Write() queue serializes all SQLite writes to prevent database is locked
  • Webserver stops first on shutdown to drain in-flight requests before disconnecting subsystems


Never store context.Context in struct fields (except lifecycle goroutines). Always pass it as the first parameter to functions that need it.