MemoDocumentation
EN

Multimodal Capabilities

Memo supports vision and speech natively — all processing happens on-device with no data leaving your machine.

Vision Analysis

Memo can analyze images using multimodal GGUF models. Images are processed entirely locally.

Supported Workflow

User drags image ──▶ Base64 encode ──▶ Send to GGUF model ──▶ Stream response
  (local)            (in-memory)       (llama.cpp server)     (SSE tokens)

Image Input

  • Drag and drop an image into the chat input area
  • Paste from clipboard (Ctrl+V / Cmd+V)
  • File picker via the attachment button

Supported formats: PNG, JPEG, GIF, WebP, BMP.

Encoding

Images are encoded as Base64 in-memory before being sent to the multimodal model. The encoding is performed client-side (Flutter) and the raw bytes are passed directly to the local llama.cpp server via the Go backend. No image data is stored on disk unless you choose to embed it in the chat log.

// Client-side Base64 encoding
final bytes = await imageFile.readAsBytes();
final base64 = base64Encode(bytes);
// Sent to local backend over HTTP

Multimodal GGUF Models

Vision requires a multimodal GGUF model. Current recommendations:

Model Size Quality Notes
llava-v1.6-mistral-7b ~4 GB (Q4_K_M) Good Broad visual understanding
llava-v1.6-vicuna-7b ~4 GB (Q4_K_M) Good Fine-tuned on detailed captions
bakllava-1.5-7b ~4 GB (Q4_K_M) Decent Lightweight alternative

These models are available in the Factory model store filtered by the "Vision" tag.

Usage Example

User: [attaches photo of error dialog]
      What does this error mean and how do I fix it?

Memo: The error "ENOSPC: no space left on device" means your disk
      is full. Here's how to free up space...

Speech-to-Text (STT)

Voice input uses whisper.cpp, running as a local subprocess with no internet dependency.

Architecture

Microphone ──▶ Audio capture (WAV) ──▶ whisper.cpp ──▶ Transcribed text ──▶ Chat input
  (Flutter)      (16kHz mono)           (local binary)   (UTF-8 string)       (ready to send)

On-Device Recording

  • Audio is captured at 16 kHz mono via the Flutter audio plugin
  • Recording is held in RAM as raw PCM, then encoded to WAV
  • The WAV file is passed to whisper.cpp via stdin or a temp file (auto-cleaned)

Transcription Pipeline

whisper-cli \
  --model models/ggml-base.bin \
  --file /tmp/memo_audio.wav \
  --language auto \
  --output-txt
  • Model: ggml-base.bin (~142 MB, bundled or downloaded on first use)
  • Language detection: --language auto detects Turkish (TR) and English (EN) automatically
  • Latency: ~200–500ms per second of audio on modern CPU

Language Auto-Detection

whisper.cpp automatically identifies the spoken language from the audio signal. Memo's bundled model is optimized for:

Language Detection Accuracy Transcription Quality
Turkish (TR) High Very good
English (EN) High Excellent
Mixed TR/EN Good Good (code-switching supported)

Additional languages are available by downloading the ggml-large.bin model in the Factory.

UX Flow

  1. Hold the microphone button in the chat input
  2. Speak — a waveform visualization confirms audio capture
  3. Release — audio is sent to whisper.cpp
  4. Transcription appears in the chat input (editable before sending)
  5. Send — transcribed text is sent as a normal message


For best STT accuracy, use the ggml-small.bin or ggml-medium.bin model. The base model is optimized for speed but larger models offer better accuracy, especially for Turkish.

Roadmap

Feature Status
Vision: image analysis via GGUF Stable
Vision: multi-image comparison Planned
STT: on-device whisper.cpp Stable
STT: real-time streaming transcription Planned
TTS: on-device text-to-speech (piper) Planned
Vision: screen capture analysis Planned