Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy
Jun 28, 2026 · 45:48
Abed Matini from Ogilvy demonstrates bypassing the multimodal tax by building a local-first hybrid RAG system that converts documents to clean Markdown via Docling, eliminating cloud vision token overhead. Using Ollama (Qwen 2.5 0.5B), PostgreSQL with pgvector, and raw SQL with Reciprocal Rank Fusion, he implements a FAQ assistant for an employee handbook with four chunking strategies—heading-based, paragraph, fixed 512-char with 64% overlap, and sentence-based. The system combines dense embedding vectors and sparse keyword indices (BM25) in a single query, retrieves top 2 chunks, and uses Python functions for speed and testability. Guardrails block prompt injections and out-of-scope questions before reaching the LLM, and LangFuse tracks token usage and latency. Matini argues that a small, local model and code-controlled pipelines reduce hallucination and cost while maintaining full observability.
