ColBERT

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.

On Engineering AI Systems that Endure The Bitter Lesson - Omar Khattab, DSPy & Databricks

Aug 6, 2025 · 19:12

Omar Khattab, research scientist at Databricks and creator of DSPy, argues that engineering AI systems that endure the rapid pace of model and technique changes requires decoupling high-level task definitions from low-level, swappable components. He reinterprets the Bitter Lesson—that scaling search and learning beats hand-coded domain knowledge—as a warning against premature optimization, which he equates to hard-coding at lower abstraction levels than justified. Khattab criticizes prompts as a poor programming abstraction that entangles task specs, formatting, and model-specific tricks, violating separation of concerns. Instead, he advocates for investing in three orthogonal concerns: natural language specs for things that can't be otherwise expressed, evals to capture true objectives, and code for reliable structure and control flow. DSPy realizes this philosophy through declarative signatures that define what an AI system should do, independent of which LLM, inference strategy (e.g., chain-of-thought, agent), or optimizer (e.g., reinforcement learning, prompt optimization) is used. The takeaway: avoid hand engineering at lower levels than current abstractions allow, and ride…