BERT

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.

$1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpentero

Apr 16, 2026 · 43:53

Diego Carpentero argues that LLM-based attacks—Prompt Injection, Indirect Injection, Model Internals (gibberish suffix), RAG Poisoning, MCP Exploits, and Agentic Escalation—are now the baseline, not the exception, and that model alignment and human review alone are insufficient. He identifies the core problem as a Zero Trust Gap: LLMs natively lack separation between system controls and data, allowing adversaries to override decisions via malicious instructions in inputs or external content. To build a protective layer, Carpentero fine-tunes ModernBERT—a state-of-the-art encoder with Alternating Attention, Unpadding & Sequence Packing, RoPE, and FlashAttention—into a safety discriminator that classifies prompts as safe or unsafe in ~35 milliseconds with 85% accuracy, all for under a dollar. He walks through the fine-tuning pipeline using the IngetGuard dataset and demonstrates live detection of real attack examples from each vector.

On Engineering AI Systems that Endure The Bitter Lesson - Omar Khattab, DSPy & Databricks

Aug 6, 2025 · 19:12

Omar Khattab, research scientist at Databricks and creator of DSPy, argues that engineering AI systems that endure the rapid pace of model and technique changes requires decoupling high-level task definitions from low-level, swappable components. He reinterprets the Bitter Lesson—that scaling search and learning beats hand-coded domain knowledge—as a warning against premature optimization, which he equates to hard-coding at lower abstraction levels than justified. Khattab criticizes prompts as a poor programming abstraction that entangles task specs, formatting, and model-specific tricks, violating separation of concerns. Instead, he advocates for investing in three orthogonal concerns: natural language specs for things that can't be otherwise expressed, evals to capture true objectives, and code for reliable structure and control flow. DSPy realizes this philosophy through declarative signatures that define what an AI system should do, independent of which LLM, inference strategy (e.g., chain-of-thought, agent), or optimizer (e.g., reinforcement learning, prompt optimization) is used. The takeaway: avoid hand engineering at lower levels than current abstractions allow, and ride…

Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs - Eugene Yan, Amazon

Jul 16, 2025 · 20:54

Eugene Yan's keynote presents three innovations for recommendation systems: Semantic IDs, LLM-augmented data, and unified models. Kuaishou’s trainable multimodal Semantic IDs increased cold-start coverage by 3.6% and velocity by 3.5% by clustering content embeddings. Indeed used GPT-4 fine-tuning and distillation to filter bad job recommendations, reducing bad recs by 20% while boosting application rate 4% and cutting unsubscribes 5%. Spotify’s LLM-generated exploratory search queries drove a 9% increase in exploratory queries for new categories like podcasts. Netflix’s Unicorn unified ranker matched or exceeded specialized models across search and recommendations, while Etsy’s unified embeddings with a quality vector achieved a 2.6% sitewide conversion lift and 5% more search purchases.

The End of Awkward AI Transcriptions - Travis Bartley and Myungjong Kim

Jun 3, 2025 · 16:24

NVIDIA speech AI researchers Travis Bartley, Myungjong Kim, and Jae-hun detail how their Fast Conformer backbone powers both streaming (Parakeet) and multitask (Canary) models, achieving top-5 Hugging Face ASR leaderboard rankings by prioritizing customization over one-size-fits-all. They explain CTC/TDT for low-latency streaming and attention encoder-decoder for high-accuracy multitask models, with Subformer enabling unified speaker diarization and target-speaker ASR. Training uses open-source and proprietary data with pseudo-labeling via the Nemo toolkit. Deployment on NVIDIA Riva NIM with TensorRT optimization supports low-latency streaming and offline processing. Customization includes fine-tuning acoustic models, language models, punctuation, and inverse text normalization for domain-specific terms like medical and food ordering.

Rethinking how we Scaffold AI Agents - Rahul Sengottuvelu, Ramp

Mar 19, 2025 · 16:32

Rahul Sengottuvelu, Head of Applied AI at Ramp and co-founder of Cohere.io, argues that building AI agents should follow the 'bitter lesson' from AI research: systems that scale with compute outperform handcrafted, deterministic code. He illustrates this with Ramp's switching report agent, which ingests arbitrary CSV files from third-party card providers. Three approaches are compared: manually coding parsers for the 50 most common vendors, using LLMs only for column classification, and a fully LLM-driven method where the model writes and runs pandas code via a code interpreter, repeated 50 times in parallel. The last approach, though using 10,000x more compute, costs under a dollar and generalizes better, saving Ramp far more in failed transactions. Sengottuvelu also demonstrates a prototype email client where the backend is an LLM with access to a code interpreter and the user's Gmail token; the LLM renders the UI as Markdown and handles clicks by re-prompting itself, simulating a full web app without traditional backend code. He contends that as models exponentially improve, shifting more execution into 'fuzzy' LLM compute—rather than rigid code—lets builders ride the trend for…