Inference Optimization

2026 State of AI Engineering — Barr Yaron, Amplify Partners

Jul 21, 2026 · 19:47

In the 2026 State of AI Engineering survey presented by Amplify Partners' Barr Yaron, 1,048 AI engineers reveal that cost is now a first-class engineering constraint—40% say it regularly shapes how ambitiously they use AI. Agents have exploded: 95% of teams now use agents, and 89% of those agents have write access, tripling from last year. Image generation adoption doubled to 36%, while audio shows the strongest intent-to-adopt at 56%. Open-weight models augment rather than replace closed models—45% use open-weight, but over 90% of them also use closed models. Evals remain the top infrastructure challenge, and inference is the most bought layer, while prompt management (61% built in-house) stays close to product logic. Teams report 97% net positive impact, but 59% fear long-term liabilities from AI code, and over a third say non-developers now ship features.

The Desktop Frontier — Ahmad Osman, Osmantic

Jul 21, 2026 · 18:02

Ahmad Osman, founder of Osmantic, argues that within roughly 18 months (by late 2027) a single RTX 5090 will run intelligence equivalent to GLM 5.2, driven by the Densing Law of increasing impact per parameter. He shows this trend through concrete examples: a 27B-parameter Qwen 3.5 now beats the 405B LLaMA 3, and the same eight RTX 3090s that once struggled with LLaMA 2 can now run 15 parallel Qwen 3.5 agents. Osman presents the Densing Law—every 3.5 months, 50% fewer parameters achieve the same capability—as a systematic pattern, not coincidence. He advocates for sovereign AI: owning your own hardware (like a DGX Station or RTX 5090) gives you control, avoids cloud limitations, and sees hardware appreciate in utility as models become more efficient. He asks why fund cloud data centers when local hardware can run frontier intelligence and grow more valuable over time.

State of the Union: Why Local, Why Now — NVIDIA, Osmantic, Roboflow, EXO Labs, @matthew_berman

Jul 11, 2026 · 44:29

Nader Khalil (NVIDIA), Joseph Nelson (Roboflow), Alex Cheema (Exo Labs), Matthew Berman, and Ahmad Osman (Osmantic, r/LocalLLaMA) argue that local AI is now useful, driven by stronger open models and better hardware. They cite inflection points like Llama 2, DeepSeek v3, and GLM 5.2, which closed the gap with frontier cloud models. Sovereignty and control are key: enterprises need to choose their own model versions and avoid lock-in. Specialized models, such as Roboflow's fine-tuned vision models for deep-sea fish discovery, outperform general ones for specific tasks. Optimization is critical: EXO Labs achieved 10x performance on the DGX Spark by tuning existing NVIDIA kernels. The panel emphasizes that simplicity remains a barrier—most users need point-and-click solutions—and advocates for open-source AI to ensure freedom and innovation.

You Might Not Need 50 Diffusion Steps — Ziv Ilan, Nvidia

Jun 16, 2026 · 18:46

Nvidia's Ziv Ilan explains how combining quantization, caching, and step distillation enables near real-time video diffusion on a single Blackwell B200 GPU. Working with Black Forest Labs on Flux 2, dynamic quantization reduces memory and compute, caching skips redundant denoising steps, and distillation cuts steps from fifty to as few as one. The open-source FastGen repo packages these post-training and sharding techniques, achieving 10–200x speedups for real-time generation.

Sovereign Escape Velocity: Ownership w Open Models — Gus Martins, & Ian Ballantyne, Google DeepMind

Jun 10, 2026 · 20:52

Gus Martins and Ian Ballantyne of Google DeepMind introduce Gemma 4, a family of open-weight models that deliver high quality per parameter, enabling deployment on a single GPU or even a phone. They argue that the models' efficiency — a 31B model rivals those twenty times larger — and the shift to Apache 2.0 licensing remove barriers for sovereign institutions like those in Ukraine, Bulgaria, and Brazil. Ian demonstrates multi-agent translation running locally on an M4 Mac, showcasing ownership and control over agentic workloads.

Text Diffusion — Brendan O’Donoghue, Google DeepMind

Jun 4, 2026 · 28:03

Brendan O'Donoghue, a research scientist at Google DeepMind, explains that text diffusion models generate tokens 10x faster than autoregressive models by performing 24 denoising steps to produce 256 tokens, dramatically reducing memory transfers. Unlike GPT-4o and Gemini 2.5 Flash, which incorrectly answered 40 and 42 on a math problem, Gemini Diffusion used bidirectional attention to self-correct from 60 to 49 to 39. The model adaptively allocates compute: 4 steps for memorized digits of π, 31 for quantum mechanics, and automatically stops when satisfied. Text diffusion also enables in-place editing, demonstrated by fixing code bugs or adding paragraphs. However, lower throughput on large batches makes it expensive to serve at scale today. O'Donoghue showcases low-latency applications: a fully generated Wikipedia, a Reddit clone with AI text and images, an on-the-fly operating system, and a to-do app built in 15 seconds by voice.

20 days of compute vs 7 hours: rethinking what state-of-the-art means — Bertrand Charpentier, Pruna

Jun 1, 2026 · 19:36

Bertrand Charpentier, cofounder and chief scientist at Pruna AI, argues that state-of-the-art is not a single model but multiple on a Pareto front that balances quality and efficiency. He highlights that public leaderboards disagree—Hunyuan ranks 10th on Artificial Analysis but 5th on Arena—and most models lose 40% of head-to-head battles, meaning the top-ranked model is wrong for nearly half of use cases. Evaluating a large model like ChatGPT image on Design Arena (26k battles, 62 seconds each) costs $5,000 and 20 days of compute, consuming energy equivalent to 400 marathons, while a fast compressed model completes the same evaluation in 7 hours for $265. Charpentier advocates plotting quality against latency or cost to find the frontier, which often surfaces small specialized models instead of large foundation models, with 20x efficiency differences at similar quality scores.

Run Frontier AI at Home — Alex Cheema, EXO Labs

May 26, 2026 · 1:45:02

Alex Cheema of EXO Labs argues that running frontier AI locally has 100x improvement potential in cost and performance, demonstrated with GLM 5.1—a trillion-parameter model—running across four Mac Studios at roughly 20 tokens per second for $40,000. He details kernel fusion that recovered 30% performance on Qwen 3.5 by eliminating unnecessary kernel launches, and RDMA integration that cut node-to-node latency from 300 microseconds to single digits, enabling tensor parallelism to actually scale. Cheema advocates splitting inference: prefill on compute-dense hardware (e.g., an RTX Spark) and decode on high-bandwidth hardware (e.g., Mac), cutting large-prompt inference roughly in half. He warns against misleading benchmarks like one-bit quantized models, and outlines how multi-agent setups, test-time scaling, and continual learning could further improve local inference efficiency. The talk includes a live demo of GLM 5.1 across four Mac Studios connected via Thunderbolt 5, and a preview of EXO's upcoming benchmarking site to track intelligence per Joule.

Scaling the Next Paradigm of Heterogeneous Intelligence — Adrian Bertagnoli, Callosum

May 24, 2026 · 15:13

Adrian Bertagnoli, founding engineer at Callosum, argues that the era of homogeneous intelligence—scaling single models on identical chips—is ending, and heterogeneous intelligence, which routes tasks to optimal chips and models, is the next paradigm. He demonstrates this with two case studies: on the Ulong benchmark, running recursive language models on Cerebras instead of GPT-5.2 cuts cost by 7x and latency by 5x while matching accuracy; on Video Web Arena, a mixture of Qwen 3 VL8B and Kimi K2.5 beats GPT-5.2 and Gemini 2.5 by 18% and 25%, while costing 3.7x less and running 3x faster. The key insight is that complex problems decompose into subtasks—like zooming on a webpage—which require different intelligence levels; offloading those to smaller models yields 11x speed and 43x cost improvements. Callosum builds an automation layer that predicts the best model and hardware for each subtask, and has secured a £3 million grant with the UK's Arya institute to operate the first heterogeneous co-located cluster.

Gemini Nano on device — Florina Muntenescu & Oli Gaymond, Google DeepMind

May 22, 2026 · 19:38

Florina Muntenescu and Oli Gaymond from Google DeepMind explain Gemini Nano, an on-device model that ships at 3–4 GB and is shared across apps via the AI Core system service, which handles scheduling, queuing background batch jobs overnight, and prioritizing foreground apps. The MLKit GenAI APIs (prompt API for text/image input, text output) give access to Gemini Nano, but require flagship devices from the last two years for optimal performance; classic MLKit (vision, OCR) runs on over a billion devices. Hybrid inference, launched weeks before this talk, automatically falls back to Gemini Flash in the cloud when the on-device model isn't available, extending reach. An embedding API for RAG-style solutions is coming soon. For fully custom models, LiteRT offers an alternative path but requires more developer effort for testing and optimization.

Why MLX — Prince Canuma, Neywa Labs

May 11, 2026 · 23:10

Prince Canuma argues that on-device AI, powered by Apple's MLX framework, is a viable alternative to cloud-dependent services, especially for users in regions with unreliable internet. MLX, an array framework for Apple Silicon, has reached 1.5 million downloads and 4000 ported models, including day-zero support for Gemma 4 and Qwen 3 Omni. Canuma demonstrates real-time vision models (e.g., RF Deter for object detection), sub-100ms text-to-speech via Marvis TTS, and modular speech-to-speech pipelines that run entirely on-device. Community projects showcased include a native voice app (Locally), a robot with real-time voice cloning, and a video generation system that chains coherent stories on 16GB VRAM. A recent breakthrough, Turbo Quant, reduces KV cache by 4x, enabling 1 million context windows on-device. Canuma emphasizes that these capabilities provide accessibility for his blind father and enable agents that hear, see, and respond without phoning home.

Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

May 5, 2026 · 23:58

Chintan Parikh and Weiyi Wang from Google DeepMind present Gemma 4E edge models (2B and 4B) and the LiteRT framework for on-device AI, arguing that edge AI delivers latency, privacy, offline capability, and cost savings. The Gemma 4E models introduce agentic capabilities including built-in function calling, structured JSON output, and chain-of-thought reasoning, all optimized for hardware-native support across CPUs, GPUs, and NPUs. LiteRT supports cross-platform deployment on Android, iOS, macOS, Linux, Windows, and IoT devices like Raspberry Pi, with a CLI tool and AI Edge portal for benchmarking. Performance benchmarks show up to 56 tokens per second on iOS, 30x speedups with NPU acceleration, and 35x faster than Llama on mobile. The Gallery app demonstrates on-device skills such as Wikipedia querying, mood tracking, photo-to-music generation, and voice agents, with open-source code and a Hugging Face repository. Q&A addresses use cases like local security camera face recognition, LiteRT vs TensorRT on Orin, multi-agent architectures, and audio model support.

Everything I Learned Training Frontier Small Models — Maxime Labonne, Liquid AI

Apr 29, 2026 · 20:13

Maxime Labonne, head of post-training at Liquid AI, presents the LFM2.5 recipe for training frontier small models, arguing they require specialized approaches distinct from scaled-down big models. He details how Liquid's architecture uses gated short convolutions for latency-sensitive on-device deployment, achieving faster throughput than Gemma 3 or Qwen 3.5. Post-training stages—SFT, on-policy preference alignment, and RL—are tailored for narrow task focus like data extraction and tool use. A key challenge is 'doom loops' (repetition), which reached 15% after mid-training in a 1.2B reasoning model; solutions include preference alignment rejecting looped responses and RL with verifiable rewards and n-gram penalties, nearly eliminating the issue. He advocates combining small models with agentic tools (e.g., web search) to overcome memory limits, as they excel at reasoning and tool use despite lower knowledge capacity. The talk also covers decisions on when to use small vs. large models—latency, privacy, offline use—and notes that distillation alone likely won't fully solve doom loops.

Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI

Apr 20, 2026 · 10:51

Adrien Grondin, developer of Locally AI, demonstrates how to run Gemma 4 and other LLMs on iPhone using Apple's MLX framework, achieving 40 tokens per second on the latest devices. He explains that MLX is optimized for Apple Silicon and that the open-source mlx-swift-lm GitHub repo enables easy integration into iOS, macOS, and iPadOS apps in under 10 minutes. Grondin recommends quantized models from the Hugging Face MLX community—typically 4-bit to 8-bit—and shows a live demo of Gemma 4 generating text offline. He confirms that mlx-swift-lm supports tool calling, though structured generation is not yet available. He also notes that Locally AI has been acquired by LM Studio, which now lets users run models via MLX or llama.cpp and connect them through OpenAI or Anthropic-compatible APIs.

Running LLMs locally: Practical LLM Performance on DGX Spark — Mozhgan Kabiri chimeh, NVIDIA

Apr 10, 2026 · 10:16

NVIDIA’s Mozhgan Kabiri Chimeh demonstrates that running LLMs locally on the DGX Spark workstation, powered by the GB10 Grace Blackwell superchip and 128GB unified memory, achieves practical performance for models up to 14B parameters. Using a reproducible vLLM benchmarking methodology, she shows that the 14B NVFP4 quantized model delivers 20.19 tokens per second and a time-to-first-token 3.4× faster than the unoptimized 14B base model. The DGX Spark supports the same NVIDIA AI software stack as production environments, enabling local development, fine-tuning, and privacy-sensitive workloads before scaling to the cloud. NVFP4 quantization is highlighted as critical for balancing intelligence and throughput on single-system setups.

Compilers in the Age of LLMs — Yusuf Olokoba, Muna

Nov 24, 2025 · 17:36

Yusuf Olokoba, founder of Muna, explains how they built a Python compiler that converts plain Python inference functions—like Google's 270M-parameter Gemma embedding model—into self-contained C++ and Rust binaries using LLMs within a verifiable pipeline. The process involves symbolic tracing to generate an intermediate representation, type propagation to infer types for native compilation, and LLM-driven code generation to mass-produce native implementations of Python operations. After compiling into a shared library, the model can be loaded via FFI from any language (e.g., Node.js) and exposed through an OpenAI-compatible client, enabling developers to run open-source models anywhere—locally, cloud, mobile—with minimal code changes. Olokoba details why they abandoned PyTorch FX due to its PyTorch-only focus and reliance on fake inputs, and how LLMs help scale the coverage of elementary operations. The talk argues that this compiler approach solves hybrid inference—small edge models working with large cloud models—by moving beyond Python and Docker to more portable, low-latency native binaries.

Context Platform Engineering to Reduce Token Anxiety — Val Bercovici, WEKA

Nov 24, 2025 · 23:52

WEKA's Val Bercovici and Callan Fox present Context Platform Engineering, open-sourcing a toolkit that maximizes KV cache hit rates—called the single most important metric for production-grade AI agents by Manus AI. They argue that without this engineering, users resort to Context Financial Engineering, a clairvoyant prompt cache arbitrage against time-to-live (5 minutes to 1 hour) and cache read/write pricing. The toolkit includes a load generator that configures agent swarms with specific SLOs, cycling through deterministic and random prompts. Callan's WEKA Labs research shows that 1-minute TTL causes 15-16x token repetition, while 1-hour TTL approaches 1x, but requires larger cache capacity. Benchmarks compare HBM+WEKA (purple) vs HBM+DRAM (orange) vs HBM+DRAM+slow storage (pink); WEKA's NVMe-backed augmented memory grid maintains higher output token rates under increasing concurrent users—up to the point where DRAM tiers drop off sharply due to insufficient speed. The episode covers how SLA requirements translate into SLOs via memory tiering and KV cache offloading, and emphasizes that subscription users effectively purchase cache allotments to keep inference providers in…

Hacking the Inference Pareto Frontier - Kyle Kranen, NVIDIA

Aug 1, 2025 · 20:25

Kyle Kranen, architect of NVIDIA Dynamo, explains how to break the inference Pareto frontier by manipulating trade-offs between quality, latency, and cost using a toolkit of techniques. Disaggregation separates prefill and decode phases, achieving up to 2× tokens per second per GPU at fixed latency for LLaMA 70B on 16 H100s. Smart routing maximizes KV cache hits, asymptotically reducing prefill work as deployments scale. Structure from agentic workloads, like inference-time scaling, enables smaller models (e.g., 8B queried 3–4 times) to match larger models' quality at lower cost. KV manipulation offloads caches during tool calls (e.g., 30-second delays) to avoid re-prefill. Dynamism adjusts worker specialization and autoscales prefill/decode ratios in real time to handle shifting user distributions, ensuring disaggregation reaches maximum potential.

Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing - Neil Dwyer, Gabber

Jul 31, 2025 · 16:09

Neil Dwyer, CTO of Gabber, details how his startup serves real-time voice AI for under $1 per hour using open-source Orpheus TTS, LoRAs for emotive voice cloning, and vLLM with FP8 dynamic quantization to achieve 95-105 tokens per second on L40S GPUs. He explains the critical latency challenge of 'head of line silence' in Orpheus (600ms in default voices) and how fine-tuning LoRAs reduces it to ~100ms, fitting within a 1.5-second budget for real-time conversation. Gabber batches multiple LoRA generations per GPU and uses a consistent hash ring for load balancing across servers, enabling popular clones to be dynamically replicated. The talk argues that with current open-source tools, building affordable consumer voice AI is accessible to small teams.

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Jul 26, 2025 · 43:42

Philip Kiely and Yineng Zhang introduce SGLang, an open-source fast serving framework for LLMs and VLMs, arguing it offers production-ready performance with day-zero support for new models like DeepSeek and Qwen, and a customizable codebase for contributions. Yineng, a core maintainer, traces SGLang's rapid growth from a December 2023 paper to nearly 15,000 GitHub stars and adoption by xAI, AMD, and Meituan. The workshop demonstrates deploying a first model via Baseten's trunk packaging, then tuning the CUDA Graph max batch size flag on an L4 GPU to maintain CUDA Graph-enabled decoding during higher concurrency, boosting generation throughput. They also cover Eagle 3 speculative decoding, where a draft model derived from the target model speculates tokens; users can benchmark different step and top-k configurations on representative prompts to find optimal settings for production. Finally, they invite contributions via GitHub's 'good first issue' tags and highlight Baseten's job openings.

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Jul 19, 2025 · 19:03

Keegan McCallum, Head of ML Infrastructure at Luma AI, details how the company's Dream Machine model scaled from 500 to 9,000 H100 GPUs within hours to handle 1 million users in four days, outpacing ChatGPT's initial growth. He explains that their initial Triton inference server setup was brittle and ill-suited for multi-GPU, multi-node video models, prompting a re-architecture to a custom serving stack on vanilla PyTorch. To solve work starvation across user tiers, they implemented an SLO-based aging system that ranks jobs by the percentage of their worst-case wait time elapsed. For managing dozens of model versions, they store immutable full Python environments and checkpoints in object storage, with a YAML file controlling active deployments and enabling zero-downtime rollouts across thousands of GPUs. McCallum also discusses partnerships with Nvidia, AMD, and Grok, and how Luma's broader mission is to build general multimodal intelligence that generates, understands, and operates in the physical world.

Teaching Gemini to Speak YouTube: Adapting LLMs for Video Recommendations to 2B+DAU - Devansh Tandon

Jul 16, 2025 · 22:51

Devansh Tandon, a Product Manager at Google leading YouTube's discovery system, details how YouTube adapted Gemini LLMs to power its recommendation engine for billions of daily active users. The team built SemanticID, a tokenization system that compresses video features into semantically meaningful tokens, creating a new language for YouTube content. They then continued pre-training Gemini on sequences of user watches to make the model bilingual in English and this video language. For generative retrieval, they prompt the adapted model with user demographics and watch history to output video recommendations as SemanticIDs, achieving 95%+ cost savings to serve at scale. Challenges include serving billions of users with low latency and handling video freshness—Taylor Swift's new music video must be recommendable within minutes. Tandon argues LLM-led recommendations are a bigger consumer application than search and hints at future interactive, steerable recommendations and even personalized content creation.

360Brew: LLM-based Personalized Ranking and Recommendation - Hamed and Maziar, LinkedIn AI

Jul 16, 2025 · 22:00

LinkedIn AI's Hamed Firooz and Maziar Sanjabi present BrewXL, a 150-billion-parameter foundation model for ranking and recommendation that personalizes the platform's feeds, jobs, and search. They show that training a large model then distilling to a 3B model outperforms training a small model from scratch. Scaling data, model size (from 7B to 8x22B), and context length all improve performance, though context beyond the trained length degrades accuracy. The model achieves zero-shot generalization on out-of-domain tasks, matching or beating task-specific production models, and significantly reduces the cold-start gap for users with fewer than five interactions. For serving, they combine gradual pruning, mixed-precision quantization (FP8 for most layers, FP32 for the LM head), and 4D attention masks to score up to 500 items without cross-attention, yielding a 7X latency reduction and 30X throughput increase per GPU.

What We Learned from Using LLMs in Pinterest — Mukuntha Narayanan, Han Wang, Pinterest

Jul 16, 2025 · 18:13

Pinterest search engineers Han Wang and Mukuntha Narayanan present four key learnings from integrating LLMs into the platform's search relevance system. Fine-tuned LLM cross-encoders improve relevance prediction by 12% over multilingual BERT and 20% over SearchSage using an 8B model. VLM-generated image captions and user action features enrich pin text representations and further boost performance. Knowledge distillation into a bi-encoder student model, trained on 100x more data via semi-supervised learning, enables production serving with real-time query embeddings and 85% cache hit rate. Relevance-tuned embeddings serve as general-purpose semantic representations across multiple surfaces, and the system yields relevance gains in multiple languages and countries despite a predominantly US training set.

Thinking Deeper in Gemini — Jack Rae, Google DeepMind

Jul 10, 2025 · 18:13

Jack Rae, lead of Gemini Thinking at Google DeepMind, presents thinking as a solution to the fixed test-time compute bottleneck in large language models. He explains that Gemini inserts a thinking stage where the model iterates via reinforcement learning, learning to self-correct and explore multiple strategies. This enables a continuous cost-performance tradeoff via thinking budgets, improving reasoning on math and code. Future work includes Deep Think, which raises USA Math Olympiad performance from the 50th to the 65th percentile by scaling inference compute further. Rae envisions models that, like mathematician Ramanujan, achieve deep, data-efficient reasoning from limited knowledge.

A year of Gemini progress + what comes next — Logan Kilpatrick, Google DeepMind

Jul 10, 2025 · 11:58

Logan Kilpatrick, head of product for Google AI Studio at DeepMind, announces the final update to Gemini 2.5 Pro, which achieves state-of-the-art results on Aider and HLE benchmarks. He details Google's 50x increase in AI inference over the past year, driven by merging research and product teams into DeepMind. The episode outlines Gemini's evolution toward a universal assistant that unifies Google products, with upcoming features including proactivity, native audio and video capabilities (Veo), and smaller models. Kilpatrick also previews developer-focused updates: a SOTA embeddings model, a deep research API, and Veo 3 and Imagine 4 in the API, alongside repositioning AI Studio as a dedicated developer platform.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

Optimizing inference for voice models in production - Philip Kiely, Baseten

Jul 1, 2025 · 15:13

Philip Kiely of Baseten shows how open-source TTS models like Orpheus TTS, built on a LLaMA 3.2 3B backbone, can be optimized for production inference using LLM tooling such as TensorRT-LLM and FP8 quantization, achieving time-to-first-byte (TTFB) under 150 milliseconds and supporting 16–24 simultaneous streams on a single half-H100 GPU. He argues that because TTS models are architecturally similar to LLMs, techniques like dynamic batching, KV-cache quantization, and Torch Compile on the audio decoder can dramatically reduce latency and increase concurrency. He emphasizes that for real-time voice agents, the goal is not raw tokens per second (real-time requires only 83 TPS for Orpheus) but low TTFB and high throughput to minimize GPU spend. However, Kiely warns that non-runtime factors—such as client code using sequential requests without session reuse or sending traffic to distant data centers—can easily add back the milliseconds saved at the model level, and that infrastructure connecting listening, thinking, and talking pipelines is often the dominant source of latency.

Serving Voice AI at Scale — Arjun Desai (Cartesia) & Rohit Talluri (AWS)

Jun 27, 2025 · 17:05

Arjun Desai of Cartesia AI and AWS's Rohit Talluri discuss scaling voice AI for enterprise, arguing that latency and controllability are critical, with Cartesia's state-space model Sonic 2 achieving 40ms model latency for real-time applications. Desai explains that traditional transformer models scale quadratically, while Cartesia's SSMs maintain O(1) generation, enabling 2.5x faster inference than their earlier models. He emphasizes that edge deployment is 5x faster than cloud round-trips, making local models essential for interactive use cases. On quality, Desai notes that voice AI must handle interruptions, accents, and background noise, and that Cartesia's voice marketplace amplifies human voice actors rather than replacing them. Looking to 2030, he predicts voice AI will become the default interface across healthcare, customer support, and gaming, with interactive models extending beyond audio to full world models.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.

Fun stories from building OpenRouter and where all this is going - Alex Atallah, OpenRouter

Jun 25, 2025 · 18:47

Alex Atallah, founder of OpenRouter, tells the story of how he launched the LLM aggregator in early 2023 after observing the open-source race sparked by Meta's Llama 1 and Stanford's Alpaca distillation under $600. He argues the inference market is not winner-take-all, citing OpenRouter's data showing Google Gemini growing from 2-3% to 34-35% of tokens over 12 months. The platform evolved from a simple model collection into a marketplace with over 400 models and 60 providers, solving architecture challenges like 30-millisecond latency, cancelable streams, and a middleware plugin system for web search and PDF parsing. Atallah explains that OpenRouter grew 10-100% month-over-month for two years, and he predicts future additions like transfusion models that generate images and more powerful geographic routing for enterprise optimization.

Luminal - Search-Based Deep Learning Compilers - Joe Fioti

Jun 3, 2025 · 24:35

Joe Fioti presents Luminal, a search-based deep learning compiler that simplifies ML libraries to 12 primitive operations and uses search to automatically discover optimized kernels like flash attention. By representing models as directed acyclic graphs of these simple ops, Luminal keeps its codebase under 5,000 lines yet can run all major models. Its compiler applies 20-25 rewrite rules to search through equivalent GPU kernels, profiling to find the fastest—automatically rediscovering flash attention, an algorithm that took five years for the industry to develop. Data movement accounts for 99% of runtime, so kernel fusion merges many ops into one, dramatically speeding execution. An external auto-grad crate adds training support without altering the core. Future plans include supporting AMD, TPUs, and a serverless cloud that exports optimized graphs for inference.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis

Feb 11, 2025 · 18:29

Dylan Patel of SemiAnalysis breaks down the inference challenges for next-generation frontier models like GPT-4 (1.8 trillion parameters) and upcoming models trained on 100,000+ GPU clusters. He emphasizes that prefill (prompt processing) is compute-intensive while decode (token generation) is memory bandwidth-intensive, creating a systems problem where serving 64 users at 30 tokens/second requires 60 terabytes/second of memory bandwidth. Patel details engineering strategies such as continuous batching to improve batch utilization by 10-100x, disaggregated prefill to isolate noisy neighbors and maintain time-to-first-token SLAs, and context caching (like Google's) to cache KV cache on CPU/storage instead of GPU memory, dramatically reducing prefill costs. He warns that open-source tools like LLaMA.cpp lack these optimizations, making high-performance serving of models like LLaMA 405b infeasible without libraries like vLLM or TensorRT-LLM. On scaling, Patel notes that 100,000 GPU clusters (e.g., Microsoft's Arizona data center consuming 150 MW) face reliability issues — optical transceivers fail every five minutes — and straggler chips (silicon lottery) can degrade training…

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

Jan 1, 2025 · 33:39

NVIDIA solutions architect Mark Moyou explains that LLM inference differs fundamentally from standard deep learning deployment, requiring careful management of KV Cache, attention mechanisms, and GPU memory to control cost. He details how tokens are processed: prefill computes attention across the entire prompt, then generation produces one token at a time, with KV Cache storing key-value pairs to avoid recomputation. Llama's 32 attention heads and FP8 quantization (halving memory with near-identical accuracy) are cited as key optimizations. Moyou emphasizes measuring time to first token, inter-token latency, and input/output sequence length distributions to size inference engines. He presents NVIDIA's TRT-LLM (model compilation for LLMs) and Triton inference server as tools to maximize throughput, and discusses how query patterns like long-input-short-output or short-input-long-output impact GPU utilization and deployment cost.

AI Frontiers in Trust and Safety Combatting Multifaceted Harm on Tinder at Scale: Vibhor Kumar

Dec 2, 2024 · 14:36

Vibhor Kumar, senior AI engineer at Tinder, explains how the company uses open-source LLMs and LoRAX to detect a long tail of trust and safety violations at global scale. Facing challenges like content pollution and automated fraud from generative AI, Tinder leverages pre-trained models such as LLaMA and Mistral, fine-tuning them with LoRA and QLoRA on hybrid datasets generated by GPT-4 and manually verified. They serve dozens of fine-tuned adapters on a single GPU using LoRAX, achieving real-time inference (tens of QPS, ~100ms latency) for categories including hate speech, pig butchering scams, and underage users. The approach yields near 100% recall on simpler tasks and significant improvements over baselines, with better generalization that resists adversarial evasion. Future directions include visual language models for explicit image detection and automating retraining pipelines.

A Practical Guide to Efficient AI: Shelby Heinecke

Nov 18, 2024 · 17:45

Shelby Heinecke, who leads an AI research team at Salesforce, presents five orthogonal dimensions for making AI models efficient: efficient architecture selection, pre-training, fine-tuning, inference, and prompting. She highlights the power of small models like Phi-3 (3.8B parameters outperforming a 7B model), mobile LLM (350M parameters on par with 7B after fine-tuning), and Octopus (2B fine-tuned Gemma exceeding GPT-4 on Android tasks). For efficient inference, she explains post-training quantization, showing 4-bit quantization nearly halves memory usage without performance loss (e.g., LLaMA models), but warns 3-bit can degrade quality. She recommends frameworks like LLaMA CBP and ONNX Runtime for quantization and introduces her team's open-source Mobile AI Bench for evaluating quantized models, including an iOS app to measure latency and battery drain. The central claim is that deploying AI in constrained environments—cloud, on-prem, or edge—demands efficiency, and these practical techniques bridge the gap from demo to production.

State Space Models for Realtime Multimodal Intelligence: Karan Goel

Oct 29, 2024 · 14:26

Karan Goel, founder of Cartesia, argues that state space models (SSMs) are key to real-time multimodal intelligence, offering cheaper, faster, and higher-quality alternatives to transformers for streaming applications like conversational voice and on-device assistants. He contrasts batch intelligence (cloud APIs) with streaming intelligence needed for low-latency tasks, emphasizing that SSMs compress information linearly rather than storing all tokens, enabling efficient long-context processing. Cartesia’s voice generation model achieves instant latency in the data center and is being optimized for on-device Mac and desktop deployment. Goel asserts that compression helps long-context tasks (e.g., 24-hour security footage analysis) more than retrieval, and that SSMs now match transformer quality while scaling better for multimodal data.

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

Sep 13, 2024 · 1:40:01

Philip Kiely and Pankaj Gupta of Baseten lead a workshop on TensorRT-LLM, NVIDIA's high-performance inference framework for LLMs, arguing its use delivers best-in-class throughput and latency on NVIDIA GPUs. They explain that TensorRT-LLM optimizes computational graphs via plugin kernels and in-flight batching, achieving 216 tokens per second and 180ms time to first token on Mistral 7B. The workshop demonstrates building an engine for TinyLlama 1.1B, including FP8 quantization that reduced engine size from 2 GB to 1.2 GB with minimal quality loss. They benchmark a deployed model, showing 7,000 total tokens per second at batch size 64 on an A10G. The presenters compare TensorRT-LLM favorably to VLLM for high-throughput production use and introduce Truss, Baseten's open-source packaging tool, alongside their managed platform for automatic scaling and fast cold starts.

Unlocking Developer Productivity across CPU and GPU with MAX: Chris Lattner

Jul 25, 2024 · 18:33

Chris Lattner, CEO of Modular, presents MAX, a unified AI framework that accelerates Gen AI inference by combining CPU and GPU programming into a single Pythonic model, and Mojo, a new programming language that extends Python to systems programming with 100–1000x speedups. Lattner argues that current fragmentation across PyTorch, ONNX, TensorRT, and hardware-specific libraries slows innovation, and MAX replaces the entire stack—including cuDNN and Intel MKL—with a consistent, compiler-driven approach. He demonstrates that MAX's Int4/Int6 quantization achieves 5x faster performance than llama.cpp on cloud CPUs, and that its GPU matrix multiplication beats NVIDIA's cuBLAS by up to 30%. Mojo enables developers to write for loops and tokenizers (e.g., for LLaMA 3) in Python-like syntax without dropping to C++ or Rust. MAX is free and available now for CPU inference; GPU support launches in September with early access via Discord.

Llamafile: bringing AI to the masses with fast CPU inference: Stephen Hood and Justine Tunney

Jul 16, 2024 · 17:25

Stephen Hood and Justine Tunney present Mozilla's Llamafile project, which turns AI model weights into single-file executables that run on any OS and CPU without installation, democratizing access to AI. They claim CPU inference can match GPU performance through techniques like outer-loop unrolling in matrix multiplication and using a GPU-like programming model with sync threads, achieving 30-500% speed increases. Justine demonstrates a summarization task where the optimized version completes in seconds versus the old version's many seconds. Hood announces the Mozilla Builders accelerator offering $100,000 in non-dilutive funding for open-source local AI projects, emphasizing that individuals and small groups can still make impactful contributions in AI.

Harnessing the Power of LLMs Locally: Mithun Hunsur

Nov 22, 2023 · 17:09

Mithun Hunsur presents llm.rs, a Rust library for running large language models locally, arguing it gives developers ownership, lower latency, and privacy compared to cloud APIs. He explains how quantization makes inference viable on consumer hardware, and shows that llm.rs supports architectures like LLaMA and Falcon through a unified interface. Practical code examples demonstrate customization, and community projects like LocalAI and LLMchain illustrate real-world use. Hunsur shares his own date-extraction pipeline, fine-tuning a small model with GPT-3 data to replace expensive cloud calls. He also cautions about hardware requirements, trade-offs between speed and quality, and ecosystem churn from rapid innovation.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.