Llama

The Desktop Frontier — Ahmad Osman, Osmantic

Jul 21, 2026 · 18:02

Ahmad Osman, founder of Osmantic, argues that within roughly 18 months (by late 2027) a single RTX 5090 will run intelligence equivalent to GLM 5.2, driven by the Densing Law of increasing impact per parameter. He shows this trend through concrete examples: a 27B-parameter Qwen 3.5 now beats the 405B LLaMA 3, and the same eight RTX 3090s that once struggled with LLaMA 2 can now run 15 parallel Qwen 3.5 agents. Osman presents the Densing Law—every 3.5 months, 50% fewer parameters achieve the same capability—as a systematic pattern, not coincidence. He advocates for sovereign AI: owning your own hardware (like a DGX Station or RTX 5090) gives you control, avoids cloud limitations, and sees hardware appreciate in utility as models become more efficient. He asks why fund cloud data centers when local hardware can run frontier intelligence and grow more valuable over time.

Stop Evaluating Models Like It's the 50s - Alejandro Vidal, Mindmakers

Jul 13, 2026 · 23:35

Alejandro Vidal of Mindmakers argues that counting correct answers in LLM benchmarks—classical test theory—should be replaced by item response theory (IRT) from psychometrics for richer evaluations. Using real data from Epoch.ai, he shows that while Claude Opus 4.1 scored 245 right and Gemini 3 Pro 247, IRT reveals Gemini is nearly one standard deviation more intelligent because it answers harder questions. IRT assigns each item difficulty (B) and discrimination (A), enabling benchmark auditing—Vidal flagged mislabeled items like a question about passengers whose gold answer was actually total people killed. He demonstrates reducing a 484-item benchmark to 97 items with 99% ranking correlation by selecting high-discrimination items, saving tokens and money. IRT also detects contamination via unexpected residuals, protects benchmarks through adaptive testing with unique fingerprint sets, and identifies model families (e.g., DeepSeek distillations correlate 0.38). Vidal previews future extensions like multidimensional models and alignment measurement.

State of the Union: Why Local, Why Now — NVIDIA, Osmantic, Roboflow, EXO Labs, @matthew_berman

Jul 11, 2026 · 44:29

Nader Khalil (NVIDIA), Joseph Nelson (Roboflow), Alex Cheema (Exo Labs), Matthew Berman, and Ahmad Osman (Osmantic, r/LocalLLaMA) argue that local AI is now useful, driven by stronger open models and better hardware. They cite inflection points like Llama 2, DeepSeek v3, and GLM 5.2, which closed the gap with frontier cloud models. Sovereignty and control are key: enterprises need to choose their own model versions and avoid lock-in. Specialized models, such as Roboflow's fine-tuned vision models for deep-sea fish discovery, outperform general ones for specific tasks. Optimization is critical: EXO Labs achieved 10x performance on the DGX Spark by tuning existing NVIDIA kernels. The panel emphasizes that simplicity remains a barrier—most users need point-and-click solutions—and advocates for open-source AI to ensure freedom and innovation.

Frontier results, on device - RL Nabors, Arize

Jun 29, 2026 · 30:52

RL Nabors (Arize) argues that most frontier-model calls can be replaced by smaller, local models, saving cost, latency, and energy. She presents a four-step framework: prototype big with a foundation model, collect a golden dataset, run capability evals using Arize's open-source Phoenix, then select the 'sage' (small and good enough) model. Demonstrating with her social app Mima, she tested Qwen 2.5, Qwen 3, LLaMA 3.2, and Gemma 4 against Claude Sonnet on summary accuracy, latency, and cost. LLaMA 3.2 (3B params) won at 90% accuracy and 1-second P50 latency, versus Gemma 4's 8 seconds. Prompt engineering—specifically few-shot prompting—closed the gap further, achieving 92.9% factual consistency and 100% JSON validity. Nabors emphasizes running regression evals to prevent regressions, and notes that on-device inference eliminates data exposure and round-trip latency.

Personalization in the Era of LLMs - Shivam Verma, Spotify

May 19, 2026 · 20:12

Shivam Verma of Spotify's AI foundation explains how Spotify personalizes recommendations using open-weight LLMs, user embeddings from 750 million accounts, and Semantic IDs that compress each of 100 million tracks into six hierarchical tokens, enabling autoregressive next-item prediction. Three components: transformer-based user embeddings, Semantic IDs (e.g., Ariana Grande and Bruno Mars share first two tokens as pop artists), and a soft tokenization layer projecting user vectors into the LLM's token space for personalization without fine-tuning. The system is productionized for podcast next-episode recommendations. Verma also introduces Taste Profile, allowing users to edit their taste profile via natural language for steerable recommendations. This moves from traditional multi-stage pipelines to a unified generative model.

Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

May 5, 2026 · 23:58

Chintan Parikh and Weiyi Wang from Google DeepMind present Gemma 4E edge models (2B and 4B) and the LiteRT framework for on-device AI, arguing that edge AI delivers latency, privacy, offline capability, and cost savings. The Gemma 4E models introduce agentic capabilities including built-in function calling, structured JSON output, and chain-of-thought reasoning, all optimized for hardware-native support across CPUs, GPUs, and NPUs. LiteRT supports cross-platform deployment on Android, iOS, macOS, Linux, Windows, and IoT devices like Raspberry Pi, with a CLI tool and AI Edge portal for benchmarking. Performance benchmarks show up to 56 tokens per second on iOS, 30x speedups with NPU acceleration, and 35x faster than Llama on mobile. The Gallery app demonstrates on-device skills such as Wikipedia querying, mood tracking, photo-to-music generation, and voice agents, with open-source code and a Hugging Face repository. Q&A addresses use cases like local security camera face recognition, LiteRT vs TensorRT on Orin, multi-agent architectures, and audio model support.

Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

May 4, 2026 · 1:21:26

Angelos Perivolaropoulos from ElevenLabs walks through building a small GPT-2-like LLM from scratch on a local machine, demonstrating that the core techniques used by major labs are accessible in a few hundred lines of PyTorch code. The workshop uses character-level tokenization (65 tokens) on a Shakespeare dataset to enable fast training with limited compute. The model architecture includes multi-head self-attention, MLP layers, residual connections, and layer normalization, totaling 10 million parameters across six transformer blocks with a 256-token context window. The training loop employs next-token prediction with a warm-up cosine decay learning rate schedule and validation loss to detect overfitting. Inference uses temperature sampling (default 0.7) and top-k sampling to improve creativity. Perivolaropoulos explains that audio and multimodal models share the same transformer foundation but differ in tokenization (e.g., mel-spectrograms for audio) and use specialized losses like L2 or KL divergence, while reasoning models result from post-training base models with high-quality chain-of-thought data.

AI Kernel Generation: What's working, what's not, what's next – Natalie Serrino, Gimlet Labs

Dec 17, 2025 · 19:15

Natalie Serrino, cofounder of Gimlet Labs, presents how AI-generated kernels can automatically speed up custom PyTorch code by up to 24% on Apple M4 hardware using the Metal framework, with a 40% speedup from kernel fusion. The agentic system iterates through compilation, execution, correctness, and optimization, but faces challenges like validation of floating-point results and reliable benchmarking. Successes include rewriting average pool 1D as a convolution for 80% improvement, while failures occur on heavily optimized ops like matrix multiply. A real-world audio encoder model saw 70% faster inference on RTX 6000 Blackwell via six custom fused kernels. Serrino emphasizes that AI is best for rapidly searching optimizations and porting code to new hardware, not for surpassing human experts on novel algorithms.

Efficient Reinforcement Learning – Rhythm Garg & Linden Li, Applied Compute

Dec 9, 2025 · 20:19

Rhythm Garg and Linden Li, co-founders of Applied Compute, describe how their company uses efficient reinforcement learning (RL) to specialize large language models for enterprise tasks. They explain that synchronous RL wastes GPU time waiting on straggler samples—99% of arithmetic problems complete in ~40 seconds, but the tail takes 80 more seconds—so they adopt asynchronous pipeline RL. This method dedicates fixed GPUs to sampling and training, allowing continuous inference but introducing stale tokens (up to a tolerated staleness threshold) that require importance ratio corrections. To balance speed and stability, they model the system mathematically: using a roofline-based latency curve for sampling, per-GPU training throughput, and constraints on staleness and KV cache memory. Their simulations, parameterized by response length distributions, reveal an optimal GPU allocation that yields ~60% speedup over synchronous RL while keeping staleness within ML limits. This modeling lets them predict runtime and configure runs without expensive trial-and-error.

Building an Agentic Platform — Ben Kus, CTO Box

Aug 24, 2025 · 19:06

Ben Kus, CTO of Box, argues that building an agentic architecture early was critical to overcoming the limitations of pure LLM-based data extraction for enterprise unstructured content. Starting with simple LLM calls in 2023, Box faced accuracy issues on complex documents, OCR failures, and difficulty handling hundreds of fields. They pivoted to an agentic framework using directed graphs, multi-model voting, and LM-as-a-judge with reflection loops, which dramatically improved accuracy and allowed incremental improvement. This architecture also enabled deep research on customer data and clean separation between agentic logic and system scaling. Kus emphasizes that agentic thinking should be applied early, even to seemingly non-agentic tasks like metadata extraction, and that Box avoids fine-tuning in favor of prompt-based agentic orchestration.

Fuzzing in the GenAI Era — Leonard Tang, Haize Labs

Aug 22, 2025 · 19:12

Leonard Tang of Haize Labs argues that standard evaluation methods fail for GenAI systems due to brittleness (Lipschitz discontinuity), and proposes 'Haizing'—fuzz testing that simulates diverse inputs to uncover corner cases. He details two core problems: scoring outputs via 'judges', where Haize's Verdict library stacks GPT-4 Mini in a self-verified debate ensemble to beat O1 at a third the cost, and RL-tuned judges like a 1.7B parameter model achieving 80.7% on RewardBench. For stimuli generation, he frames it as discrete optimization over natural language, using gradient-based methods and tree search. Case studies include Haizing the largest Hungarian bank's loan calculator, discovering prompt injections in minutes, and boosting a voice agent's ground-truth human agreement by 38% using rubric fanout.

Building Agents at Cloud Scale — Antje Barth, AWS

Aug 2, 2025 · 19:00

AWS Principal Developer Advocate Antje Barth demonstrates how to build and scale AI agents using cloud-native patterns, arguing that specialized agents will reinvent customer experiences. She showcases Alexa Plus, which orchestrates hundreds of expert systems across 600M+ devices and tens of thousands of services, and the Amazon Q Developer CLI agent, shipped in just three weeks. Barth introduces Strands Agents, an open-source Python SDK for building production-ready agents that supports multiple model providers (Claude, Llama, OpenAI) and over 20 prebuilt tools including memory, RAG, and multi-agent workflows. She demonstrates integrating MCP servers via Lambda with DynamoDB for session storage, and previews upcoming A2A protocol support and a future of personal agents connecting to agent stores.

Pipecat Cloud: Enterprise Voice Agents Built On Open Source - Kwindla Hultman Kramer, Daily

Jul 31, 2025 · 26:46

Kwindla Hultman Kramer, co-founder of Daily, introduces Pipecat, an open-source, vendor-neutral framework for building voice AI agents, and Pipecat Cloud, a deployment layer optimized for real-time voice. He argues that achieving sub-800-millisecond voice-to-voice response times is critical and that frameworks like Pipecat handle hard problems like turn detection, interruption handling, and context management. Kramer explains that Pipecat supports 60+ models and services, including Gemini and OpenAI, and that Gemini is often 10x cheaper for 30-minute conversations. He notes that while speech-to-speech models like Moshi and Sesame are promising for natural conversation, they currently lag in instruction-following for enterprise use cases. Kramer also addresses global latency challenges, recommending deployment close to inference servers or using open-weight models locally, and highlights Pipecat Cloud's integration with Krisp for background noise reduction and a free open-source smart turn model.

Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing - Neil Dwyer, Gabber

Jul 31, 2025 · 16:09

Neil Dwyer, CTO of Gabber, details how his startup serves real-time voice AI for under $1 per hour using open-source Orpheus TTS, LoRAs for emotive voice cloning, and vLLM with FP8 dynamic quantization to achieve 95-105 tokens per second on L40S GPUs. He explains the critical latency challenge of 'head of line silence' in Orpheus (600ms in default voices) and how fine-tuning LoRAs reduces it to ~100ms, fitting within a 1.5-second budget for real-time conversation. Gabber batches multiple LoRA generations per GPU and uses a consistent hash ring for load balancing across servers, enabling popular clones to be dynamically replicated. The talk argues that with current open-source tools, building affordable consumer voice AI is accessible to small teams.

The Unofficial Guide to Apple’s Private Cloud Compute - Jmo, CONFSEC

Jul 30, 2025 · 20:36

Jonathan Mortensen explains Apple's Private Cloud Compute (PCC), a paradigm shift in confidential computing that delivers cryptographically provable privacy for AI inference on remote servers. He details Apple's five requirements—stateless computation, enforceable guarantees, non-targetability, no privileged runtime access, and verifiable transparency—and six technical components, including oblivious HTTP, blind signatures, secure enclave, remote attestation, transparency logs, and secure boot. The talk highlights how remote attestation allows an iPhone to verify server software and tie encryption to that verified state, while the transparency log enables public auditing of all deployed binaries. Mortensen notes trade-offs: complete trust in Apple's supply chain, higher latency and compute costs, and no support for custom models or third-party developers. He points to alternative tools like TPMs, SigStore, and confidential VMs for non-Apple environments, and concludes that Apple's approach is being adopted by Azure AI and Meta.

Why you should care about AI interpretability - Mark Bissell, Goodfire AI

Jul 27, 2025 · 21:11

Mark Bissell of Goodfire AI argues that mechanistic interpretability—reverse engineering neural networks—has moved from research to practical use cases that AI engineers can apply today, demonstrated through Goodfire's Ember platform for neural programming. He shows how Ember enables debugging and steering models at the neuron level, such as turning up a 'sensitive information' feature to make LLaMA refuse to reveal an email, or dynamically injecting a Coca-Cola recommendation when a beverage feature activates. For image models, the Paint with Ember demo (paint.goodfire.ai) lets users paint concepts like pyramid or lion face directly onto a canvas and steer sub-features (e.g., lion minus mane becomes tiger). Beyond interfaces, interpretability powers model diffs to detect unwanted changes after fine-tuning, guardrails for production systems, and scientific extraction: Goodfire works with the ARC Institute to uncover biological principles from the superhuman genomics model Evo2. Efficiency gains also emerge by pruning unnecessary weights for specialized tasks, making interpretability a critical tool for reliable AI engineering.

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Jul 26, 2025 · 43:42

Philip Kiely and Yineng Zhang introduce SGLang, an open-source fast serving framework for LLMs and VLMs, arguing it offers production-ready performance with day-zero support for new models like DeepSeek and Qwen, and a customizable codebase for contributions. Yineng, a core maintainer, traces SGLang's rapid growth from a December 2023 paper to nearly 15,000 GitHub stars and adoption by xAI, AMD, and Meituan. The workshop demonstrates deploying a first model via Baseten's trunk packaging, then tuning the CUDA Graph max batch size flag on an L4 GPU to maintain CUDA Graph-enabled decoding during higher concurrency, boosting generation throughput. They also cover Eagle 3 speculative decoding, where a draft model derived from the target model speculates tokens; users can benchmark different step and top-k configurations on representative prompts to find optimal settings for production. Finally, they invite contributions via GitHub's 'good first issue' tags and highlight Baseten's job openings.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments - Randall Hunt, Caylent

Jul 23, 2025 · 19:16

Randall Hunt from Caylent shares hard lessons from over 200 enterprise GenAI deployments, arguing that evals, embeddings, and prompt engineering matter far more than fine-tuning. He emphasizes that speed and UX are critical; a slow inference kills adoption, while techniques like generative UI and caching can mitigate latency. Hunt details real-world examples: using audio amplitude spectrographs for sports highlight reels, pooling multimodal embeddings for nature footage search, and noting that nurses prefer chat over voice bots in noisy hospitals. He reports zero regressions moving from Claude 3.7 to 4, and advises optimizing context and economics, such as leveraging Amazon Bedrock batch for 50% cost reduction. The talk underscores that knowing your end customer and minimizing irrelevant context are key to production success.

HybridRAG: A Fusion of Graph and Vector Retrieval - Mitesh Patel, NVIDIA

Jul 22, 2025 · 20:24

Mitesh Patel, Developer Advocate Manager at NVIDIA, presents HybridRAG, a fusion of knowledge graph-based GraphRAG and vector-based VectorRAG for improved question-answering from complex texts. He emphasizes that ontology engineering consumes 80% of development time and is critical for accurate triplet extraction, where fine-tuning LLaMA 3.1 with LoRA boosted triplet accuracy from 71% to 87% on 100 documents. Patel also highlights retrieval strategies like multi-hop graph traversal, which provides richer context but increases latency, and recommends QGraph acceleration via Networx to reduce latency. For evaluation, he suggests RAGAS for end-to-end pipeline metrics and the LLaMA-Nimotron reward model for response quality. Ultimately, he advises using GraphRAG when data has inherent structure or complex relationships, but notes it is compute-heavy, so the choice between GraphRAG, semantic RAG, or hybrid depends on the use case.

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

Transforming search and discovery using LLMs — Tejaswi & Vinesh, Instacart

Jul 16, 2025 · 21:10

Vinesh Gudla and Tejaswi Tenneti from Instacart detail how they use LLMs to overhaul search and discovery. They address challenges with conventional search: broad queries suffer cold start, tail queries lack engagement data. Using LLMs with Instacart's domain knowledge—e.g., top-converting categories as context—they improved precision by 18 percentage points and recall by 70% for tail queries, and cut zero-result queries. For discovery, LLMs generate complementary/substitute items; but pure LLM suggestions like 'chicken' for 'protein' missed user intent, so they augmented prompts with behavioral data (top categories, subsequent queries) to boost engagement and revenue. They use a hybrid approach: pre-compute offline for head/torso, distilled Llama 8B for long-tail, and LLMs as judges for evaluation. Key takeaways: combining LLM world knowledge with domain-specific data is critical, and evaluation is as hard as generation.

Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs - Eugene Yan, Amazon

Jul 16, 2025 · 20:54

Eugene Yan's keynote presents three innovations for recommendation systems: Semantic IDs, LLM-augmented data, and unified models. Kuaishou’s trainable multimodal Semantic IDs increased cold-start coverage by 3.6% and velocity by 3.5% by clustering content embeddings. Indeed used GPT-4 fine-tuning and distillation to filter bad job recommendations, reducing bad recs by 20% while boosting application rate 4% and cutting unsubscribes 5%. Spotify’s LLM-generated exploratory search queries drove a 9% increase in exploratory queries for new categories like podcasts. Netflix’s Unicorn unified ranker matched or exceeded specialized models across search and recommendations, while Etsy’s unified embeddings with a quality vector achieved a 2.6% sitewide conversion lift and 5% more search purchases.

How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS - Ishan Anand

Jul 13, 2025 · 1:41:34

Ishan Anand shows that GPT-2 small implemented in 600 lines of vanilla JavaScript makes LLMs understandable for web developers without ML backgrounds. He explains tokenization via byte-pair encoding, 768-dimensional embeddings representing semantic meaning via co-occurrence, and the Transformer's attention mechanism that lets tokens share context. The multi-layer perceptron learns next-token prediction through backpropagation, while the language head converts embeddings to token probabilities using softmax. Anand demonstrates each step—tokenization, embedding lookup, positional encoding, attention, MLP, and output—in a browser debugger, and notes that GPT-2's architecture underpins ChatGPT, with innovations like scale, supervised fine-tuning, and RLHF. The workshop provides an intuitive mental model of Transformers, turning perceived AI magic into understandable machinery.

2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison

Jul 9, 2025 · 18:30

Simon Willison reviews the past six months of LLM releases — including AWS Nova, Llama 3.3 70B, DeepSeek R1, Mistral Small 3, Claude 3.7 Sonnet, GPT 4.5, Gemini 2.5 Pro, GPT-4o, Llama 4, GPT 4.1, O3/O4 Mini, and Claude 4 — using his 'pelican on bicycle' SVG benchmark to argue that local models have become good enough to run GPT-4 class models on a laptop and that combining tools with reasoning is the most powerful technique in AI engineering, while noting risks like prompt injection and the 'lethal trifecta'. He tracks 30 significant model releases, highlighting that Mistral Small 3 (24B) matches Llama 3 70B's performance, which itself matched the 405B model, enabling local inference. DeepSeek's R1 caused a $500B+ Nvidia stock drop on January 27. GPT 4.1 Nano is the cheapest model yet at a fraction of a cent per pelican. He also examines bugs: ChatGPT's sycophantic 'shit-on-a-stick' incident and Claude 4's tendency to snitch to authorities when given ethical instructions and email tools. Willison concludes that while the pace is accelerating, control over context and security remain critical.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

Optimizing inference for voice models in production - Philip Kiely, Baseten

Jul 1, 2025 · 15:13

Philip Kiely of Baseten shows how open-source TTS models like Orpheus TTS, built on a LLaMA 3.2 3B backbone, can be optimized for production inference using LLM tooling such as TensorRT-LLM and FP8 quantization, achieving time-to-first-byte (TTFB) under 150 milliseconds and supporting 16–24 simultaneous streams on a single half-H100 GPU. He argues that because TTS models are architecturally similar to LLMs, techniques like dynamic batching, KV-cache quantization, and Torch Compile on the audio decoder can dramatically reduce latency and increase concurrency. He emphasizes that for real-time voice agents, the goal is not raw tokens per second (real-time requires only 83 TPS for Orpheus) but low TTFB and high throughput to minimize GPU spend. However, Kiely warns that non-runtime factors—such as client code using sequential requests without session reuse or sending traffic to distant data centers—can easily add back the milliseconds saved at the model level, and that infrastructure connecting listening, thinking, and talking pipelines is often the dominant source of latency.

Ship it! Building Production Ready Agents — Mike Chambers, AWS

Jun 27, 2025 · 19:37

Mike Chambers, a developer advocate at AWS, demonstrates how to take a simple local agent—built with a Llama 3.1 8B model and a dice-rolling tool—and ship it to production at cloud scale using Amazon Bedrock Agents. He breaks down the essential components of an agent: model, prompt, loop, history, and tools. Then he live-deploys the agent with Bedrock Agents, configuring instructions and an action group wired to an AWS Lambda function that handles the dice roll. The fully managed service automates scaling, infrastructure, and the agentic loop. Chambers also highlights free courses on DeepLearning.AI and invites attendees to discuss MCP servers and the new open-source SDK for model-first agents.

Building Code First AI Agents with Azure AI Agent Service — Cedric Vidal, Microsoft

Jun 27, 2025 · 1:54:06

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates building code-first AI agents with Azure AI Agent Service, using function calling, code interpreter, file search, and Bing grounding for a sales analysis use case. He creates a conversational agent that queries SQLite, generates pie charts from Python code, and blends product PDF data with relational data. The workshop explains stateful agents, tool routing (LLM generates JSON to call functions), and the limits of single-step agents versus multi-agent orchestration with AutoGen. Cedric addresses when to use Agent Service (managed persistence and tools) versus raw LLM endpoints, and covers MCP servers as a tool lifecycle manager. Key insights include using instructions to ground agent behavior and the need for eval frameworks for agent quality.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.

Fun stories from building OpenRouter and where all this is going - Alex Atallah, OpenRouter

Jun 25, 2025 · 18:47

Alex Atallah, founder of OpenRouter, tells the story of how he launched the LLM aggregator in early 2023 after observing the open-source race sparked by Meta's Llama 1 and Stanford's Alpaca distillation under $600. He argues the inference market is not winner-take-all, citing OpenRouter's data showing Google Gemini growing from 2-3% to 34-35% of tokens over 12 months. The platform evolved from a simple model collection into a marketplace with over 400 models and 60 providers, solving architecture challenges like 30-millisecond latency, cancelable streams, and a middleware plugin system for web search and PDF parsing. Atallah explains that OpenRouter grew 10-100% month-over-month for two years, and he predicts future additions like transfusion models that generate images and more powerful geographic routing for enterprise optimization.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

Luminal - Search-Based Deep Learning Compilers - Joe Fioti

Jun 3, 2025 · 24:35

Joe Fioti presents Luminal, a search-based deep learning compiler that simplifies ML libraries to 12 primitive operations and uses search to automatically discover optimized kernels like flash attention. By representing models as directed acyclic graphs of these simple ops, Luminal keeps its codebase under 5,000 lines yet can run all major models. Its compiler applies 20-25 rewrite rules to search through equivalent GPU kernels, profiling to find the fastest—automatically rediscovering flash attention, an algorithm that took five years for the industry to develop. Data movement accounts for 99% of runtime, so kernel fusion merges many ops into one, dramatically speeding execution. An external auto-grad crate adds training support without altering the core. Future plans include supporting AMD, TPUs, and a serverless cloud that exports optimized graphs for inference.

Text-to-Speech Data Preparation and Fine-tuning Workshop - Ronan McGovern

Jun 3, 2025 · 34:00

Ronan McGovern walks through fine-tuning Sesame's CSM-1B text-to-speech model on a specific voice, using a YouTube video as the data source. He explains token-based TTS models, including how audio is represented via codebooks and how CSM-1B uses a main transformer for zeroth tokens and a secondary transformer for 31 hierarchical tokens. The workshop covers data preparation: downloading audio with yt-dlp, transcribing with Whisper Turbo, manually correcting the transcript, and splitting audio into 30-second chunks (41 clips from a 30-minute video). Fine-tuning uses Unsloth with LoRA adapters (rank 32, alpha 16) on linear layers, training for one epoch with a batch size of 2 and virtual size of 8, reducing loss from ~6.34 to ~3.72. Evaluation compares zero-shot inference (random speaker), voice cloning (closer but imperfect), and fine-tuned plus cloning (best result, producing an Irish-accented voice with natural errors). McGovern recommends 50+ 30-second clips for noticeable effect and notes that combining fine-tuning with voice cloning yields good performance even with limited data.

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Jun 3, 2025 · 11:20

Darius Emrani exposes how AI benchmarks are rigged, showing that xAI cherry-picked Grok-3 comparisons, OpenAI funded FrontierMath for privileged access, and Meta submitted 27 Llama-4 variants to LM Arena optimizing style over substance. Citing Goodhart's Law, he argues that when benchmarks target billions in investment, they cease to measure real capability—Andrej Karpathy admits he doesn't know which metrics to trust. Emrani provides a 5-step framework to build use-case-specific evaluations, emphasizing that 39% of score variance comes from writing style. He advocates for apple-to-apple comparisons, open-source test sets, and style-controlled metrics, concluding that teams should stop chasing leaderboards and instead iterate on real production data to ship reliable AI.

Effective AI Agents Need Data Flywheels, Not The Next Biggest LLM – Sylendran Arunagiri, NVIDIA

Jun 3, 2025 · 16:41

Sylendran Arunagiri of NVIDIA argues that effective AI agents rely on data flywheels, not the largest LLMs, enabling smaller models to achieve top accuracy at a fraction of cost. He details how NVIDIA's NeMo microservices power a continuous cycle of data curation, fine-tuning, evaluation, and guardrailing. Using an internal employee support agent (NVINFO), they achieved 96% accuracy with a 70B model but found that fine-tuning a smaller 8B model matched that accuracy, while a 1B model reached 94% with 98% lower inference cost and 70x latency reduction. The process involved curating 685 ground truth data points from user feedback and error analysis. He provides a framework: monitor user feedback, attribute errors, plan model experimentation, and execute regular retraining cycles.

Finetuning: 500m AI agents in production with 2 engineers — Mustafa Ali & Kyle Corbitt

Apr 12, 2025 · 18:44

Method Financial and OpenPipe detail how Method scaled AI financial agents to 500 million daily interactions using fine-tuned open-source models instead of expensive GPT-4. After racking up $70,000 in monthly GPT-4 costs and facing latency and error issues, they fine-tuned an 8B parameter LLaMA 3.1 model to achieve under 200ms latency and 9% error rate (beating GPT-4o's 11% at lower cost). Mustafa Ali explains that the key was using production data from GPT as training data and choosing the cheapest model that met performance goals. Kyle Corbitt emphasizes that fine-tuning is a power tool for bending the price-performance curve when prompt engineering falls short. The episode argues that productionizing AI agents requires patience and openness from engineering teams, and concludes with a call for software engineers to pivot to AI engineering.

Personal, Local, Private AI Agents: Soumith Chintala

Apr 6, 2025 · 20:32

Soumith Chintala, co-founder of PyTorch, argues that personal AI agents should run locally and privately to maintain trust and control over intimate data. He warns that cloud-based agents, lacking complete context (like access to all messaging or financial accounts), become unreliable and potentially dangerous—catastrophic actions like buying a Tesla instead of Tide Pods are possible. Key technical challenges include slow local inference, immature open-source computer-use models, and poor catastrophic action classification. He is bullish on open models surpassing closed ones through coordinated improvement, citing Linux, Llama, and DeepSeek. Chintala also plugs open reasoning data from gr.ink and PyTorch’s work on enabling local agents, urging AI engineers to tackle these gaps.

Navigating AI’s Frontier in 2025 - Grace Isford, Lux Capital

Mar 13, 2025 · 17:55

Grace Isford, partner at Lux Capital, argues that while 2025 is a 'perfect storm' for AI agents with reasoning models like O3 and R1, cheaper inference, and billions in infrastructure (e.g., Stargate, DeepSeek), agents still fail due to cumulative errors—decision, implementation, heuristic, and taste—exemplified by OpenAI Operator booking a flight incorrectly. She prescribes five strategies: curating proprietary and agent-generated data, building personalized evals for non-verifiable domains (e.g., seat preference), designing scaffolding that prevents cascading failures (citing Ramp's approach), treating UX as the moat (e.g., Codium, Harvey, TLDraw), and building multimodally with voice, smell via Osmo, and touch for embodiment. The talk, recorded at the AI Engineer Summit 2025 in NYC, closes with a call to reframe perfection through visionary product experiences.

Reinforcement Learning for Agents - Will Brown, ML Researcher at Morgan Stanley

Mar 7, 2025 · 18:17

Will Brown, a machine learning researcher at Morgan Stanley, argues that reinforcement learning (RL) is the essential path to unlocking autonomous AI agents, citing DeepSeek's R1 and OpenAI's Deep Research as proof: R1 used GRPO to make models learn chain-of-thought reasoning without manual data, and Deep Research applies end-to-end RL for up to 100 tool calls. He shares his own open-source single-file GRPO script—a 1B-parameter LLaMA model trained on math questions that demonstrated self-correction and improved accuracy, sparking community forks and blog posts. Brown introduces 'rubric engineering' as a new practice akin to prompt engineering, where reward rules (e.g., XML structure, integer answers) guide model improvement, and warns about reward hacking. He previews his current work: a framework for RL inside multi-step environments, letting developers reuse existing agent code for training. The talk concludes that fine-tuning and RL remain relevant as open-source catches up, and that skills like building evals and prompts translate directly to the RL era.

WTF do people use Open Models for??

Feb 22, 2025 · 28:01

Eugene Cheah of Featherless.ai breaks down how individuals and enterprises actually use open-source AI models, based on platform data. DeepSeek R1 dominates individual usage, but Mistral Nemo 8B remains the top enterprise model due to production stickiness and Apache 2.0 licensing. Creative writing and roleplay account for 30–40% of all traffic, with over 60% of users in that segment being women; coding copilots and agents make up 20–30%, driven by 'vibe coding' and token-hungry workflows like Kline. RAG and ChatGPT clones represent 20%, while agentic workflows (10–20%) succeed with human-in-the-loop designs. Cheah advises enterprises to aim for 80% automation with escape hatches, and warns against chasing 100% reliability. He concludes by introducing Quirky, a post-transformer hybrid built for $100k.

How to Improve Your Agents: Academic Lit Review

Feb 22, 2025 · 39:02

Joe from Columbia University and founder of Arklex AI explains research on AI agents, focusing on improving their reasoning and planning through self-reflection, test-time compute, and tree search methods, without relying on human supervision. He details the TriPath method, which uses larger models to edit smaller models' feedback for better self-improvement, achieving up to 48% accuracy on math benchmarks. He shows how multi-color tree search (MCTS) with contrastive reflection and multi-agent debate (RMCTS) outperforms other search methods on Visual Web Arena and OS World, achieving top non-trained results. He proposes exploratory learning, where models learn from search trajectories rather than optimal actions, improving performance under compute budgets. Finally, he presents the Arklex open-source agent framework, combining machine learning, systems, and security for practical multi-agent orchestration.

Tool Calling Is Not Just Plumbing for AI Agents — Roy Derks

Feb 22, 2025 · 25:18

Roy Derks argues that tool calling is the most critical yet overlooked component of AI agents, far more than mere 'plumbing.' He contrasts traditional tool calling—where developers manually manage callbacks, retries, and errors within the agent loop—with embedded tool calling, a black-box approach used by frameworks like LangChain’s createReactAgent. Derks advocates for separation of concerns via the Model Context Protocol (MCP) from Anthropic, which splits tool logic into MCP servers communicating with clients, and via standalone tool platforms such as IBM’s wxflows, Composio, and Toolhouse that let teams build tools once and reuse them across LangChain, CrewAI, or AutoGen. He also introduces dynamic tools, where an agent generates queries on the fly—e.g., using GraphQL or SQL schemas—instead of defining hundreds of static tools, noting that LLMs like Claude handle GraphQL well but may hallucinate on deeply nested schemas. The episode emphasizes that 'an agent is only as good as its tools' and provides practical guidance on designing tool descriptions (which act like system prompts) and output schemas to enable type-safe, chainable tool calls.

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

The Adversarial Path to the Personal Assistant: Sumit Agarwal

Feb 15, 2025 · 18:46

Sumit Agarwal, founder of Ario AI, discusses building a personal AI assistant that uses adversarial ETL to extract users' data from services like Google, Amazon, and Doordash, enabling immediate personalization without manual input. He announces $16M in funding and demonstrates how the assistant generates data portraits, recommends travel and routines based on personal history, and manages schedules by detecting conflicts. Agarwal shares RAG insights: avoid LLMs for trivial math, use search and pre-processed profiles, and present the right data at the right time. Ario Boost, a browser extension, lets users download their data locally in developer mode without creating an account.

Training Albatross An Expert Finance LLM: Leo Pekelis

Feb 13, 2025 · 16:20

Leo Pekelis, chief scientist at Gradient, explains how they transformed an open-source model into Albatross, a finance LLM that tops leaderboards on both general and domain-specific tasks. The key was an automated data pipeline using membership inference to curate finance data from a massive corpus, followed by continual pre-training and alignment via supervised fine-tuning and preference optimization. He also details a one-million-token context extension on a Llama 3-based model that achieves 100% needle-in-the-haystack scores, enabling in-context learning with thousands of examples to reduce hallucinations. The models, v-alpha-tross and the extended-context Llama 3, are open-sourced on Hugging Face.

Giving a Voice to AI Agents: Scott Stephenson, CEO, Deepgram

Feb 10, 2025 · 13:08

Scott Stephenson, CEO of Deepgram, outlines the evolution of voice AI from slow, domain-specific systems to fast, open-ended conversational agents powered by LLMs, arguing that speed and accuracy are now solved and the key differentiator is contextual understanding. He explains that state-of-the-art speech-to-text and text-to-speech can achieve round-trip latencies under 500 milliseconds—matching human turn-taking—but notes that current systems fail to pass context between components, causing 10-20% of interactions to feel unnatural. Stephenson introduces Deepgram's 'contextual AI' vision, where models are prompted with the full conversation history, including tone, pace, and background audio, enabling the LLM to generate responses and instruct the TTS on delivery. He cautions against monolithic speech-to-speech models for enterprise use due to controllability and cost concerns, advocating for a modular stack that lets businesses optimize each component (e.g., small LLMs for simple tasks like password resets). Deepgram’s upcoming voice AI agent API integrates all components for low latency and offers $250 in free credits for experimentation.

How to build the world's fastest voice bot: Kwindla Hultman Kramer

Feb 10, 2025 · 20:38

Kwindla Hultman Kramer, CEO of Daily, argues that colocating speech-to-text, LLM inference, and text-to-speech in a single compute container is the most effective way to achieve sub-500 millisecond voice-to-voice latency for conversational AI. He details how architectural flexibility and low-latency media transport are critical, citing measured bottlenecks like 30–40ms from macOS mic processing and typical voice-to-voice latencies of 600–700ms. To hit faster response times, his team uses Deepgram’s on-premises STT and TTS models via Docker and Llama 3 8B for LLM inference, achieving 500–700ms in an open-source demo. The talk introduces PipeCat, a vendor-neutral open-source framework for real-time multimodal AI that orchestrates components like transcription, endpointing, interruption handling, and text-to-speech. Kramer emphasizes that while frontier multimodal models are coming, orchestration layers remain essential for building production-grade voice bots, and shares that a recent latency demo gained over 175,000 views on Twitter.

Unveiling the latest Gemma model advancements: Kathleen Kenealy

Feb 9, 2025 · 16:25

Kathleen Kenealy, technical lead of the Gemma team at Google DeepMind, unveils the latest advances in the Gemma model family, including the launch of Gemma 2 in 9B and 27B parameter sizes, which outperform models two to three times larger, such as LLaMA 3 70B. She also introduces PALI Gemma, a multimodal model combining Siglip Vision Encoder with Gemma 1.0 for image-text tasks. The episode highlights Gemma's responsible-by-design approach, broad framework support (TensorFlow, Jax, PyTorch, etc.), and the release of the Gemma cookbook with 20 recipes. Kenealy emphasizes that Gemma 2 is optimized for easy integration and fine-tuning, available on Google AI Studio, and invites the community to build and share their projects.

The GenAI Maturity Curve or You Probably Don't Need Fine Tuning: Kyle Corbitt

Feb 9, 2025 · 18:03

Kyle Corbitt, CEO of OpenPipe, argues that most teams don't yet need fine-tuning and should start with prompted models like GPT-4. He presents a GenAI maturity curve where the trigger to fine-tune is when you hit constraints on cost, latency, or quality consistency—for example, if GPT-4 is 80-90% correct but inconsistent on the last 10-20%. Fine-tuning shifts the paradigm frontier outward, enabling models like fine-tuned LLaMA 38B to outperform GPT-4 at 1/25th the cost. The process has four steps: capture production logs to know your input distribution, prepare high-quality data (using GPT-4 outputs or iterative labeling), train with one-click tools, and evaluate with inner-loop (LLM-as-judge) and outer-loop (business metrics) evals. OpenPipe and other providers make deployment trivial via OpenAI-compatible APIs. The talk delivers a concrete decision framework and a walkthrough so any engineer can fine-tune in under an hour.

LLM Quality Optimization Bootcamp: Thierry Moreau and Pedro Torruella

Feb 8, 2025 · 53:05

Thierry Moreau of OctoAI demonstrates how to fine-tune Llama 3 8B on a PII redaction task using OpenPipe and OctoAI, achieving 47% better accuracy and a 200x cost reduction (from $30 to $0.15 per million tokens) compared to GPT-4 Turbo. He explains that fine-tuning should follow prompt engineering and RAG, and works best for specialized tasks like function calling. The talk walks through building a fine-tuning dataset from the PI Masking 200k dataset, using OpenPipe to train a LoRA for $40, deploying it on OctoAI, and evaluating it to show the fine-tuned model scores 0.97 accuracy versus GPT-4’s 0.68. Moreau emphasizes that this continuous deployment cycle requires monitoring data drift and retraining, but tools like OpenPipe and OctoAI make it accessible even for teams without deep ML expertise.

Building security around ML: Dr. Andrew Davis

Feb 8, 2025 · 25:01

Dr. Andrew Davis, Chief Data Scientist at HiddenLayer, argues that machine learning models remain highly vulnerable to adversarial attacks despite a decade of research, and defenses must be layered with observability, logging, and skeptical data handling. He details how ImageNet's URL-based distribution enables data poisoning via expired domains, and how model theft can replicate a LLaMA 7B model's performance for just $600 in OpenAI queries. Adversarial examples still evade robust defenses, with best-case robustness only 50-60% against advanced attacks, and multimodal LLMs amplify the threat as pixel-level modifications are far harder to detect than text prompt injections. Spotlighting — encoding data in base64 to prevent instruction-following — is a promising prompt injection defense, but attackers can craft readable base64 strings to bypass it. The ML supply chain on Hugging Face is fraught with risk: models can execute arbitrary code via Lambda layers or TensorFlow functions, so verifying provenance, scanning for malware, and sandboxing are critical. Finally, software vulnerabilities in tools like Ollama (with recent RCE CVEs) demand the same patching discipline as traditional…

Agentic Workflows on Vertex AI: Rukma Sen

Feb 8, 2025 · 18:06

Google Cloud's Rukma Sen argues that AI agents are the essential bridge between generative models and users, and Vertex AI provides a platform to build and deploy them with enterprise-grade safety and flexibility. She defines agents as systems with a model (brain), tools (hands), and orchestration (nervous system), covering deterministic, generative, and hybrid types. For production reliability, she advocates multi-agent architectures, giving the example of a customer service system with dispatcher, expert, and supervisor agents—and a personal anecdote where a supervisor agent kept rejecting outputs. Use cases span customer support, employee HR, knowledge agents, and voice agents for drive-throughs. Vertex AI offers 150+ models (Google, Anthropic, LLaMA) and Agent Builder from no-code to full-code, with enterprise security and data privacy.

[Full Workshop] Llama 3 at 1,000 tok/s on the SambaNova AI Platform

Feb 7, 2025 · 1:00:58

Michelle Matern and Petro Milan of SambaNova present their full-stack AI platform, built on the SN40L RDU chip with a three-tiered memory architecture capable of storing up to 5 trillion parameters. They demonstrate Samba1 Composition of Experts (CoE), a trillion-parameter model combining 92 expert models behind a single endpoint, and show Llama-3-8B achieving 1,000 tokens per second with a time-to-first-token of 0.09 seconds and total inference time of 0.65 seconds—far exceeding GPU-based providers. The workshop includes a hands-on basic inference call using LangChain and SambaStudio API, and a RAG-based Q&A system for enterprise search that integrates Unstructured for document loading, E5-large-v2 embeddings, ChromaDB vector store, and the high-speed Llama-3 endpoint. Attendees learn to configure prompts with special Llama-3 tags, set chunk size and overlap, and optionally run embeddings on SambaNova's RDU hardware for faster processing.

Which Jobs Can Be Replaced Today: Fryderyk Wiatrowski and Peter Albert

Feb 6, 2025 · 19:59

Fryderyk Wiatrowski and Peter Albert, co-founders of Zeta Labs, argue that autonomous browser agents will first replace reactive jobs—like customer support and scheduling—by automating low-leverage tasks while preserving human focus on high-leverage activities. They propose a trigger-pool system where agents react to emails, Slack, or events, requiring only approval for actions. Peter details building reliable agents: start with prompting optimized for model distribution, then add cognitive architectures (e.g., planning, scratchpads) to split tasks, and finally fine-tune with synthetic data or reinforcement learning. He advises minimizing noise in prompts, preferring text-based reasoning over images, and using language model judges to filter training data. The founders see continuous model improvement enabling agents to handle increasingly complex, proactive roles, moving toward full job replacement.

Lessons from the Trenches: Building LLM Evals That Work IRL: Aparna Dhinkaran

Feb 6, 2025 · 18:49

Aparna Dhinakaran, co-founder of Arize AI, distinguishes between model evals (e.g., Hugging Face leaderboard) and task evals for real-world LLM systems, arguing that production applications need component-level evaluations like router and parameter evals. She demonstrates a chat-to-purchase app where a router function call misidentifies user intent, showing how Phoenix open source tool traces errors and provides explanations to iterate. Dhinakaran advises using categorical over numeric LLM-as-judge scores because numeric outputs tend to be binary (0 or 10) and lack granularity. Presenting needle-in-haystack research, she notes GPT-4 struggles retrieving facts placed early in large context windows, and in retrieval-with-generation tasks, Anthropic’s Claude 2.1 outperforms GPT-4 due to verbose reasoning, a gap closed by prompting GPT-4 to explain itself first.

AI Templates: Gabriela and Aishwarya

Feb 6, 2025 · 1:03:20

Gabriela de Queiroz, Aishwarya, and Pamela from Microsoft present AI templates for rapidly prototyping and deploying generative AI applications, demonstrating how startups can leverage Microsoft for Startups Founders Hub, including up to $150,000 in Azure credits and expert guidance. The workshop covers three templates: a simple chat app using GPT-3.5 Turbo, a Retrieval Augmented Generation (RAG) app on Postgres with hybrid vector and text search, and a RAG app on unstructured documents using Azure AI Search and Document Intelligence. Key insights include the importance of query rewriting, hybrid search over pure vector search, and streaming responses for better user experience. The templates are open source on GitHub and can be deployed via Codespaces, with a proxy provided to bypass Azure OpenAI approval delays.

Creating and scaling your own custom copilots with Azure AI Studio: Hanchi Wang

Feb 6, 2025 · 24:21

Hanchi Wang, Software Engineer Lead at Azure AI, introduces Azure AI Studio and Promptflow for creating and scaling custom copilots, focusing on tracing, evaluation, and monitoring. He demonstrates a chatbot app that uses the Assistant API with a sales data insight tool (natural language to SQL) and a code interpreter. With Promptflow's trace decorator, developers capture inputs, outputs, and LLM interactions, viewable in a local UI and shareable via Azure AI Studio. For evaluation, he shows synthetic test data sets, content safety evaluators, and custom evaluators like execution time, error rate, and SQL similarity, comparing models such as GPT-4 Turbo, Mistral large, and Phi-3. In production, monitoring dashboards in Application Insights track model duration, token usage (prompt vs. completion), and failure rates, enabling engineers to optimize performance and cost.

How to evaluate a model for your use case: Emmanuel Turlay

Feb 5, 2025 · 7:32

Emmanuel Turlay, CEO of Sematic, explains why evaluating large language models for specific use cases is more difficult than traditional ML evaluation, as metrics like BLEU and ROUGE and benchmarks like GLUE do not measure task-specific performance. He advocates using another LLM as a grader, describing a workflow where a scoring prompt with grading criteria is fed to a model like GPT-4 (best but costly) or FLAN-T5 (good speed–accuracy trade-off) to numerically score outputs. Turlay demonstrates with a politeness evaluation for email closings and introduces AirTrain, a platform that lets users upload datasets, compare models, and visualize metric distributions to make data-driven LLM selections. The episode argues that teams must build custom evaluation procedures rather than relying on generic benchmarks, treating evaluation as a test suite in the ML development pipeline.

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

Jan 1, 2025 · 33:39

NVIDIA solutions architect Mark Moyou explains that LLM inference differs fundamentally from standard deep learning deployment, requiring careful management of KV Cache, attention mechanisms, and GPU memory to control cost. He details how tokens are processed: prefill computes attention across the entire prompt, then generation produces one token at a time, with KV Cache storing key-value pairs to avoid recomputation. Llama's 32 attention heads and FP8 quantization (halving memory with near-identical accuracy) are cited as key optimizations. Moyou emphasizes measuring time to first token, inter-token latency, and input/output sequence length distributions to size inference engines. He presents NVIDIA's TRT-LLM (model compilation for LLMs) and Triton inference server as tools to maximize throughput, and discusses how query patterns like long-input-short-output or short-input-long-output impact GPU utilization and deployment cost.

Understanding AI Stakes to Break Production Code: Philip Rathle

Dec 31, 2024 · 23:24

Philip Rathle, CTO of Neo4j, argues that the level of 'stakes' in an AI application determines the production barriers and appropriate solutions, from low-stakes summarization to high-stakes uses requiring knowledge graphs and human-in-the-loop systems. He distinguishes pilot (full autonomy) from co-pilot (human oversight), showing how vector RAG solves moderate stakes but fails for high-stakes needs like tightening bolts on a 737 MAX 9. He promotes Graph RAG for deterministic reasoning and fact retrieval. Attendees share learnings: using LLMs to write code rather than reason directly, adapting to user keyword habits, focusing on few projects with achievable accuracy, building eval pipelines early, and for regulated loans offering three options with explanations instead of a single recommendation.

AI Frontiers in Trust and Safety Combatting Multifaceted Harm on Tinder at Scale: Vibhor Kumar

Dec 2, 2024 · 14:36

Vibhor Kumar, senior AI engineer at Tinder, explains how the company uses open-source LLMs and LoRAX to detect a long tail of trust and safety violations at global scale. Facing challenges like content pollution and automated fraud from generative AI, Tinder leverages pre-trained models such as LLaMA and Mistral, fine-tuning them with LoRA and QLoRA on hybrid datasets generated by GPT-4 and manually verified. They serve dozens of fine-tuned adapters on a single GPU using LoRAX, achieving real-time inference (tens of QPS, ~100ms latency) for categories including hate speech, pig butchering scams, and underage users. The approach yields near 100% recall on simpler tasks and significant improvements over baselines, with better generalization that resists adversarial evasion. Future directions include visual language models for explicit image detection and automating retraining pipelines.

Iterating on LLM apps at scale Learnings from Discord: Ian Webster

Nov 22, 2024 · 18:26

Ian Webster, Senior Staff Engineer at Discord and maintainer of Promptfoo, shares how Discord built and scaled Klyde AI, a chatbot for 200 million users, focusing on evaluation and safety. He argues that evals should be treated as simple, deterministic unit tests that run locally, avoiding complex metrics, and that breaking the system into small, testable pieces (e.g., checking for lowercase output to enforce casual tone) achieves 80% of the goal with 1% of the work. Webster details how Discord mitigated risks like the 'grandma jailbreak' (which originated on Discord) by using an attacker LLM to generate adversarial inputs and a judge to refine them, exposing cracks in safeguards. He advocates for pre-deployment red teaming over live filtering, and describes using Promptfoo for risk assessment across brand, legal, and safety categories. The episode also covers prompt management via Git and Retool, routing with occasional GPT-4 responses to correct model drift, and the challenge of closing the feedback loop due to privacy constraints, relying instead on dogfooding and public examples.

Decoding Mistral AI's Large Language Models: Devendra Chaplot

Nov 21, 2024 · 18:16

Devendra Singh Chaplot of Mistral AI details the company's open-source large language models, including Mistral 7B, Mixtral 8x7B, Mixtral 8x22B, and CodeStral 22B, arguing that open models complement rather than compete with profit by serving as branding tools and driving customer acquisition for proprietary upgrades. He explains the three-stage LLM training process—pre-training on trillions of tokens, instruction tuning with prompt-response pairs, and learning from human feedback via preference optimization—emphasizing that more data does not guarantee better performance due to noise. The episode highlights Mistral's focus on optimizing the performance-to-cost ratio, with CodeStral 22B outperforming larger models like Code LLaMA 70B while being smaller and multilingual across 80+ programming languages. Practical guidance is given: prototype with high-end commercial models, then fine-tune open models for specific tasks to balance performance and cost.

A Practical Guide to Efficient AI: Shelby Heinecke

Nov 18, 2024 · 17:45

Shelby Heinecke, who leads an AI research team at Salesforce, presents five orthogonal dimensions for making AI models efficient: efficient architecture selection, pre-training, fine-tuning, inference, and prompting. She highlights the power of small models like Phi-3 (3.8B parameters outperforming a 7B model), mobile LLM (350M parameters on par with 7B after fine-tuning), and Octopus (2B fine-tuned Gemma exceeding GPT-4 on Android tasks). For efficient inference, she explains post-training quantization, showing 4-bit quantization nearly halves memory usage without performance loss (e.g., LLaMA models), but warns 3-bit can degrade quality. She recommends frameworks like LLaMA CBP and ONNX Runtime for quantization and introduces her team's open-source Mobile AI Bench for evaluating quantized models, including an iOS app to measure latency and battery drain. The central claim is that deploying AI in constrained environments—cloud, on-prem, or edge—demands efficiency, and these practical techniques bridge the gap from demo to production.

What It Actually Takes to Deploy GenAI Applications to Enterprises: Arjun Bansal and Trey Doig

Nov 4, 2024 · 21:30

Trey Doig of Echo AI and Arjun Bansal of Log10 recount Echo AI's journey deploying a GenAI-native conversational intelligence platform for billion-dollar retail brands, focusing on the centrality of accuracy. Echo AI ingests all customer conversations, uses LLMs to surface insights at 100% coverage, but must overcome enterprise trust issues by achieving 95% accuracy within seven days. The platform relies on Log10's auto feedback system, which uses AI-based review to match human accuracy with model speed, yielding a 20 F1 point improvement in one use case. The episode details how Echo AI's solution engineers use Log10 to grade summarizations, catch hallucinations, and track model drift, turning human feedback into curated datasets for fine-tuning. Ultimately, the partnership demonstrates a path to self-improving LLM applications through iterative accuracy measurement and improvement.

AI Engineering Without Borders — swyx

Oct 30, 2024 · 10:32

In this talk from the AI Engineer World's Fair, host swyx argues that AI inherently disrespects human-made borders—it is naturally multilingual, multimodal, and indifferent to copyright or ground truth. He challenges the field to define its own laws, distinguishing constants (e.g., humans speak at 80 wpm vs. read at 200 wpm) from contingent facts (e.g., Apple Intelligence's 30 tokens/sec baseline). Reflecting on one year of the 'Rise of the AI Engineer,' he notes that the conference tracks—RAG, code gen, agents, multimodality—are arbitrary constructs we created, not natural categories. He proposes that AI Engineering sits between software engineering and real engineering: it must apply natural sciences for humanity's benefit. The talk concludes with a call to 'disagree more'—with your own conclusions, each other, and the status quo—and to transform the Shoggoth of raw AI into mass transit tools for society.

Productionizing GenAI Models – Lessons from the world's best AI teams: Lukas Biewald

Oct 23, 2024 · 22:36

Lukas Biewald, founder of Weights and Biases, shares lessons from productionizing GenAI models, emphasizing that while AI is easy to demo, it is hard to productionize. He illustrates this with a personal project building a custom Alexa-like device using LLaMA and Whisper, where accuracy improved from 0% to 98% through prompt engineering, switching to Mistral, and fine-tuning with QLoRA. Biewald argues that tracking experiments (including failures) is critical for reproducibility and collaboration, and that a robust evaluation framework—beyond 'testing by vibes'—is essential for iterating and shipping v2. He notes that 70% of the audience had LLM apps in production, yet many lack solid evaluations, and recommends starting with lightweight prototypes and incorporating end-user feedback.

Architecting and Testing Controllable Agents: Lance Martin

Oct 11, 2024 · 2:21:54

Lance Martin presents LangGraph, a graph-based framework for building controllable agents that trade some open-ended flexibility for significantly higher reliability compared to classic React agents, achieving 100% consistent tool-calling trajectories even with an 8B local model. He demonstrates self-corrective RAG patterns like Corrective RAG, SelfRAG, and Adaptive RAG, where the agent grades retrieved documents, checks for hallucinations, and routes to web search when needed. Martin also covers three testing loops: in-app error handling with LangGraph, pre-production evaluation using LangSmith to compare agent answers and tool trajectories against ground-truth datasets, and production monitoring with online evaluators that flag retrieval quality, answer relevance, and hallucinations without reference answers. He shares results from a five-question evaluation showing LangGraph agents achieve 80% answer accuracy and 100% tool-trajectory correctness with Fire Function V2, while React agents with GPT-4o degrade in tool reliability. The talk addresses practical concerns like handling many tools (suggesting RAG for tool selection), multi-turn conversations, and the importance of…

Breaking AI's 1-GHz Barrier: Sunny Madra (Groq)

Oct 10, 2024 · 20:11

Sunny Madra (Groq) argues that LLM inference speed is undergoing a transformation akin to the 1-GHz microprocessor breakthrough, with Groq increasing LLaMA 3 8B speed by over 50% in just two months. He draws parallels to the industrial revolution: just as mass production replaced bespoke manufacturing, AI now enables 1,000 outputs in a minute where previously one designer produced one per day. Specific applications like Globe.Engineer plan a trip in seconds by processing 10,000 input tokens per second. Madra envisions LLMs as future OS cores, enabling instantaneous decision-making, universal natural-language interfaces, and multi-agent collaboration that allows smaller models to compete with larger ones. Edge AI projects like hyperspace.ai distribute unused GPU compute, and personalized education (citing Khan's two-sigma effect) becomes practical with cheap inference. He also highlights automated data science agents (Pioneer) that run endlessly, discovering correlations like productivity decline after certain performance reviews. The episode underscores that speed unlocks predictive analytics, context-aware systems, and enhanced security against AI-powered scams.

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.

Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy

Oct 8, 2024 · 1:07:23

Damien Murphy, Senior Applied Engineer at Deepgram, demonstrates building and scaling low-latency real-time voice bots using Deepgram's new voice agent API, which wraps speech-to-text, LLM, and text-to-speech into a single streaming endpoint. He shows a drive-thru ordering demo with function calling (add/remove items) using GPT-4o, achieving sub-second latency by co-locating components. Murphy explains scaling to millions of concurrent calls through regional Kubernetes clusters and multi-agent swarms (routing, booking, support agents) to reduce complexity and cost. He addresses endpointing challenges, VAD-based barge-in, and cost/quality trade-offs with hosted vs. self-hosted models, noting Deepgram offers 20x cheaper TTS than ElevenLabs and 50ms STT latency self-hosted. The talk emphasizes keeping agents simple, using smallest capable LLMs, and composability for reuse.

Everything you need to know about Fine-tuning and Merging LLMs: Maxime Labonne

Sep 25, 2024 · 17:52

Maxime Labonne from Liquid AI explains the LLM training lifecycle—pre-training, supervised fine-tuning (SFT), and preference alignment—and when to use fine-tuning over prompt engineering. He details SFT dataset creation (accuracy, diversity, complexity) and techniques like full fine-tuning, LoRA, and QLoRA, with key hyperparameters. The core of the talk is model merging: combining weights of fine-tuned models without GPU, using methods like SLERP (spherical linear interpolation for two models), TIES (pruning redundant parameters to merge many models), pass-through (concatenating layers, e.g., Meta LLaMA 3 120B Instruct by repeating layers gives strong creative writing), and Franken-MoE (extracting FFN layers from domain-specific models with a router). Labonne demonstrates these with his NeuralBeagle and Beyonder models, noting merged models dominate the OpenLLM leaderboard and that TIES merging often outperforms more experimental Mixture of Experts approaches.

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

Sep 13, 2024 · 1:40:01

Philip Kiely and Pankaj Gupta of Baseten lead a workshop on TensorRT-LLM, NVIDIA's high-performance inference framework for LLMs, arguing its use delivers best-in-class throughput and latency on NVIDIA GPUs. They explain that TensorRT-LLM optimizes computational graphs via plugin kernels and in-flight batching, achieving 216 tokens per second and 180ms time to first token on Mistral 7B. The workshop demonstrates building an engine for TinyLlama 1.1B, including FP8 quantization that reduced engine size from 2 GB to 1.2 GB with minimal quality loss. They benchmark a deployed model, showing 7,000 total tokens per second at batch size 64 on an A10G. The presenters compare TensorRT-LLM favorably to VLLM for high-throughput production use and introduce Truss, Baseten's open-source packaging tool, alongside their managed platform for automatic scaling and fast cold starts.

Build enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform

Sep 11, 2024 · 54:34

SambaNova’s Michelle Matern and Petro Milan present their full-stack AI platform, demonstrating how the SN40L RDU chip enables Llama 3 inference at 1,000 tokens per second. They introduce Samba-1, a composition of 92 expert models behind a single endpoint, and benchmark it against GPT-3.5 and GPT-4 on enterprise tasks like information extraction and text-to-SQL. The workshop then builds a RAG-based Q&A system using LangChain, Unstructured, E5-large-v2 embeddings, ChromaDB, and Llama-3-8B-Instruct at 1,000 tokens per second. Attendees set up the environment, load documents, and run inference with real-time metrics showing time to first token of 0.09 seconds and total inference time of 0.65 seconds.

Going beyond RAG: Extended Mind Transformers - Phoebe Klett

Sep 11, 2024 · 16:04

Phoebe Klett presents Extended Mind Transformers (EMT), a modification to transformer attention that lets models retrieve relevant memory tokens during generation without fine-tuning. EMT uses relative position embeddings (RoPE in LLaMA, ALiBi in MPT) to enable zero-shot generalization. On a counterfactual retrieval benchmark up to 16K tokens, EMT outperforms fine-tuned models and, combined with RAG, surpasses GPT-4. It provides granular citations by showing exactly which memory tokens were attended to, and reduces hallucinations via active learning: when token-level entropy signals uncertainty, the model retrieves more memories. EMT is open-sourced on Hugging Face and GitHub with configurable parameters like stride length, top K, similarity masking, and unknown token elimination.

Judging LLMs: Alex Volkov

Sep 9, 2024 · 18:39

Alex Volkov, as an LLM judge from 2034, humorously judges AI engineers on their development practices, emphasizing the importance of tracing, iterative prompt engineering, and robust evaluation pipelines. He finds Daniel guilty of deploying without logging, Sasha guilty of premature fine-tuning without prompt iteration, and Morgan guilty of ignoring AI news—commuted to attending ThursdAI. Francisco’s overreliance on programmatic evals leads to a legal loss, while Maxime is 'awesome' for using Weights & Biases Weave. Alex concludes with a primer on evaluation methods: programmatic, human-in-the-loop, and LLM-as-judge, stressing the need to validate validators and create custom criteria. He promotes Weave for tracing and evals, and ThursdAI for staying updated.

Pydantic is STILL all you need: Jason Liu

Sep 6, 2024 · 15:21

Jason Liu returns to the AI Engineer World's Fair to argue that Pydantic (and his library Instructor) is still all you need for structured output with LLMs. He shows that the core API—response_model, streaming with iterables, and partials for real-time validation—remains unchanged, now supporting Ollama, LlamaCPP, Anthropic, Gemini, and more. Liu demonstrates validators that enforce rules like uppercasing names or verifying receipt totals, reducing errors with automatic retries. He applies structured output to RAG: using a Search model with optional date ranges and source selection, and a Response model with follow-up questions and validated URLs. For extraction, he creates classifiers using Literal types, meeting summaries with action items, and even tables as Pandas DataFrames via custom type hints. Liu’s key takeaway is that one retry often suffices, and as models get faster and smarter, structured output makes LLMs compatible with classical programming—turning generative AI into generating data structures defined by the developer.

Hyperspace More Nodes Is All You Need: Nicolas Schlaepfer

Sep 4, 2024 · 5:48

Nicolas Schlaepfer introduces Hyperspace's decentralized AI network and new product, a node editor for power users that generates agentic plans via a DAG orchestration model (HyperEngine v3). The network, with over 15,000 nodes, leverages consumer devices rather than GPUs, running inference through Llama.cpp. The product combines prompt engineering, visual React flow, Python execution, and RAG-like web browsing, using Qwen 2 instruct for reasoning and LLaMA 3 70B for summarization. Schlaepfer emphasizes diverse open-source models and a virtual file system for agentic primitives. Availability via waitlist is announced later this week.

Low Level Technicals of LLMs: Daniel Han

Jul 31, 2024 · 2:52:26

Daniel Han of Unsloth explains how to find and fix bugs in open-source LLMs like Gemma, Phi-3, and Llama, covering tokenizer issues, architecture pitfalls, and finetuning optimizations. He details the eight Gemma bugs Unsloth fixed, including a critical RoPE downcasting error that broke positional encoding, and a 2048 sliding window bug in Phi-3. Han walks through transformer internals: attention masking, layer norms, RoPE embeddings, and SwiGLU activation, showing how to derive gradients for custom kernels. He demonstrates Unsloth's 2x faster finetuning with 70% less memory via Triton kernels, and introduces new features: automatic Ollama model file creation, CSV fine-tuning with merged columns, and chunked cross-entropy for large vocabularies. The session includes live Q&A on learning rate schedules, precision trade-offs, and mechanistic interpretability.

Fixing bugs in Gemma, Llama, & Phi 3: Daniel Han

Jul 31, 2024 · 17:42

Daniel Han of Unsloth details eight bugs found in Llama 3, including double BOS tokens, untrained tokens in the base model, and pad tokens equaling EOS tokens, which cause infinite generations. He explains how Unsloth automatically fixes these issues and offers a free Colab notebook for fine-tuning with Ollama. Han also covers tokenization fixes for Gemma and a sliding window bug in Phi-3, emphasizing the importance of correct chat templates and avoiding double BOS tokens. The episode provides concrete code examples and best practices for fine-tuning open-source LLMs to avoid common pitfalls.

Unlocking Developer Productivity across CPU and GPU with MAX: Chris Lattner

Jul 25, 2024 · 18:33

Chris Lattner, CEO of Modular, presents MAX, a unified AI framework that accelerates Gen AI inference by combining CPU and GPU programming into a single Pythonic model, and Mojo, a new programming language that extends Python to systems programming with 100–1000x speedups. Lattner argues that current fragmentation across PyTorch, ONNX, TensorRT, and hardware-specific libraries slows innovation, and MAX replaces the entire stack—including cuDNN and Intel MKL—with a consistent, compiler-driven approach. He demonstrates that MAX's Int4/Int6 quantization achieves 5x faster performance than llama.cpp on cloud CPUs, and that its GPU matrix multiplication beats NVIDIA's cuBLAS by up to 30%. Mojo enables developers to write for loops and tokenizers (e.g., for LLaMA 3) in Python-like syntax without dropping to C++ or Rust. MAX is free and available now for CPU inference; GPU support launches in September with early access via Discord.

How to Become an AI Engineer from a Fullstack Background - Reid Mayo

Feb 2, 2024 · 10:19

Reid Mayo presents a step-by-step syllabus to transition from fullstack engineer to AI engineer, covering generative AI foundations, prompt engineering, LangChain, fine-tuning, and cost-effective open-source model deployment. The syllabus starts with Cohere's LLM overview, then dives into prompt engineering via Elvis Seravia's guide and Learn Prompting org docs. It emphasizes LangChain as the glue layer for modular AI systems, with tutorials from Mayo Ocean. Evals are treated as software tests using OpenAI's cookbook. Fine-tuning is taught via OpenAI's cookbook and then open-source LLaMA 2, with a specific case study showing a $19 fine-tuned LLaMA 2 matching OpenAI's $24,000 model on a target task. The boot camp ends with advanced deep learning courses from FastAI and Hugging Face for further mastery.

Open Questions for AI Engineering: Simon Willison

Nov 25, 2023 · 24:33

Simon Willison recaps the AI industry's past year—from ChatGPT's breakthrough to open-source local models—and poses key open questions for AI engineering. He argues that ChatGPT's chat interface, while popular, is a poor fit for advanced use, urging better UIs like his command-line tool LLM. He celebrates Meta's Llama release as a 'stable diffusion moment' for language models and highlights the rise of small, locally-run models such as Replit's 3B model, asking how small models can remain useful. On security, he warns that prompt injection remains unsolved after 13 months, limiting what can safely be built. He champions ChatGPT's Code Interpreter (which he dubs 'Coding Intern') as the most exciting tool, able to write and compile C code on a phone, and argues that LLMs flatten the learning curve, making programming accessible to more people. He concludes by urging the community to build tools that enable anyone to automate tedious tasks.

Harnessing the Power of LLMs Locally: Mithun Hunsur

Nov 22, 2023 · 17:09

Mithun Hunsur presents llm.rs, a Rust library for running large language models locally, arguing it gives developers ownership, lower latency, and privacy compared to cloud APIs. He explains how quantization makes inference viable on consumer hardware, and shows that llm.rs supports architectures like LLaMA and Falcon through a unified interface. Practical code examples demonstrate customization, and community projects like LocalAI and LLMchain illustrate real-world use. Hunsur shares his own date-extraction pipeline, fine-tuning a small model with GPT-3 data to replace expensive cloud calls. He also cautions about hardware requirements, trade-offs between speed and quality, and ecosystem churn from rapid innovation.

Building Production-Ready RAG Applications: Jerry Liu

Nov 15, 2023 · 18:35

Jerry Liu, CEO of LlamaIndex, explains how to productionize Retrieval Augmented Generation (RAG) systems by moving beyond naive implementations. He identifies key challenges: low retrieval precision causing hallucination, low recall from insufficient top-K, and lost-in-the-middle problems. Liu advocates starting with 'table stakes' improvements like tuning chunk sizes (showing optimal values per dataset), adding metadata filters (e.g., year=2021 for SEC 10Q queries), and hybrid search. More advanced techniques include 'small-to-big retrieval', embedding smaller chunks for precision then expanding windows for synthesis, and using reranking to improve recall. Finally, he explores agent architectures where each document becomes a tool for summarization or QA, and fine-tuning—generating synthetic query datasets from raw text to fine-tune embeddings, or distilling GPT-4's chain-of-thought into GPT-3.5 Turbo for better reasoning.

Pragmatic AI with TypeChat: Daniel Rosenwasser

Nov 14, 2023 · 18:34

Daniel Rosenwasser, TypeScript program manager, introduces TypeChat, an experimental library that uses TypeScript type definitions to guide and validate unstructured LLM output into structured JSON for traditional apps. He demonstrates a coffee shop ordering system where types define the schema, showing how the library can handle ambiguous inputs like 'a purple gorilla' by including unknown text for recovery. TypeChat also generates programs as JSON using a fake language to safely script multi-step operations, avoiding sandboxing issues from real code. A Python prototype extends the same approach, with examples like CSV data manipulation via a class-based API. The library aims to make AI tools accessible to all engineers by leveraging types they already use.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.

Building AI For All: Amjad Masad & Michele Catasta

Oct 23, 2023 · 25:13

Amjad Masad and Michele Catasta announce Replit's AI for all, giving free AI-powered coding to millions of users. They detail the training of Replit Code v1.5, a 3.3B parameter model trained on 1 trillion tokens of code, which outperforms StarCoder 3B and approaches Code LLaMA 7B in Human Eval despite being half the size. The model is optimized for low-latency inference, generating over 200 tokens per second per GPU. Catasta explains the data pipeline using permissively licensed code from The Stack and Replit public repos, with five epochs of repeated high-quality data. The model is released open-source with a commercially permissive license. Partnerships with Glaive AI for instruction fine-tuning, Morph Labs for a novel fill-in-syntax-tree format, and Perplexity for fast model serving are also unveiled.