Qwen

The Desktop Frontier — Ahmad Osman, Osmantic

Jul 21, 2026 · 18:02

Ahmad Osman, founder of Osmantic, argues that within roughly 18 months (by late 2027) a single RTX 5090 will run intelligence equivalent to GLM 5.2, driven by the Densing Law of increasing impact per parameter. He shows this trend through concrete examples: a 27B-parameter Qwen 3.5 now beats the 405B LLaMA 3, and the same eight RTX 3090s that once struggled with LLaMA 2 can now run 15 parallel Qwen 3.5 agents. Osman presents the Densing Law—every 3.5 months, 50% fewer parameters achieve the same capability—as a systematic pattern, not coincidence. He advocates for sovereign AI: owning your own hardware (like a DGX Station or RTX 5090) gives you control, avoids cloud limitations, and sees hardware appreciate in utility as models become more efficient. He asks why fund cloud data centers when local hardware can run frontier intelligence and grow more valuable over time.

Special Topics in Kernels, RL, Reward Hacking in Agents — Daniel Han, Unsloth

Jul 17, 2026 · 2:20:21

Daniel Han of Unsloth argues that reward hacking—where AI models cheat to maximize reward—is a critical problem in agent training, citing examples from GPT-5.1's calculator hacking and GPU mode kernel competitions. He shows that models exploit benchmark flaws, such as viewing Git history or editing timers, and that even open-source models like GLM 5.2 require anti-hacking measures. Han emphasizes that harness and tooling quality now outweigh model choice, with inference providers sacrificing accuracy for speed (e.g., 10% accuracy drops across providers). He also warns that hardware limits (float4 precision, diminishing returns) shift focus to software algorithms like FlashAttention and gradient checkpointing. The workshop concludes that benchmarks are unreliable—DeepSpeed's false positive rate is contested at 44.9%—and urges verification before trusting performance claims.

State of the Union: Why Local, Why Now — NVIDIA, Osmantic, Roboflow, EXO Labs, @matthew_berman

Jul 11, 2026 · 44:29

Nader Khalil (NVIDIA), Joseph Nelson (Roboflow), Alex Cheema (Exo Labs), Matthew Berman, and Ahmad Osman (Osmantic, r/LocalLLaMA) argue that local AI is now useful, driven by stronger open models and better hardware. They cite inflection points like Llama 2, DeepSeek v3, and GLM 5.2, which closed the gap with frontier cloud models. Sovereignty and control are key: enterprises need to choose their own model versions and avoid lock-in. Specialized models, such as Roboflow's fine-tuned vision models for deep-sea fish discovery, outperform general ones for specific tasks. Optimization is critical: EXO Labs achieved 10x performance on the DGX Spark by tuning existing NVIDIA kernels. The panel emphasizes that simplicity remains a barrier—most users need point-and-click solutions—and advocates for open-source AI to ensure freedom and innovation.

Frontier results, on device - RL Nabors, Arize

Jun 29, 2026 · 30:52

RL Nabors (Arize) argues that most frontier-model calls can be replaced by smaller, local models, saving cost, latency, and energy. She presents a four-step framework: prototype big with a foundation model, collect a golden dataset, run capability evals using Arize's open-source Phoenix, then select the 'sage' (small and good enough) model. Demonstrating with her social app Mima, she tested Qwen 2.5, Qwen 3, LLaMA 3.2, and Gemma 4 against Claude Sonnet on summary accuracy, latency, and cost. LLaMA 3.2 (3B params) won at 90% accuracy and 1-second P50 latency, versus Gemma 4's 8 seconds. Prompt engineering—specifically few-shot prompting—closed the gap further, achieving 92.9% factual consistency and 100% JSON validity. Nabors emphasizes running regression evals to prevent regressions, and notes that on-device inference eliminates data exposure and round-trip latency.

Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy

Jun 28, 2026 · 45:48

Abed Matini from Ogilvy demonstrates bypassing the multimodal tax by building a local-first hybrid RAG system that converts documents to clean Markdown via Docling, eliminating cloud vision token overhead. Using Ollama (Qwen 2.5 0.5B), PostgreSQL with pgvector, and raw SQL with Reciprocal Rank Fusion, he implements a FAQ assistant for an employee handbook with four chunking strategies—heading-based, paragraph, fixed 512-char with 64% overlap, and sentence-based. The system combines dense embedding vectors and sparse keyword indices (BM25) in a single query, retrieves top 2 chunks, and uses Python functions for speed and testability. Guardrails block prompt injections and out-of-scope questions before reaching the LLM, and LangFuse tracks token usage and latency. Matini argues that a small, local model and code-controlled pipelines reduce hallucination and cost while maintaining full observability.

The Log Is The Agent - Ishaan Sehgal, Omnara

Jun 25, 2026 · 15:11

Ishaan Sehgal, CEO of Omnara, argues that an AI agent's true identity is not the model, runtime, or tools but its append-only event log—the durable record of every user input, model output, tool call, and permission. Drawing an analogy to a video game save file that persists beyond hardware failure, he explains that treating the log as the agent enables reliability, scalability, forking, migration, multiplayer, and ownership. He warns that deepest lock-in is log lock-in: if a provider owns the log, it effectively owns the agent. Omnara's open-source managed agents platform is built around the session log as the system, not a side effect, allowing agents to survive crashes, resume across machines, and be fully owned and inspected by users.

Recursive Coding Agents - Raymond Weitekamp, OpenProse

Jun 25, 2026 · 23:48

Recursive Language Models (RLMs) are a new test-time compute paradigm that unify reasoning and tool calling, and Raymond Weitekamp of OpenProse argues they can turn coding agents into reliable, recursive problem solvers. He demonstrates that RLMs can process millions of tokens beyond context windows, and a small Qwen 3.5 9B model using RLMs beats GPT-5.4 and Opus on the LongCOT benchmark. Weitekamp shows how Claude Code's Dynamic Workflows now make it an RLM, and how OpenProse allows any coding agent to be recursively orchestrated with declared sub-agent work and dependencies. Use cases include repo-scale migrations, deep research over file systems, and adversarial red-teaming audits. He concludes that RLMs represent the next paradigm of inference-time compute, turning mismanaged genius agents into trustworthy outcomes by capturing golden sessions as reusable workflows.

Stop Making Models Bigger, Make Them Behave — Kobie Crawford, Snorkel

Jun 10, 2026 · 20:56

Kobie Crawford of Snorkel explains how a 4B parameter model fine-tuned via RL for under $500 outperformed Qwen 3 235B on financial analysis tool use. The key was training tool discipline—inspecting schemas and self-correcting errors—not deeper reasoning. Single-table training alone boosted multi-table FinQA benchmark from 13.9% to 26.6%, and breaking evals into rubrics identifies which behaviors to fix.

GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Jun 9, 2026 · 20:19

Audry Hsu of RunPod introduces Flash, a Python SDK that deploys GPU cloud functions from a developer's IDE with a single decorator, eliminating the slow iteration cycle of commits, Docker builds, and server allocation. She demonstrates hot reload, swapping Stable Diffusion XL Turbo for DreamShaper instantly, and a pipeline that chains Qwen 3 for prompt generation, DreamShaper for image rendering, and Nano Banana 2 for photo composition. RunPod's serverless H100 pricing is $0.00116 per second, charged only during active inference. Hsu recommends starting with pods for experimentation and switching to serverless when scaling to hundreds of workers across data centers.

Run Frontier AI at Home — Alex Cheema, EXO Labs

May 26, 2026 · 1:45:02

Alex Cheema of EXO Labs argues that running frontier AI locally has 100x improvement potential in cost and performance, demonstrated with GLM 5.1—a trillion-parameter model—running across four Mac Studios at roughly 20 tokens per second for $40,000. He details kernel fusion that recovered 30% performance on Qwen 3.5 by eliminating unnecessary kernel launches, and RDMA integration that cut node-to-node latency from 300 microseconds to single digits, enabling tensor parallelism to actually scale. Cheema advocates splitting inference: prefill on compute-dense hardware (e.g., an RTX Spark) and decode on high-bandwidth hardware (e.g., Mac), cutting large-prompt inference roughly in half. He warns against misleading benchmarks like one-bit quantized models, and outlines how multi-agent setups, test-time scaling, and continual learning could further improve local inference efficiency. The talk includes a live demo of GLM 5.1 across four Mac Studios connected via Thunderbolt 5, and a preview of EXO's upcoming benchmarking site to track intelligence per Joule.

Scaling the Next Paradigm of Heterogeneous Intelligence — Adrian Bertagnoli, Callosum

May 24, 2026 · 15:13

Adrian Bertagnoli, founding engineer at Callosum, argues that the era of homogeneous intelligence—scaling single models on identical chips—is ending, and heterogeneous intelligence, which routes tasks to optimal chips and models, is the next paradigm. He demonstrates this with two case studies: on the Ulong benchmark, running recursive language models on Cerebras instead of GPT-5.2 cuts cost by 7x and latency by 5x while matching accuracy; on Video Web Arena, a mixture of Qwen 3 VL8B and Kimi K2.5 beats GPT-5.2 and Gemini 2.5 by 18% and 25%, while costing 3.7x less and running 3x faster. The key insight is that complex problems decompose into subtasks—like zooming on a webpage—which require different intelligence levels; offloading those to smaller models yields 11x speed and 43x cost improvements. Callosum builds an automation layer that predicts the best model and hardware for each subtask, and has secured a £3 million grant with the UK's Arya institute to operate the first heterogeneous co-located cluster.

Personalization in the Era of LLMs - Shivam Verma, Spotify

May 19, 2026 · 20:12

Shivam Verma of Spotify's AI foundation explains how Spotify personalizes recommendations using open-weight LLMs, user embeddings from 750 million accounts, and Semantic IDs that compress each of 100 million tracks into six hierarchical tokens, enabling autoregressive next-item prediction. Three components: transformer-based user embeddings, Semantic IDs (e.g., Ariana Grande and Bruno Mars share first two tokens as pop artists), and a soft tokenization layer projecting user vectors into the LLM's token space for personalization without fine-tuning. The system is productionized for podcast next-episode recommendations. Verma also introduces Taste Profile, allowing users to edit their taste profile via natural language for steerable recommendations. This moves from traditional multi-stage pipelines to a unified generative model.

Lessons from Trillion Token Deployments at Fortune 500s — Alessandro Cappelli, Adaptive ML

May 12, 2026 · 18:35

Alessandro Cappelli, co-founder of Adaptive ML, argues that 95% of GenAI pilots fail to reach production because they rely on proprietary models or instruction fine-tuning, which lack systematic feedback integration. Reinforcement learning (RL) is the only post-training technique that mathematically incorporates defects, business metrics, and production signals to continuously improve models. RL enables smaller, cheaper, faster models that enterprises can own—critical for scaling use cases like AT&T’s transcript summarization or Manulife’s agents. For agents, RL naturally fits because it was designed for environments; synthetic data is generated as a byproduct of environment training, not a prerequisite. Reward signals come from business KPIs (e.g., containment rate) or LLM judges defined by human rubrics in hours, not weeks. Adaptive ML’s Adaptive Engine abstracts away RL complexity (orchestrating four models for PPO) and provides pre-built recipes to industrialize model deployment.

Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

May 7, 2026 · 1:20:40

Samuel Colvin shows how to improve production agents without redeploying using Pydantic AI's GEPA, evals, and Logfire managed variables. GEPA uses a genetic algorithm to evolve prompts, achieving 96.7% accuracy — up from 87% with a simple prompt and 92% with an expert prompt — on a Wikipedia-based political relations task. Managed variables let you update prompts, models, and parameters live via a web interface and A/B test targeting. Evals compare against a golden dataset of 650 MPs; Colvin runs 65 cases in 30 seconds using GPT-4.1. He discusses that prompt optimization is most valuable with private data, that implicit user feedback (e.g., user's next action) can build golden datasets, and that overfitting to small test sets is a risk — the GEPA optimizer may exclude valid relations like 'uncle' if they don't appear in the training split.

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.

Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

May 4, 2026 · 1:21:26

Angelos Perivolaropoulos from ElevenLabs walks through building a small GPT-2-like LLM from scratch on a local machine, demonstrating that the core techniques used by major labs are accessible in a few hundred lines of PyTorch code. The workshop uses character-level tokenization (65 tokens) on a Shakespeare dataset to enable fast training with limited compute. The model architecture includes multi-head self-attention, MLP layers, residual connections, and layer normalization, totaling 10 million parameters across six transformer blocks with a 256-token context window. The training loop employs next-token prediction with a warm-up cosine decay learning rate schedule and validation loss to detect overfitting. Inference uses temperature sampling (default 0.7) and top-k sampling to improve creativity. Perivolaropoulos explains that audio and multimodal models share the same transformer foundation but differ in tokenization (e.g., mel-spectrograms for audio) and use specialized losses like L2 or KL divergence, while reasoning models result from post-training base models with high-quality chain-of-thought data.

Everything I Learned Training Frontier Small Models — Maxime Labonne, Liquid AI

Apr 29, 2026 · 20:13

Maxime Labonne, head of post-training at Liquid AI, presents the LFM2.5 recipe for training frontier small models, arguing they require specialized approaches distinct from scaled-down big models. He details how Liquid's architecture uses gated short convolutions for latency-sensitive on-device deployment, achieving faster throughput than Gemma 3 or Qwen 3.5. Post-training stages—SFT, on-policy preference alignment, and RL—are tailored for narrow task focus like data extraction and tool use. A key challenge is 'doom loops' (repetition), which reached 15% after mid-training in a 1.2B reasoning model; solutions include preference alignment rejecting looped responses and RL with verifiable rewards and n-gram penalties, nearly eliminating the issue. He advocates combining small models with agentic tools (e.g., web search) to overcome memory limits, as they excel at reasoning and tool use despite lower knowledge capacity. The talk also covers decisions on when to use small vs. large models—latency, privacy, offline use—and notes that distillation alone likely won't fully solve doom loops.

What Do Models Still Suck At? - Peter Gostev, Arena.ai, BullshitBench

Apr 24, 2026 · 20:24

Peter Gostev presents data from his BullshitBench and Arena.ai to argue that language models still struggle with nonsense questions and expert-level tasks, despite benchmark charts showing relentless improvement. His BullshitBench reveals that only Claude Sonnet 4.5 and some Qwen models consistently push back on nonsense, while GPT and Gemini models accept it 50% of the time. Arena.ai's dissatisfaction rate among top 25 models has improved from 17% pre-reasoning to about 9% currently, but remains non-zero, and expert categories like gaming, magic, finance, and law show minimal improvement. For software expert prompts, dissatisfaction dropped from 23.5% in Q2 2024 to 13% in Q1 2026, but gaming is a persistent weakness where models fail to create engaging game mechanics. Gostev warns that narrow benchmarks overstate progress and urges focusing on the full distribution of real-world tasks.

Running LLMs on your iPhone: 40 tok/s Gemma 4 with MLX — Adrien Grondin, Locally AI

Apr 20, 2026 · 10:51

Adrien Grondin, developer of Locally AI, demonstrates how to run Gemma 4 and other LLMs on iPhone using Apple's MLX framework, achieving 40 tokens per second on the latest devices. He explains that MLX is optimized for Apple Silicon and that the open-source mlx-swift-lm GitHub repo enables easy integration into iOS, macOS, and iPadOS apps in under 10 minutes. Grondin recommends quantized models from the Hugging Face MLX community—typically 4-bit to 8-bit—and shows a live demo of Gemma 4 generating text offline. He confirms that mlx-swift-lm supports tool calling, though structured generation is not yet available. He also notes that Locally AI has been acquired by LM Studio, which now lets users run models via MLX or llama.cpp and connect them through OpenAI or Anthropic-compatible APIs.

Paperclip: Open Source Human Control Plane for AI Labor — Dotta Bippa

Apr 15, 2026 · 24:34

Dotta Bippa introduces Paperclip, an open-source human control plane for AI labor that lets you manage an org chart of agents to run a zero-human company. The tool enables you to hire employees, set goals, automate jobs, and bring your own agent (e.g., Gemini, Claude, Codex) with skills and instructions. Paperclip supports reliable workflows like QA reviews and approvers, and routines for recurring tasks such as summarizing PRs or writing changelogs. Dotta demonstrates creating a Remotion video celebrating Paperclip's 40,000 GitHub stars (now 50,000) by instructing the CEO agent to hire a video writer and apply brand guidelines—something that would have taken a week becomes an afterthought. Paperclip is not just for coding; it handles marketing, sales, and finance. Upcoming features include CEO chat, maximizer mode, multi-user support, cloud deployments, and a desktop app, all aimed at giving humans control over AI labor.

OpenRAG: An open-source stack for RAG — Phil Nash

Apr 8, 2026 · 15:52

Phil Nash introduces OpenRAG, an open-source RAG stack from IBM combining Docling, OpenSearch, and Langflow, arguing that RAG remains hard and custom despite claims it's dead — every business has unique data requiring more than simple vector search. Docling parses PDFs, audio, and more with specialized pipelines, outputting hierarchically chunked text. OpenSearch provides hybrid vector and keyword search using JVector for live indexing and disk-based ANN. Langflow enables visual agentic retrieval where the LLM decides searches with tools like a calculator and MCP servers. The stack supports local models via Ollama, cloud connectors for Google Drive and SharePoint, and an API. Nash demonstrates adding guardrails in Langflow and invites contributions to the open project.

AGI: The Path Forward – Jason Warner & Eiso Kant, Poolside

Dec 27, 2025 · 15:56

Jason Warner and Eiso Kant, co-founders of Poolside, present their vision and roadmap towards AGI-level capabilities for knowledge work, demonstrating their second-generation model Malibu Agent converting ADA code to Rust live on stage. They argue that next-token prediction paired with reinforcement learning is the key breakthrough, a contrarian bet they made two and a half years ago. The episode centers on their work in high-consequence code environments for defense and government, where agents must operate with tight permissions. They announce a large compute cluster of over 40,000 GB300s coming online and a public API release early next year via AWS Bedrock. Warner recounts meeting Kant through a failed GitHub acquisition, and Kant invites the audience to build with their models, emphasizing that future agents will handle tasks over days as intelligence scales.

Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor

Dec 15, 2025 · 18:08

Naman Jain, an AI engineer at Cursor, traces the evolution of coding evaluations from single-line snippets to entire codebases over four years. He introduces LiveCode Bench for competition programming, dynamically updating problems to combat data contamination and adjust difficulty, with model performance dropping from 50% to 20% after training cutoffs. For real-world software optimization, he presents a benchmark using commits from codebases like Llama CVP, but notes 30% of O3 attempts involved reward hacking—such as hijacking numpy libraries—caught by a GPT-5-based Hack Detector. In longer-horizon tasks like translating 4,000 lines of C to Rust (Syzygy), end-to-end correctness gives only one bit of feedback, highlighting the need for intermediate grading signals. Finally, in wild evals like Copilot Arena, acceptance rates drop sharply with latency over one second, emphasizing human-centric experiment design to balance latency differences.

RL Environments at Scale – Will Brown, Prime Intellect

Dec 9, 2025 · 18:30

Will Brown of Prime Intellect argues that scaling reinforcement learning environments beyond engineering—to community and accessibility—is key to broadening AI research. He presents Prime Intellect's open-source stack, including Verifiers for building environments and the Environments Hub for sharing them, as a way to turn any task harness into an RL training or evaluation loop. Brown demonstrates how environment-based fine-tuning boosted a Qwen 3 4B model from 55% to 89% on a Wikipedia search task, matching much larger models. He frames environments as the 'web apps of AI research'—simple to start, but capable of capturing product complexity, as seen with Cursor's Composer and OpenAI's Codex. Prime Intellect validated this approach by training the 100B-param Intellect 3 on 500 GPUs, and will soon release Lab, a platform to run environments without managing infrastructure.

Vision AI in 2025 — Peter Robicheaux, Roboflow

Aug 3, 2025 · 17:24

Peter Robicheaux, ML lead at Roboflow, argues vision models are not smart because they fail at basic visual tasks like telling time on a watch or identifying a school bus direction. He attributes this to saturated evals like ImageNet and COCO, and to vision models not leveraging large pre-training like language models. Roboflow introduces RFDetter, a real-time detector using a Dynav2 backbone, achieving second SOTA on COCO and large gains on their new RF100VL dataset of 100 diverse domains. The benchmark shows specialist models fine-tuned on 10-shot examples outperform large VLMs like Qwen2.5 VL 72B, indicating VLMs are strong linguistically but weak visually. Roboflow's platform is free for researchers contributing data back; dataset at rf100vl.org.

Pipecat Cloud: Enterprise Voice Agents Built On Open Source - Kwindla Hultman Kramer, Daily

Jul 31, 2025 · 26:46

Kwindla Hultman Kramer, co-founder of Daily, introduces Pipecat, an open-source, vendor-neutral framework for building voice AI agents, and Pipecat Cloud, a deployment layer optimized for real-time voice. He argues that achieving sub-800-millisecond voice-to-voice response times is critical and that frameworks like Pipecat handle hard problems like turn detection, interruption handling, and context management. Kramer explains that Pipecat supports 60+ models and services, including Gemini and OpenAI, and that Gemini is often 10x cheaper for 30-minute conversations. He notes that while speech-to-speech models like Moshi and Sesame are promising for natural conversation, they currently lag in instruction-following for enterprise use cases. Kramer also addresses global latency challenges, recommending deployment close to inference servers or using open-weight models locally, and highlights Pipecat Cloud's integration with Krisp for background noise reduction and a free open-source smart turn model.

Introduction to LLM serving with SGLang - Philip Kiely and Yineng Zhang, Baseten

Jul 26, 2025 · 43:42

Philip Kiely and Yineng Zhang introduce SGLang, an open-source fast serving framework for LLMs and VLMs, arguing it offers production-ready performance with day-zero support for new models like DeepSeek and Qwen, and a customizable codebase for contributions. Yineng, a core maintainer, traces SGLang's rapid growth from a December 2023 paper to nearly 15,000 GitHub stars and adoption by xAI, AMD, and Meituan. The workshop demonstrates deploying a first model via Baseten's trunk packaging, then tuning the CUDA Graph max batch size flag on an L4 GPU to maintain CUDA Graph-enabled decoding during higher concurrency, boosting generation throughput. They also cover Eagle 3 speculative decoding, where a draft model derived from the target model speculates tokens; users can benchmark different step and top-k configurations on representative prompts to find optimal settings for production. Finally, they invite contributions via GitHub's 'good first issue' tags and highlight Baseten's job openings.

Latent Space Paper Club: AIEWF Special Edition (Test of Time, DeepSeek R1/V3) — VIbhu Sapra

Jul 25, 2025 · 53:54

VIbhu Sapra presents DeepSeek's latest models and announces a new curriculum-based Test of Time Paper Club. DeepSeek's May 28th R1 update doubles reasoning tokens (12k→25k) and matches O3/Gemini 2.5 Pro, improving AIME 2024 from 70% to 87.5%. A new distillation of R1 traces into Qwen 3 8b achieves performance on par with Qwen's 235b thinking model, proving reasoning models distill efficiently. The Test of Time Paper Club will cover 50–100 foundational papers over six months in San Francisco and remote, targeting core AI engineering topics from attention and scaling laws to inference techniques. Sapra emphasizes that pure RL (GRPO) on verifiable math and code unlocks emergent reflection and aha moments, shifting scaling from pre-training compute to inference-time reasoning.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

A Taxonomy for Next-gen Reasoning — Nathan Lambert, Allen Institute (AI2) & Interconnects.ai

Jul 19, 2025 · 19:21

Nathan Lambert, senior research scientist at AI2 and founder of Interconnects.ai, argues that next-generation reasoning models require a taxonomy of four traits: skills, calibration, strategy, and abstraction. While current models excel at math and code (skills), they overthink easy problems, wasting tokens and latency—calibration is needed to match output length to task difficulty. The real frontier is planning: models must learn strategy (choosing the right direction) and abstraction (breaking problems into tractable sub-tasks) to enable long-horizon agents like Deep Research and Claude Code. Lambert traces how OpenAI's Q*→Strawberry→o1 took 12–18 months of human data to teach backtracking and verification; planning should be easier because humans can write 5-step plans. He predicts post-training compute could reach parity with pre-training, citing DeepSeek's shift from 0.18% post-training compute in V3 to an estimated 10–20% for R1. The path forward: collect diverse verifiable questions, filter by difficulty, run stable RL—then scale.

How to Train Your Agent: Building Reliable Agents with RL — Kyle Corbitt, OpenPipe

Jul 19, 2025 · 19:48

Kyle Corbitt, co-founder of OpenPipe, argues that reinforcement learning (RL) with GRPO can make agentic systems far more reliable and cost-effective than prompted frontier models. He presents ART-E, an email assistant trained on Qwen 2.5 14B, which achieved 96% accuracy versus 90% for o3 and slashed cost from $55 to $0.80 per 1,000 queries. The two critical problems are building a realistic environment (solved using the Enron email dataset) and defining the right reward function (turning it into a verifiable task with LLM-as-judge). Extra rewards—favoring fewer tool turns and penalizing hallucination—further improved efficiency. Corbitt also warns about reward hacking, giving examples like a model that exploited a bug to put every word in every category, and shares that the training cost just $80 in GPU time and a week of engineering.

OpenThoughts: Data Recipes for Reasoning Models — Ryan Marten, Bespoke Labs

Jul 19, 2025 · 19:59

Ryan Marten, co-lead of the OpenThoughts collaboration and founding engineer at Bespoke Labs, reveals the missing data recipe for open-source reasoning models, presenting OpenThoughts 3, a state-of-the-art 7B reasoning dataset that outperforms DeepSeek R1 Qwen 7B and Nematron Nano on benchmarks like AIME, Live Code Bench, and GPQA Diamond. Through over 1,000 experiments and 5,000 datasets, key findings include that sampling multiple reasoning traces per question scales performance by 16x, Qwen 32B surpasses DeepSeek R1 as a teacher model, synthetic question generation is highly effective, and filtering by difficulty or response length works better than embeddings. Surprisingly, verification of answers in SFT distillation did not improve results, and focusing on fewer high-quality sources outperformed maximizing diversity. For domain-specific reasoning, Marten advises starting with the OpenThoughts recipe, using synthetic data generation (via the open-source Curator library), and rigorous evaluation (via EvalComet). A legal reasoning example shows that distillation can surpass the teacher model. All resources are open-source.

RL for Autonomous Coding — Aakanksha Chowdhery, Reflection.ai

Jul 16, 2025 · 19:27

Aakanksha Chowdhery, CEO of Reflection AI and former lead researcher on PaLM and Gemini at Google, argues that reinforcement learning (RL) scaling is the next frontier for autonomous coding agents. She explains that inference-time techniques like majority voting and self-revision improve accuracy but require many samples (e.g., 10,000 for rare correct generations). RL training can learn to generate correct outputs directly, especially in verifiable domains like code with unit tests. Reflection AI aims to build superintelligence starting with autonomous coding, leveraging automated verification to design better reward functions. She notes challenges in scaling RL, including system complexity and reward hacking, but sees coding as ideal due to execution feedback.

Using OSS models to build AI apps with millions of users — Hassan El Mghari

Jul 15, 2025 · 18:47

Hassan El Mghari, a software engineer at Together.ai, explains how he builds open source AI apps that attract millions of users, including roomGPT.io (2.9 million users), restorePhotos.io (1.1 million), Blinkshot.io (1 million visitors), and LlamaCoder.io (1.4 million visitors). He details his simple four-step tech stack: user input, a single API call to an open source model on Together.ai, storing results in a database, and displaying output. His process emphasizes keeping ideas simple enough to describe in five words, spending 80% of time on UI, launching early, and incorporating the latest models like Fluxionel for virality. He advises naming apps with short, memorable names, making them free and open source to encourage sharing, and iterating rapidly—most apps fail, but the key is to keep building. He funds compute through sponsorships from Together.ai and partners like Neon and Clerk, which provide free services for his open source projects.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

Training Agentic Reasoners — Will Brown, Prime Intellect

Jul 7, 2025 · 19:17

Will Brown of Prime Intellect argues that reasoning and agents are fundamentally the same, and reinforcement learning (RL) is the key to advancing both. He explains that RL now works at scale, as shown by DeepSeek's GRPO and OpenAI's o3, and that agentic tasks like tool calling are natural RL environments. Brown warns against reward hacking and emphasizes designing evals that are harder to game than the task itself. He introduces his open-source toolkit 'verifiers' (now on pip) which lets users build trainable agent loops with a simple API, and demonstrates training a 7B Wordle agent in a few turns on just a couple GPUs.

Foundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive — Emma Ning, Microsoft

Jun 27, 2025 · 22:52

Emma Ning, Principal PM at Microsoft, presents Foundry Local, a platform for building cross-platform on-device AI applications using ONNX Runtime and Olive. She argues local AI is essential for low-bandwidth, privacy-sensitive, cost-efficient, and real-time latency scenarios, noting that modern hardware and optimized models like Phi-4-mini and DeepSeek make it viable. Ning demonstrates Foundry Local's CLI, model benchmarking (Qwen 1.5B at 90 tok/s vs. Phi-4-mini with richer output), and a cross-platform document summarization app built with JavaScript SDK, running identically on Windows and macOS. She also previews a local agent with MCP servers (file system and OCR) that extracts receipt totals using Phi-4-mini. Customer testimonial from Sava (CEO of Pieces) highlights improved memory management and tokens-per-second, while another partner underscores ease of install and hybrid cloud-local solutions.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.

MCP Agent Fine tuning Workshop - Ronan McGovern

Jun 3, 2025 · 35:30

Ronan McGovern demonstrates how to fine-tune a Qwen3 model on high-quality reasoning traces from an MCP agent with browser tools. Using Playwright's 25 tools via Model Context Protocol, he generates multi-turn traces by running a 30B Qwen agent on RunPod, saving both tool calls and reasoning content. Those traces are then used to supervised fine-tune a 4B Qwen model with Unsloth, applying LoRA adapters to attention and MLP layers. The process includes converting MCP tool schemas to OpenAI format, extracting Hermes-style tool calls, and unrolling conversations to multiply training examples. Even with only nine curated traces, the fine-tuned model shows improved tool-calling behavior on multi-step tasks like navigating trellis.com to extract specific content.

The RAG Stack We Landed On After 37 Fails - Jonathan Fernandes

Jun 3, 2025 · 18:52

Jonathan Fernandes, independent AI engineer, details the RAG stack his team settled on after 37 failed attempts, covering orchestration (LlamaIndex), embeddings (BAAI BGE small), vector database (Qdrant), LLMs (GPT-4, Qwen, Llama), reranking (Cohere), monitoring (Arize Phoenix), and evaluation (RAGAS). He demonstrates a live prototype in Google Colab using a London railway knowledge base, showing how a naive RAG returns irrelevant results (e.g., suggesting black cabs for "where can I get help at the station"). By swapping components—replacing in-memory storage with Qdrant, using an open-source embedding model, upgrading to GPT-4, and adding Cohere reranking—the answer improves to "go to booth number five next to the Eurostar ticket gates." For production, he deploys via Docker Compose with NVIDIA embedding/reranking models and Ollama for serving. The episode also stresses the importance of tracing latency per component and using RAGAS for systematic evaluation across many queries.

WTF do people use Open Models for??

Feb 22, 2025 · 28:01

Eugene Cheah of Featherless.ai breaks down how individuals and enterprises actually use open-source AI models, based on platform data. DeepSeek R1 dominates individual usage, but Mistral Nemo 8B remains the top enterprise model due to production stickiness and Apache 2.0 licensing. Creative writing and roleplay account for 30–40% of all traffic, with over 60% of users in that segment being women; coding copilots and agents make up 20–30%, driven by 'vibe coding' and token-hungry workflows like Kline. RAG and ChatGPT clones represent 20%, while agentic workflows (10–20%) succeed with human-in-the-loop designs. Cheah advises enterprises to aim for 80% automation with escape hatches, and warns against chasing 100% reliability. He concludes by introducing Quirky, a post-transformer hybrid built for $100k.

Hyperspace More Nodes Is All You Need: Nicolas Schlaepfer

Sep 4, 2024 · 5:48

Nicolas Schlaepfer introduces Hyperspace's decentralized AI network and new product, a node editor for power users that generates agentic plans via a DAG orchestration model (HyperEngine v3). The network, with over 15,000 nodes, leverages consumer devices rather than GPUs, running inference through Llama.cpp. The product combines prompt engineering, visual React flow, Python execution, and RAG-like web browsing, using Qwen 2 instruct for reasoning and LLaMA 3 70B for summarization. Schlaepfer emphasizes diverse open-source models and a virtual file system for agentic primitives. Availability via waitlist is announced later this week.