Code & Developer Tools

HTML Is All Agents Need — James Russo, HeyGen

Jul 21, 2026 · 15:13

James Russo, software engineer at HeyGen, argues that HTML, CSS, and JavaScript are the native languages of LLMs and all agents need to create great videos—leading to HyperFrames, an open-source framework that turns agent-authored HTML into deterministic MP4s. By letting the small Gemini 3 Flash model author code first, they ensured the thinnest wrapper won, with only a few data attributes for timing. Rendering works by freezing the browser clock and seeking frame by frame, ensuring everything loads before capturing each frame—enabling deterministic video from any web technology (Three.js, WebGL, Lottie). HyperFrames skills focus on taste and video craft rather than teaching frameworks, raising the floor for single-shot output. The framework has already rendered over 1.3 million videos in 90 days from 267,000 creators, with 15,000 daily renders and 32,000 GitHub stars—proving the approach at scale.

In the Land of AI Agents, the Verifiers Are King — Tariq Shaukat, Sonar

Jul 20, 2026 · 18:53

Tariq Shaukat, CEO of Sonar, argues that rigorous verification is the key to unlocking sustained value from AI coding agents, introducing the Agent-Centric Development Cycle (AC/DC) framework: Guide, Verify, Solve. He warns that without it, the initial 3–5x productivity boost from agents dissipates within three months as technical debt—security issues, maintainability issues, and complexity—compounds. Sonar's tests show that multi-layered, zero-trust verification (combining algorithmic and agentic methods) reduces AI-derived production outages by 44% and, in a trial with a large bank, achieved a 92% reduction in issues. Clean codebases, he argues, make agents faster and cheaper, with over 30% fewer tokens consumed when guided by context and constraints. The framework embeds verification into three loops—agentic, CI, and code maintenance—to create a self-reinforcing positive cycle, countering the downward spiral of neglected code quality.

From Blind Spots to Merged PRs: Continuous Agentic Performance Optimization - May Walter, Hud

Jul 19, 2026 · 22:46

May Walter, CTO of Hud, presents a case study on using coding agents with runtime intelligence to automate continuous performance optimization in production. The system analyzes real production traces, queries, and latencies to surface high-ROI fixes like N+1 queries and missing indexes, scoring them by complexity and impact. It runs weekly via GitHub Actions and Claude Code, generating reports with verified fixes and evidence of impact, such as P90 latency improvements after deployment. Key challenges included 'plausible unverified' fixes, complex ClickHouse queries, and lazy fixes that only mask exceptions rather than root causes. Walter emphasizes the need for prod-to-code context mapping, scoring guardrails, and human review to achieve reliable automation. The talk concludes that context over cleverness works, and agents can automate the investigation phase that teams rarely do proactively, turning dormant performance debt into actionable merged PRs.

Content Is Code - Matt Palmer, Conductor

Jul 18, 2026 · 10:53

Matt Palmer, head of developer experience at Conductor, argues that code is now the fastest way to produce technical content, shifting left to code as the source of truth. He outlines three eras of content creation—handcrafted, expensive code, and cheap code with AI—and claims that the scarce resource today is not code but structure and conscientiousness, meaning meticulous care in maintaining design tokens, brand guidelines, and clean codebases. AI rewards organizational excellence over raw technical skill, making the best communicating teams those that instill discipline and rigor into their software creation. Palmer predicts 2026 as the year of the creative technologist and 2027 as the year of the content engineer, with declarative pipelines that automatically generate documentation, walkthroughs, product tours, and updates from structured code.

The Great Loops Debate — Dex Horthy, Geoff Huntley, Ian Livingstone, Greg Pstrucha, @insecure-agents

Jul 17, 2026 · 1:00:16

In the 'Great Loops Debate' hosted by Ali Howe, Geoff Huntley, Ian Livingstone, Dex Horthy, and Greg Pstrucha argue whether there is a delta between the hype behind loops in AI-assisted development and what works in practice. Team No Delta (Ian, Geoff) contends loops are inevitable and already cost-effective at $10.42/hour, with models surpassing average coders. Team Delta (Dex, Greg) counters that hype outruns discipline: loops work only for deterministic tasks, quality remains low, and token spend scales unsustainably. The debate covers loop history, anatomy (verification, context windows), and the feasibility of fully autonomous software factories, concluding that while loops boost productivity 2-3x, the vision of lights-off factories is premature.

"Software engineering is not about writing code" — Benoit Schillings, Google DeepMind VP of Research

Jul 17, 2026 · 20:26

Benoit Schillings, VP of Technology at Google DeepMind, argues that the era of syntax generation is over, with AI now capable of superhuman code writing, shifting the bottleneck to architecture and validation. He explains that 80% of new code on GitHub is machine-generated, leading to a saturation of human training data and the need for self-play techniques inspired by AlphaZero. Schillings highlights DeepMind's use of self-play to generate and verify coding challenges, enabling models to reach superhuman performance. He emphasizes the shift to inductive architecture, where models must plan, decompose complex problems, and transfer knowledge across domains. Additionally, he calls for new programming languages designed for models rather than humans, and explores how AI's rapid experimentation will transform fields like chemistry and biology by uncovering patterns invisible to humans.

Imagination Engineering — Eve Bouffard, Head of Design, Y Combinator

Jul 16, 2026 · 16:04

Eve Bouffard, Head of Design at Y Combinator, introduces 'Imagination Engineering' as the art of stretching the mind to invent what seems impossible, arguing that idea generation is the new bottleneck as AI models become incredibly capable. She shares her experiment in 'thinking in public' with a Slack channel called 'Eve Thoughts,' where she dumps her stream of consciousness, and then used Opus 4.8 to build a personalized website (EveBouffard.com) that aggregates and visualizes those thoughts. The site dynamically surfaces her projects, quotes, tools, and books, and even applies shaders and translations. She also built 'Shape of Minds'—a tool that analyzes commonalities across history's greatest minds, revealing patterns like taking naps and barely eating. Bouffard demonstrates how to spin up agents on demand for learning and productivity, emphasizing that anyone can now create software on the fly from their stream of consciousness.

Claude Fable, Claude Tag, and Anthropic's Culture — Cat Wu & Thariq Shihipar ft Simon Willison

Jul 15, 2026 · 51:30

This episode features Anthropic's Cat Wu and Thariq Shihipar discussing Claude Code, Claude Tag, and Claude Fable, arguing that these tools have shifted engineering from slow spec-driven processes to rapid, ambitious building. Thariq notes that with each model generation, delegation increased, and Claude Fable now enables one-shot features. Cat says engineers now need product taste over execution, as timelines shrink from six months to a week. Claude Tag, a proactive multiplayer agent, lands 65% of product engineering PRs internally by monitoring channels and remembering preferences. The team reduced Claude Code's system prompt by 80% for frontier models by removing examples and hard constraints, relying on model judgment. Auto mode, used internally since January, mitigates prompt injection through thousands of evals and Sonnet classifiers. Cultural hacks include default-public channels and a 'don't negotiate against ourselves' mindset, leading to ambitious builds like Thariq's Claude-powered video editing and a Street Fighter game.

RLM: Recursive Language Models for Large Codebases - Shashi, Superagentic AI

Jul 12, 2026 · 17:27

Shashi from Superagentic AI explains how Recursive Language Models (RLM), a pattern from MIT, solve the context window problem in large codebases by externalizing context management into a programmable REPL where the model writes code to inspect, slice, and compute relevant chunks, and recursively delegates sub-questions via llm_query. The RLM loop loads the repo as data, produces bounded observations, and can recursively call another model for deeper insights. Shashi demonstrates RLM Code, an open-source independent implementation running locally and on Gemini with a Docker sandbox, showing two tool calls and a complete trajectory. He notes that similar RLM concepts are used in proprietary systems like Codex, Claude managed agents, and dynamic workflows, making it a practical pattern for AI engineers dealing with monorepos.

A Song of Types and Agents - Roberto Stagi, Ratel

Jul 12, 2026 · 14:16

Roberto Stagi (Ratel) argues that TypeScript is overtaking Python as the dominant language for building AI agents and applications, driven by the rise of coding agents like Lovable, Claude Code, and Cursor that default to TypeScript. He notes that Python still rules training, research, and GPU serving, but the application layer—where AI is shipped inside products—now belongs to TypeScript. Stagi highlights that TypeScript surpassed Python as the most used language on GitHub in August 2025, with GitHub attributing the shift to AI agents. He cites the Vercel AI SDK growing from 1.6M to 15.1M weekly downloads in one year and npm's rich ecosystem of authentication, payments, and UI packages as reasons to build agents in TypeScript. Consistent end-to-end typing via Zod and a unified codebase for agent loops, tools, backends, and UIs further strengthen the case. Stagi concludes by invoking Atwood's Law: any application that can be written in JavaScript will eventually be written in TypeScript, urging developers to build agents in TypeScript to avoid falling behind.

ReviewDebt: a practical framework for scoring every pull request — Sachin Gupta, Ebay

Jul 12, 2026 · 25:00

Sachin Gupta introduces ReviewDebt, a deterministic framework scoring every pull request on the hidden review burden it creates, arguing that coding agents produce code faster than humans can responsibly review, accumulating an unmeasured debt. He defines 5 signal families—diff size and coupling, test evidence gap, directory and ownership spread, AI authorship indicators, and evidence and rational gaps—with 10 deterministic checks computable from PR data without LLMs, to produce a 0–100 score. Real PR examples show a clean human PR scoring 0, a high-debt AI PR scoring 60 with 86 estimated review minutes, and a well-shaped AI PR scoring 7. A cross-repo scan of 524 PRs over 27–90 days found 228 senior reviewer hours accumulated, with 5–20% of PRs weekly firing AI indicators but none of those landing in the high-burden band—complexity, not authorship, drives burden. Gupta advises teams to backfill the scorer on last 200 PRs, set a threshold (e.g., 50), surface scores as PR comments, aggregate weekly per team, and bring the number to retrospectives, turning vibe into measurement for 2027's governance conversation.

remobi.app: Don't change your terminal workflow for mobile

Jul 12, 2026 · 9:21

Connor Adams presents remobi.app, a progressive web app that lets developers check on their terminal-based coding agents from a mobile phone without changing their existing workflow. Built for TMUX users, it mirrors a user's remote terminal session with gesture controls for zoom, scroll, and pane switching. Adams argues that unlike Claude Code's built-in handoff or dedicated mobile terminals, remobi is agent-agnostic and preserves custom TMUX setups with tools like lazygit and critique. The app uses Tailscale by default for secure remote access, though users can substitute Ngrok or Cloudflare tunnels. Open source and installable via npm, remobi aims to satisfy the compulsive need to monitor agents while away from the desk.

Develop at Idea Velocity - Jeffrey Lee-Chan, Snapchat

Jul 11, 2026 · 15:28

Jeffrey Lee-Chan (Snapchat) and Austin (CMUX) explain how to achieve 'idea velocity' by building parallel multi-agent harnesses that let one engineer direct 10–20 coding agents instead of becoming the bottleneck. Lee-Chan argues the key is separating 'Agent Orchestrator Managers' from specialized workers to prevent low-level implementation bias, using OpenClaw for frictionless Slack-based communication and CMUX terminals for real-time parallelization and unbiased manager oversight. He demonstrates two apps—an AI RPG with dice-roll mechanics and a multi-AI analysis tool—built with this stack, and discusses token-burn tradeoffs between models like Codex 5.3, GPT-5.4, and Minimax. The episode also covers staging environments to avoid doubling token usage, the benefit of persistent memory and contextual guardrails, and the importance of pushing human interaction to the start or end of the workflow for improved parallelization.

Understanding is the new bottleneck — Geoffrey Litt, Notion

Jul 10, 2026 · 19:33

Geoffrey Litt from Notion argues that understanding code written by AI agents is crucial for creative participation, not just correctness checking. He presents three techniques: explanations via his 'explain diff' skill, which generates HTML or Notion documents with background, intuition, interactive figures, and literate code diffs, capped by a five-question quiz to ensure understanding before sharing code; micro-worlds, like a step-by-step debugger for a Prolog interpreter or a 'video game' that walks through migrating a website; and shared spaces in Notion where humans and agents can collaborate in multiplayer chat threads and comment on documents. Litt emphasizes that agents can build ephemeral UIs and simulations to help humans understand better, leveraging education principles to stay deeply in the loop.

Should AI Engineers Still Read Code in 2026? The Z/L Continuum — Alex Volkov, ThursdAI

Jul 10, 2026 · 21:35

Alex Volkov examines the debate sparked by Ryan Lopopolo's claim that 'code is free' and Mario Zechner's counter that engineers must 'read every fucking line' of critical code, arguing that the Z/L Continuum is about task-level proof rather than personality. Citing a Ferrous AI survey showing an 861% increase in code deletion per PR and a 242% rise in incidents, he notes Anthropic's recursive self-improvement essay admitting human code review is a new bottleneck. Volkov's routing table prescribes reading every line for authentication, money movement, and irreversible data, while letting agents handle less critical changes. He also introduces loops—cron-like agent systems that self-verify—as the next frontier, quoting Adi Osmani that automated loops don't remove judgment. Volkov concludes that capability drift moves where proof belongs, but every system still requires human judgment.

The Golden Age of AI Engineering — Alexander Embiricos & Romain Huet & Peter Steinberger, OpenAI

Jul 9, 2026 · 25:13

OpenAI's Alexander Embiricos, Romain Huet, and Peter Steinberger reveal how AI engineering is entering its golden age, with model releases accelerating from every 15 months to roughly every 6 weeks and frontier-level intelligence now available at $1 per million input tokens and $6 per million output tokens. They demonstrate Codex's evolution from a model that couldn't test its own code to one generating 750 tokens per second, enabling a 10-second pull request, and emphasize that the product is built on the same open-source stack—responses API, AGENTS.md format, and harness—that ships to developers. The trio argue that the future is not about automating engineers but empowering them via value maxing over token maxing, with agents moving from local/cloud distinctions to persistent, long-running managers that orchestrate work and only require human steering at key decision points, as illustrated by an automated open-source issue resolution workflow.

Building an ACP-Compatible Agent Live — Bennet Fenner, Zed

Jul 8, 2026 · 18:19

Bennet Fenner from Zed demonstrates building an ACP-compatible coding agent live, showing how the Agent-Client Protocol lets agents and editors communicate over JSON-RPC. He starts with a minimal agent that reads and edits files, then adds ACP support using the TypeScript SDK—implementing initialize, session creation, and prompt handling. Next, he streams model tokens via session updates, emits tool call progress with file paths and diffs, and proxies file system operations through the client. The demo culminates in the agent bootstrapping itself to add a terminal tool, which then runs shell commands inside Zed. ACP currently works over stdio; a remote transport is in development.

The Pipeline Is Dead - Iris ten Teije, Sky Valley Ambient Computing

Jul 7, 2026 · 19:50

Iris ten Teije, co-founder of Differ, argues that the traditional software pipeline—built on one frozen artifact for everyone—is dying because producing code changes has become nearly free. This shift enables per-user adaptive software: instead of one codebase shipped to all, a canonical stem plus bounded, isolated, and reversible divergences adapt live per user. She explains how this dissolves the line between development and distribution, using examples like a CRM that learns an investor’s intro paths and skips unused fields. Key challenges include provenance (every divergence must be attributable and inspectable), correctness testing at scale, desirability measurement, and coordination—not merging code but merging intent and outcome. The hard part isn’t AI-generated code; it’s the observability, validation, and isolation that make per-user versions safe rather than terrifying.

500 people vibe-coded for 30 days. I was one of them. - Sanja Grbic, Automattic

Jul 7, 2026 · 17:49

In Automattic's Radical Speed Month, a 30-day experiment where over 500 employees paused roadmap work to ship real projects, product designer Sanja Grbic built three distinct products—a board game session manager, a design system status tracker, and an iOS chat for WooCommerce merchants—demonstrating how AI tools like Claude Code and ContextAIC can transform roles and unlock speed. The first project showed that engineers become enablers when they set up non-engineers for coding success. The second project, completed in two and a half weeks, moved Grbic from designer to design engineer by owning the full build, from proof of concept to deployment. The third project, built in just six days with a fellow designer, proved that shared understanding of AI workflows collapses iteration time. Grbic argues that large organizations can replicate this by providing tool access, finding enablers, creating space for experimentation, and granting agency—shifting human behavior to break out of established habits and release the velocity AI offers.

Field Guide to Fable — Thariq Shihipar, Anthropic

Jul 6, 2026 · 19:28

Thariq Shihipar of Anthropic discusses Fable and argues models improve in "spiky" ways: a chat model fails to list Pokémon ending in "aw" (Croconaw and Dreadnaw), but Claude Code fetches and filters the list in seconds – a gap he calls "capability overhang." He explains that to unlock Fable, Claude Code cut 80% of its system prompt because heavy instructions now constrain a more imaginative model, and the "ask user question" tool evolved from barely working under Opus 4 to generating embedded HTML questionnaires. He shares techniques like blind-spot passes and interviews to surface unknown unknowns, and reflects on the grief of moving from hand-coded programming to agentic workflows. Shihipar urges engineers to reject trade-offs – "good, fast, cheap: pick three" – and instead demand all three, citing a four-hour keynote deck built with Fable as proof that agents can deliver ambitious work faster.

Frontier results, on device - RL Nabors, Arize

Jun 29, 2026 · 30:52

RL Nabors (Arize) argues that most frontier-model calls can be replaced by smaller, local models, saving cost, latency, and energy. She presents a four-step framework: prototype big with a foundation model, collect a golden dataset, run capability evals using Arize's open-source Phoenix, then select the 'sage' (small and good enough) model. Demonstrating with her social app Mima, she tested Qwen 2.5, Qwen 3, LLaMA 3.2, and Gemma 4 against Claude Sonnet on summary accuracy, latency, and cost. LLaMA 3.2 (3B params) won at 90% accuracy and 1-second P50 latency, versus Gemma 4's 8 seconds. Prompt engineering—specifically few-shot prompting—closed the gap further, achieving 92.9% factual consistency and 100% JSON validity. Nabors emphasizes running regression evals to prevent regressions, and notes that on-device inference eliminates data exposure and round-trip latency.

The Prompt is the Platform - Dominik Tornow, Resonate HQ

Jun 29, 2026 · 17:33

Dominik Tornow, founder and CEO of Resonate, argues that coding agents will retire general-purpose platforms by 2026, replacing them with bespoke implementations generated on demand from reusable specifications. He details Resonate's shift from delivering implementations to offering specifications — the product becomes the protocol. Tornow recounts how agents initially failed to build a production-grade Resonate server on Postgres due to the gap between abstract specs and concrete implementations. By inserting a concrete specification and deterministic simulation environment, agents could design correct algorithms for distributed systems. Working with NATS I/O, he explains how simulation exposes hidden facts like stale reads, letting agents debug failures. The resulting workflow — abstract spec, simulated implementation, concrete spec, production code — enables agents to drive design, not just coding, making "the prompt the platform."

Research to Reality: Bringing Frontier ML Research to Production - Vaidas Razgaitis, Higharc

Jun 28, 2026 · 14:57

Vaidas Razgaitis, Senior Research Engineer at Higharc, presents three tactical focus areas to accelerate the handoff of frontier ML research into production features. First, he advocates for a Research Prototype Taxonomy Document—a technical design document tailored for ML that maps domain context, business goals, type safety, persistence, and system architecture. Second, he details Higharc's monorepo of cleanly isolated microservices with a layered API/business logic/data structure, enabling researchers to own individual services. Third, he explains using Graphite for stack diffs to decompose monolithic prototypes into small, reviewable PRs that tap subject matter experts asynchronously. The talk offers diagnostic questions to assess team velocity across these three levers.

Stop Writing Tone Instructions. Layer Them. - Isadora Martin-Dye, Isadora & Co

Jun 26, 2026 · 20:57

Isadora Martin-Dye, founder of Isadora & Co and builder of AI for a wedding venue, a personal companion, and a missing persons tool, argues brand voice requires a four-layer architecture instead of a single system prompt. The layers are immutable identity (hard rules like never claiming a body), situational mode (adjusting to user's context like grief), example-anchored voice (tone guide), and a post-generation veto (deterministic check catching errors like offering a booked date). Instructions are probabilistic; the veto is deterministic. A prompt will eventually fail; the only question is whether it's caught before reaching a customer. Examples include a wedding AI that must disclose it is AI and not say "I'd love to show you around," and a missing persons tool that must never use "confirmed" or "matched." She notes that multi-tenant voice leaks require identity to never have a default. The veto layer is the cheapest and most critical safeguard.

A Genius With Amnesia - Victor Savkin, Nx

Jun 26, 2026 · 20:00

Victor Savkin, creator of Nx and Polygraph, argues that current coding agents are like a genius with amnesia – they see only a tiny portion of the codebase and forget everything between sessions – and introduces Polygraph, a meta-harness that gives agents full organizational context and perfect memory. He explains that agents are repo-bound and lack episodic memory, forcing humans to re-explain changes (e.g., seven explanations for one UI change across four repos). Polygraph builds a unified dependency graph by analyzing thousands of repos without code changes, lets agents work across multiple repos in a single session, and captures all traces so sessions can be resumed by any agent (Claude or Codex) on any machine. It also enables context-aware queries like 'find every repo that depends on version X of this library' and allows referencing past sessions for best practices. The result is an agent that sees the entire organization's code and remembers every decision, effectively creating a hive mind.

Self Driving Products: Product Signals to Pull Requests — Joshua Snyder, PostHog

Jun 10, 2026 · 15:39

Joshua Snyder of PostHog explains how they're building a pipeline that turns product signals—errors, Slack messages, session replays—into automated pull requests. He reveals that off-the-shelf embedding models cluster signals by structural similarity, so they embed LLM-generated queries instead. He argues specificity determines whether the agent produces a useful PR, with error tracking being immediately actionable while Slack and replay usually are not. He advises starting with costly agents to discover patterns, then collapsing expensive steps into one-shot calls.

GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Jun 9, 2026 · 20:19

Audry Hsu of RunPod introduces Flash, a Python SDK that deploys GPU cloud functions from a developer's IDE with a single decorator, eliminating the slow iteration cycle of commits, Docker builds, and server allocation. She demonstrates hot reload, swapping Stable Diffusion XL Turbo for DreamShaper instantly, and a pipeline that chains Qwen 3 for prompt generation, DreamShaper for image rendering, and Nano Banana 2 for photo composition. RunPod's serverless H100 pricing is $0.00116 per second, charged only during active inference. Hsu recommends starting with pods for experimentation and switching to serverless when scaling to hundreds of workers across data centers.

Benchmarking semantic code retrieval on Claude Code — Kuba Rogut, Turbopuffer

Jun 3, 2026 · 16:08

Kuba Rogut from Turbopuffer benchmarks semantic code retrieval on Claude Code, finding that raw Claude Code wastes one in every three file reads (65% precision), windowed grep drops that to one in five, and adding semantic search cuts it to one in eight (87% precision). Semantic search wins on behavior-adjacent tasks where files share no keywords; grep wins on import tracing. Cursor's production numbers show a 24% relative improvement in answer accuracy from semantic retrieval plus a 2.6% increase in code retention in large codebases. Rogut explains Cursor's model knows when and why to call semantic search, while Claude Code treats it as just another tool, limiting gains.

How to talk to statues — Joe Reeve, ElevenLabs

Jun 1, 2026 · 33:28

Joe Reeve, an ElevenLabs growth engineer, built an app that lets users talk to statues by pointing their phone at one—it identifies the statue via OpenAI deep research, generates a matching voice with ElevenLabs' voice design API, and starts an ElevenLabs agent conversation—all in 30 seconds. He created it in two hours on a Sunday using Cursor and a single prompt, posted it on Tuesday, got 50,000 impressions, then reposted about vibe coding and hit 1.5 million. Museums, auction houses (Bonhams, Christie's), and travel platforms reached out; one CEO tracked down his WhatsApp, saying a team of 10 had worked on a similar project for a year. The episode explores voice interaction patterns, the challenge of interrupting agents, the need for multimodal interfaces (voice plus visual), and the potential of vibe coding to democratize software creation, with Joe noting that music and captions are key to viral videos.

Can LLMs generate Enterprise Quality Code? — Prasenjit Sarkar, Sonar

May 31, 2026 · 15:12

Prasenjit Sarkar from Sonar evaluates whether LLMs generate enterprise-quality code using SonarQube analysis of 4,444 Java assignments across 53 models. While Gemini 3.1 Pro High achieves 84.17% pass rate, GPT-5.4 Pro High generates 1.2 million lines (high bloat), and Claude Sonnet 4.6 has 300 security issues per million lines. The ACDC framework addresses these gaps: Guide (Sonar Sweep and Context Augmentation), Verify (SonarQube Agentic Analysis in 1–5 seconds pre-commit), and Solve (Remediation Agent that fixes issues and checks regressions before presenting fixes). Sonar's leaderboard at sonar.com/leaderboard provides detailed pass rates, cyclomatic/cognitive complexity, bug density, and security metrics per model.

The AI Skill I Rely On Daily — Priscila Andre de Oliveira, Sentry

May 27, 2026 · 17:05

Priscila Andre de Oliveira, a senior software engineer at Sentry, reveals that 67% of her AI usage is for comprehension and only 2% for code generation, based on analyzing 116 of her own Claude sessions. Working in Sentry's 15-year-old codebase with 100 PRs merged daily and 100,000 organizations depending on it, she built a personal skill called 'Catch Me Up' with six exploration modes (architecture, conventions, feature traces, syntax, testing, history). She argues that understanding what the agent found before letting it plan and implement prevents misaligned mental models that produce slop code. The episode emphasizes that in large, complex codebases, AI's biggest unlock is comprehension, not generation, and advises developers to align their mental models before prompting.

Why Rust is the Ideal Language for Vibe-Coding — Daniel Szoke, Sentry

May 27, 2026 · 16:25

Daniel Szoke, Sentry's Rust SDK maintainer, argues Rust is ideal for vibe-coding: its compiler enforces correctness, catching bugs that Python and TypeScript allow to compile and run. He challenges the preference for easy-to-write languages, warning that dynamic typing and null values invite subtle bugs. Szoke highlights Rust's type safety, null safety, and fearless concurrency, demonstrating with a 100-thread counter where TypeScript's data race compiles and fails intermittently while Rust rejects it with an error naming the non-Send type. He argues every compile error an agent fixes is a production bug avoided, and the Rust compiler is faster and more reliable than a review agent.

Does GenAI "belong" to data scientists? — Phil Hetzel, Braintrust

May 25, 2026 · 18:54

Phil Hetzel of Braintrust argues that generative AI development should not be isolated to data scientists or ML engineers, but instead requires a diverse team including product engineers, systems engineers, and non-technical domain experts. Because models are already built by OpenAI and Anthropic, the remaining work is prompt and context engineering, distributed systems, human annotation, and functional evaluation—not traditional training pipelines. Data scientists add value through rigorous testing and LLM-as-judge evaluation, but must move beyond precision/recall toward broader functional metrics. Traditional enterprises often mistakenly hand GenAI to ML platform teams, while AI natives use small cross-functional groups with closer problem proximity. The ideal mix combines technical roles for implementation and system design with non-technical experts for prompt engineering and human annotation, keeping the agent relevant through continuous feedback.

Personalization in the Era of LLMs - Shivam Verma, Spotify

May 19, 2026 · 20:12

Shivam Verma of Spotify's AI foundation explains how Spotify personalizes recommendations using open-weight LLMs, user embeddings from 750 million accounts, and Semantic IDs that compress each of 100 million tracks into six hierarchical tokens, enabling autoregressive next-item prediction. Three components: transformer-based user embeddings, Semantic IDs (e.g., Ariana Grande and Bruno Mars share first two tokens as pop artists), and a soft tokenization layer projecting user vectors into the LLM's token space for personalization without fine-tuning. The system is productionized for podcast next-episode recommendations. Verma also introduces Taste Profile, allowing users to edit their taste profile via natural language for steerable recommendations. This moves from traditional multi-stage pipelines to a unified generative model.

Beyond Code Coverage: Functionality Testing with Playwright MCP — Marlene Mhangami, Microsoft

May 16, 2026 · 19:45

Marlene Mhangami of Microsoft and GitHub argues that AI boosts developer productivity only when paired with clean code practices, citing a Stanford study of 120,000 developers and GitHub Octoverse data showing 275 million weekly commits in 2026 with a growing share co-authored by AI. She advocates test-driven development (TDD) with Playwright for functional testing over unit tests, as AI-generated tests often affirm code behavior rather than user experience. In a live demo, she uses GitHub Copilot CLI and Playwright MCP server to automatically write failing Playwright tests for a toy store's search and filter features, then generate code to pass them. She emphasizes testing one feature per test, capturing screenshots for PRs, and committing code before fixes to preserve context. She closes with advice on using Playwright agents for complex state management and confirming cross-screen-size support.

CI/CD Is Dead, Agents Need Continuous Compute and Computers — Hugo Santos and Madison Faulkner

May 13, 2026 · 18:37

Madison Faulkner and Hugo Santos (Namespace) argue that traditional CI/CD is dying because it was built for humans pushing one or two diffs a week; at agent scale, thousands of autonomous agents opening PRs cause runner saturation, cold Docker builds, cache thrash, and merge queues that behave like serialized database locks. They propose replacing PRs with intent and plan fed into an agent loop that performs fast inline validation—builds and tests in the inner loop—while humans review intent-plus-outcome instead of diffs in a premerge queue. The future they envision involves agents exploring multiple commits in parallel for the same plan (a multiverse) where the inner loop must be stateful and extremely fast to keep up with the moving tip of the repo. CI doesn't disappear but shifts into continuous enforcement of invariants and governance within the agent harness.

Feedback Loops are All You Need — Mehedi Hassan, Granola

May 10, 2026 · 10:11

Mehedi Hassan, a product engineer at Granola, argues that shipping AI features into production requires building feedback loops rather than one-shotting better prompts. Granola's chat feature for meeting notes revealed problems with web search — token costs ballooning to 10p per chat, overnight provider updates silently degrading results — and with prompt personalization, as a single prompt cannot serve salespeople, engineers, and HR managers equally. To close the gap, Granola built custom internal tracing exposing tool calls, search trails, reasoning, and cost in a UI accessible to non-engineers, not just CloudWatch queries. They also refactored their Electron app's renderer to run as a web app, enabling preview links on every PR and allowing Cursor to automatically test changes and upload screenshots. The result is faster iteration and confidence that shipped features actually work for customers.

Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops — Samuel Colvin, Pydantic

May 7, 2026 · 1:20:40

Samuel Colvin shows how to improve production agents without redeploying using Pydantic AI's GEPA, evals, and Logfire managed variables. GEPA uses a genetic algorithm to evolve prompts, achieving 96.7% accuracy — up from 87% with a simple prompt and 92% with an expert prompt — on a Wikipedia-based political relations task. Managed variables let you update prompts, models, and parameters live via a web interface and A/B test targeting. Evals compare against a golden dataset of 650 MPs; Colvin runs 65 cases in 30 seconds using GPT-4.1. He discusses that prompt optimization is most valuable with private data, that implicit user feedback (e.g., user's next action) can build golden datasets, and that overfitting to small test sets is a risk — the GEPA optimizer may exclude valid relations like 'uncle' if they don't appear in the training split.

Vibe Engineering Effect Apps — Michael Arnaldi, Effectful

May 7, 2026 · 1:43:04

Michael Arnaldi demonstrates that cloning the Effect library's repository into a project, rather than relying on prompts, is the most effective way to give coding agents the context they need to build reliably with Effect. Starting from an empty repo, he sets up a Bun, Vitest, and TypeScript project, adds the Effect repo as a git subtree, and creates an agents.md to guide the agent. He then uses the agent to research patterns, resulting in a fully functional Todo HTTP API with OpenAPI docs, SQLite persistence, and tests — all from scratch in under two hours. The workshop stresses that treating the library code as part of the project, combined with strict diagnostics and pattern files, makes agents effective even in unfamiliar codebases. Arnaldi also discusses the importance of workflow systems like Effect Cluster for long-running AI processes, where server failures become likely.

Software Engineering Is Becoming Plan and Review — Louis Knight-Webb, Vibe Kanban

May 2, 2026 · 20:23

Louis Knight-Webb, founder of Vibe Kanban, argues that software engineering is becoming all about planning and reviewing AI-generated code, not writing it. He presents two approaches: plan-heavy (spending time on specs to minimize review) and review-heavy (letting agents run and correcting output), noting five minutes of planning saves thirty minutes of review. As agents run longer—crossing the five-minute threshold—engineers must parallelize multiple agents and manage workflows, which Vibe Kanban aimed to facilitate. Knight-Webb announces shutting down Vibe Kanban despite 30,000 monthly active users because it resold tokens to enterprises and struggled to make money, advising that enterprise sales are crucial. He reflects on the journey positively, emphasizing learning to work with great people and the importance of hard work.

"Software Fundamentals Matter More Than Ever" — Matt Pocock

Apr 23, 2026 · 18:26

Matt Pocock argues that software fundamentals matter more than ever in the AI era, directly countering the 'specs-to-code' movement that treats code as cheap. He shares practical skills like 'Grill Me' to reach a shared design concept with AI, a ubiquitous language from domain-driven design to align terminology, and test-driven development to force small, deliberate steps. Drawing on John Ousterhout's 'deep modules' and Frederick Brooks's 'design concept,' he advocates structuring codebases with simple interfaces behind which AI can implement freely. He warns that bad code is the most expensive it's ever been, as it blocks the productivity gains AI can offer. His reusable skills are available in the GitHub repo MattPocock/skills.

Taste & Craft: A Conversation with Tuomas Artman, CTO Linear & Gergely Orosz, @pragmaticengineer

Apr 21, 2026 · 29:17

Tuomas Artman, CTO of Linear, warns that AI's ability to instantly ship features risks creating convoluted, low-quality software, arguing that taste and design must guide development. He explains Linear's culture of deliberate product decisions, including a 'zero bug policy' where bugs are fixed within hours, and 'Quality Wednesdays' where engineers find and fix one minute detail each week—resulting in over 2,500 quality fixes. Artman notes that AI lacks human 'taste' and cannot feel user experience nuances like animation timing. He predicts all software engineers will become product engineers, needing to focus on customer needs and UX, and advises aspiring product engineers to build for themselves, talk to customers, and study Apple's Human Interface Guidelines.

The Friction is Your Judgment — Armin Ronacher & Cristina Poncela Cubeiro, Earendil

Apr 18, 2026 · 18:38

Armin Ronacher (creator of Flask) and Cristina Poncela Cubeiro argue that while AI coding agents promise frictionless shipping, the resulting loss of human judgment introduces dangerous technical debt and brittleness. They describe the psychological trap of addictive speed and the engineering challenge of agents producing slop that creates hidden failure conditions. To counter this, they advocate for agent-legible codebases that modularize both code components and flow, enforce strict linting rules such as no bare catch-alls and unique function names, and deliberately reintroduce friction in high-stakes decisions like database migrations and permission changes. They note that agents excel at libraries with simple cores but struggle with products built on intertwined concerns, and that human responsibility must remain central as code production outpaces review capacity.

State of the Claw — Peter Steinberger

Apr 17, 2026 · 44:12

Peter Steinberger, creator of OpenClaw, presents a five-month update on the world's fastest-growing open-source project. He details the project's staggering growth—30,000 commits, nearly 2,000 contributors—and the immense security burden: 1,142 advisories (16.6 per day) with 99 critical, often AI-generated slop that demands human vetting. Steinberger refutes media fearmongering, citing how researchers ignore security docs to fabricate scary scenarios. He clarifies OpenAI did not buy OpenClaw; he joined the company while establishing the OpenClaw Foundation to remain vendor-neutral. He emphasizes the importance of local models for data sovereignty and describes his coding workflow of running 5-6 agent sessions simultaneously, iterating on taste and personality. Future visions include ubiquitous agents, 'Dreaming' for memory reconciliation, and modular plugins. For engineers, he champions taste, system design, and learning to say no.

Harness Engineering: How to Build Software When Humans Steer, Agents Execute — Ryan Lopopolo, OpenAI

Apr 17, 2026 · 46:21

Ryan Lopopolo, a member of technical staff at OpenAI, argues that software engineers must shift to 'harness engineering' where agents produce all code and humans focus on delegation, system design, and guardrails. He describes how his team banned editors, using Codex with skills to manipulate apps, and reduced review friction by automating lint rules and review agents. Lopopolo explains that code is free but human time and model context are scarce, so teams should structure repositories for consistency and use 'garbage collection days' to eliminate recurring slop. He shares practical techniques like writing tests about source code (e.g., file length limits) and using review agents that inject prompts. His future vision involves agents autonomously executing long-horizon work based on token budgets and success metrics, freeing humans for higher-level activities.

Platforms for Humans and Machines: Engineering for the Age of Agents — Juan Herreros Elorza

Apr 8, 2026 · 21:15

Juan Herreros Elorza, team lead at Banking Circle, argues that the same platform engineering best practices—self-service, API-first design, local-first workflows, and rich observability—that serve human developers are now prerequisites for AI coding agents to autonomously build, debug, and ship software. Drawing from his experience building the internal platform Atlas, which handles over €1 trillion in cross-border payments annually, he explains how exposing platform capabilities via well-defined APIs (and optionally MCP servers) lets agents iterate locally, fail fast, and close the loop using API-accessible logs and metrics. He stresses structured documentation, including agent.md files and skills, to guide agents on how to contribute to the platform, and recommends combining guardrails for security with context files for best practices. Finally, he advises measuring outcomes like DORA metrics, support requests, and developer satisfaction, and suggests using AI as leverage to finally implement these long-known best practices.

Spec-Driven Development: Agentic Coding at FAANG Scale and Quality — Al Harris, Amazon Kiro

Jan 9, 2026 · 1:03:50

Al Harris, principal engineer at Amazon, presents Spec-Driven Development (SDD) with Kiro, an agentic IDE that transforms prompts into structured requirements (EARS format), designs, and property-based tests to ensure code correctness. He argues that upfront specification—augmented by MCP servers for external context—improves reproducibility and quality over pure vibe coding. The talk includes a live demo: building an S3-backed checkpointer for a LangGraph dad joke generator, then discovering Agent Core’s native memory is more idiomatic. Harris contrasts SDD with Cursor’s planning mode, explains how specs evolve as living documentation, and discusses handling session length, context pruning, and brownfield codebases. The result is a claim that spending 5–10 minutes on specs yields higher accuracy and reliable, testable outputs.

DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Jan 8, 2026 · 1:13:13

Kevin Madura of AlixPartners argues that building robust enterprise AI applications requires shifting from brittle prompt engineering to programming with LLMs using DSPy, a declarative framework that treats prompts as implementation details optimized by the system. He demonstrates how typed interfaces (Signatures) and modular logic (Modules) allow developers to focus on control flow while deferring implementation to the LLM, with Adapters controlling prompt formats (e.g., JSON vs. BAML) to improve performance by 5-10%. The talk's core is Optimizers (like MIPRO and JEPA), which automatically tune prompts by learning from data, shown improving a time entry corrector from 86% to 89% accuracy. Real-world examples include routing files by type (SEC filings vs. contracts), using a 'poor man's RAG' with attachments for multimodal documents, and a boundary detector that segments legal documents from images. Madura emphasizes that DSPy enables transferability across models (e.g., GPT-4.1 to GPT-4.1 Nano) and addresses cost concerns by allowing offline optimization to reduce LLM calls.

Build a Prompt Learning Loop - SallyAnn DeLucia & Fuad Ali, Arize

Jan 6, 2026 · 52:08

SallyAnn DeLucia and Fuad Ali from Arize present prompt learning, a method to continuously improve AI prompts by using feedback from evals and human annotations. They argue that agent failures often stem from weak instructions rather than weak models, and that adding rules to system prompts can yield 15% improvements on SWE-Bench without fine-tuning or architecture changes. The talk compares prompt learning to GEBA, noting it achieves better results in fewer loops by leveraging text explanations. A case study shows Kline's performance improved from 30% to 45% on SWE-Bench-Lite after optimizing prompts. The workshop demonstrates building an optimization loop that ingests input-output pairs, evaluates them, and iteratively refines the system prompt using OpenAI.

Welcome to AIE CODE - Jed Borovik, Google DeepMind

Jan 5, 2026 · 4:10

Jed Borovik, Google DeepMind's Gemini assistant and lead of the Jewels coding agent product engineering team, opens the 2025 AI Engineering Code Summit in New York by declaring that the most important problem in applied AI is code. He frames the event as a single-track summit designed to push the entire AI coding industry forward, not any single company. Borovik highlights the previous leadership track, which covered how AI transforms software organizations, and sets the day's focus on patterns, systems, and products enabling that transformation. He thanks DeepMind as presenting sponsor, noting the timely release of Gemini 3 and Nano Banana Pro, along with Anthropic as platinum sponsor and other gold and silver sponsors. Borovik encourages attendees to visit sponsor booths in the expo area and expresses excitement for the keynotes to come.

AGI: The Path Forward – Jason Warner & Eiso Kant, Poolside

Dec 27, 2025 · 15:56

Jason Warner and Eiso Kant, co-founders of Poolside, present their vision and roadmap towards AGI-level capabilities for knowledge work, demonstrating their second-generation model Malibu Agent converting ADA code to Rust live on stage. They argue that next-token prediction paired with reinforcement learning is the key breakthrough, a contrarian bet they made two and a half years ago. The episode centers on their work in high-consequence code environments for defense and government, where agents must operate with tight permissions. They announce a large compute cluster of over 40,000 GB300s coming online and a public API release early next year via AWS Bedrock. Warner recounts meeting Kant through a failed GitHub acquisition, and Kant invites the audience to build with their models, emphasizing that future agents will handle tasks over days as intelligence scales.

Developer Experience in the Age of AI Coding Agents – Max Kanat-Alexander, Capital One

Dec 23, 2025 · 18:20

Max Kanat-Alexander, Capital One executive distinguished engineer, argues that developer experience teams should invest in a 'No Regrets' framework to prepare for an agentic future, regardless of unpredictable AI advancements. He identifies six key inputs that benefit both humans and AI agents: standardized development environments using industry-standard tools, native CLIs and APIs instead of browser automation, deterministic validation with clear error messages, refactoring legacy codebases for testability and reasoning, documenting external context and intentions, and improving code review velocity and quality through assignment and apprenticeship. Without these, bad codebases lead to rubber-stamped PRs and a vicious cycle of decreasing agent productivity; with them, a virtuous cycle accelerates software engineering velocity. Kanat-Alexander emphasizes that what's good for humans is good for AI, and these investments will help developers no matter what the future holds.

The Infinite Software Crisis – Jake Nations, Netflix

Dec 20, 2025 · 18:57

Jake Nations, engineering lead at Netflix, argues that AI-generated code accelerates the software crisis by conflating easy with simple—producing tangled, incomprehensible systems. He traces the crisis from 1968 to today's infinite code generation, citing Fred Brooks' 'No Silver Bullet' and Rich Hickey's definition of simple as 'one fold, no entanglement.' Nations presents a three-phase methodology—research (compressing 5 million tokens of code into a 2,000-word spec), planning (paint-by-numbers implementation steps), and implementation (using a manual migration seed)—to maintain human understanding. He warns that without this approach, engineers lose the ability to recognize dangerous complexity, and challenges listeners: will we still understand our own systems when AI writes most of the code?

From Arc to Dia: Lessons learned building AI Browsers – Samir Mody, The Browser Company of New York

Dec 19, 2025 · 17:48

Samir Mody, Head of AI Engineering at The Browser Company, explains how the team built Dia, an AI-native browser, after learning from their earlier browser Arc. He details three core lessons: optimizing tools for faster iteration by embedding prompt editors into the product itself; treating model behavior as a craft through a dedicated team formed after a strategy and ops employee rewrote all prompts in a weekend; and designing AI security as an emergent property, using confirmation steps for features like autofill to mitigate prompt injections despite the 'lethal trifecta' of private data, untrusted content, and external communication. The talk emphasizes that building an AI product required not just a technology shift, but a company-wide transformation in hiring, training, and collaboration.

Paying Engineers like Salespeople – Arman Hezarkhani, Tenex

Dec 19, 2025 · 14:53

Arman Hezarkhani, CTO of Tenex, describes his company's outcome-based compensation model where engineers are paid per story point completed, tying incentives directly to shipped value. He argues that hourly billing and salary with equity fail to motivate engineers to use AI tools effectively, leading to misaligned incentives. Tenex's system involves strategists scoping tickets and engineers receiving a flat base plus quarterly story-point bonuses. Risks like inflated story points, quality drops, and sharp elbows are mitigated by strategist counterbalance, multi-round QA, and a rigorous hiring process. He cites examples: a billboard company's image moderation AI built in two weeks at 96% accuracy, and a retail heat-mapping project with five parallel models. Arman asserts that AI amplifies existing talent, and current compensation must change to unlock potential.

Dispatch from the Future: building an AI-native Company – Dan Shipper, Every, AI & I

Dec 18, 2025 · 17:58

In this episode, Dan Shipper of Every argues that there is a 10x difference between an organization where 90% of engineers use AI and one where 100% do. At Every, 99% of code is written by AI agents, enabling 15 people to run four software products with 7,000 paying subscribers, each built by a single developer. Shipper introduces 'Compounding Engineering', where each feature makes the next easier to build through a loop of plan, delegate, assess, and codify. He describes how this approach allows managers to commit code, enables tacit code sharing across products, and lets new hires be productive on day one. The episode details how Every's shift to agentic workflows (Claude Code, Codex) and a 'demo culture' over 'memo culture' has transformed their engineering velocity and collaboration.

AI Kernel Generation: What's working, what's not, what's next – Natalie Serrino, Gimlet Labs

Dec 17, 2025 · 19:15

Natalie Serrino, cofounder of Gimlet Labs, presents how AI-generated kernels can automatically speed up custom PyTorch code by up to 24% on Apple M4 hardware using the Metal framework, with a 40% speedup from kernel fusion. The agentic system iterates through compilation, execution, correctness, and optimization, but faces challenges like validation of floating-point results and reliable benchmarking. Successes include rewriting average pool 1D as a convolution for 80% improvement, while failures occur on heavily optimized ops like matrix multiply. A real-world audio encoder model saw 70% faster inference on RTX 6000 Blackwell via six custom fused kernels. Serrino emphasizes that AI is best for rapidly searching optimizations and porting code to new hardware, not for surpassing human experts on novel algorithms.

Building in the Gemini Era – Kat Kampf & Ammaar Reshi, Google DeepMind

Dec 15, 2025 · 17:57

Kat Kampf and Ammaar Reshi from Google DeepMind present Gemini 3 and Nano Banana Pro, arguing that these models enable anyone to build complex, aesthetic applications through natural language alone. Gemini 3 achieves state-of-the-art results in one-shot UI design and agentic tool calling, with SweBench outperformance. Nano Banana Pro integrates Google Search for world knowledge and renders text accurately, handling up to 14 consistent people per image. They demonstrate vibe coding in AI Studio, building a personalized comic book with precise text, laptop stickers grounded in search, and a 3D racing game that scaled to 23 players live. Upcoming full-stack runtime adds backend support and automatic database integration, further democratizing software creation.

From Vibe Coding To Vibe Engineering – Kitze, Sizzy

Dec 14, 2025 · 25:28

Kitze, founder of Sizzy, argues that "vibe engineering"—actively steering AI agents with technical knowledge—trumps passive "vibe coding" for building real software. He shares how Cursor's Composer 1 let him port Benji.so and Glink to Next.js 16 with Monorepo in under a week, reviving near-dead projects. Kitze insists LLMs excel at React because humans are bad at it, and that repetitive code is fine since agents don't care. He warns against giving AI tools to juniors without oversight, and predicts the bottom of the job market will thin as agents replace interns. The talk ends with a pitch for "vibe code fixers" as a new role, maintaining legacy systems like cowboy coders.

Moving away from Agile: What's Next – Martin Harrysson & Natasha Maniar, McKinsey & Company

Dec 12, 2025 · 21:55

McKinsey's Martin Harrysson and Natasha Maniar argue that enterprises must overhaul their people and operating models—moving beyond Agile to AI-native workflows—to capture the full potential of AI in software development. They identify bottlenecks like work allocation, manual code review, and tech debt that limit gains to 5-15% despite promising individual productivity stories. To scale, they advocate AI-native workflows (spec-driven development, continuous planning) and AI-native roles (smaller pods of 3-5 with consolidated roles). In a client study with a bank, interventions led to 60x increase in agent consumption, 51% increase in code mergers, and faster delivery tied to business priorities. They emphasize change management: 70% of companies haven't changed roles, and top performers invest in hands-on upskilling, measurement systems that track outcomes beyond adoption, and incentives to drive adoption.

The State of AI Code Quality: Hype vs Reality — Itamar Friedman, Qodo

Dec 11, 2025 · 21:15

Itamar Friedman, CEO of Qodo, argues that while AI code generation boosts productivity, it has a glass ceiling unless organizations invest in agentic quality workflows and context. Drawing on reports from Qodo, Sonar, and Pharos, he reveals that 60% of developers say a quarter of their code is AI-generated, yet 67% have serious quality concerns, leading to 42% more time fixing bugs and 35% project delays. He notes that AI code review tools double trust and quality gains, and that better context—including standards and best practices—is the top request from developers. Friedman recommends automated quality gateways, intelligent code review, and testing, warning that without dynamic quality processes, the promised 2x productivity remains elusive.

Can you prove AI ROI in Software Eng? (Stanford 120k Devs Study) – Yegor Denisov-Blanch, Stanford

Dec 11, 2025 · 16:40

Stanford researcher Yegor Denisov-Blanch presents research based on 120,000+ developers across 600+ companies, arguing that AI ROI in software engineering is often negative despite apparent productivity gains. The study finds a median 10% productivity lift from AI, but a widening gap between top and bottom performers. Key drivers include codebase hygiene (environment cleanliness index with R² 0.40) and AI usage quality over volume. A company case study shows pull requests increased 14% but code quality dropped 9% and rework rose 2.5x, yielding no net effective output gain. Denisov-Blanch proposes a measurement framework using a primary metric (engineering output via ML model) paired with guardrail metrics, and emphasizes that companies can retroactively measure impact via git history without waiting for experiments.

2026: The Year The IDE Died — Steve Yegge & Gene Kim, Authors, Vibe Coding

Dec 6, 2025 · 24:59

Steve Yegge and Gene Kim argue that by 2026, traditional IDEs will be obsolete, replaced by AI-driven coding agents. Yegge claims current tools like Claude Code are too hard for most developers, with high cognitive overhead and a single-agent bottleneck akin to a solo diver running out of oxygen. He predicts next year will bring multi-agent systems, comparing the shift from manual tools to CNC machines, and provocatively states that engineers using an IDE after January 1st are bad engineers. Gene Kim introduces their 'Vibe Coding' book and shares case studies: Booking.com saw double-digit productivity gains, Travelopia replaced a legacy app in six weeks with a small team, and a Fidelity leader Vibe Coded an app in five days that previously was estimated to take five months. They present the FAFO framework (Faster, Ambitious, alone/autonomous, Fun, Optionality) and cite DORA research showing trust in AI grows with usage. The episode concludes that Vibe Coding will reshape organizations, enabling non-developers to ship features and potentially compressing team sizes.

Future-Proof Coding Agents – Bill Chen & Brian Fioca, OpenAI

Dec 5, 2025 · 17:48

Bill Chen and Brian Fioca from OpenAI argue that coding agents are booming but fragile, with teams rebuilding harnesses every time models change. They present Codex, OpenAI's combined model and harness, as a stable abstraction layer that handles complexity like context compaction, parallel tool calls, and MCP integration. Key lessons include avoiding overprompting—letting the model use its trained habits rather than forcing foreign instructions—as demonstrated when GPT-5 ran slowly because users told it to examine every file. They describe patterns for using Codex as an SDK, enabling products like GitHub and Cursor to integrate without maintaining their own harness, and highlight future improvements in longer-horizon tasks and self-healing software. The episode concludes that building where models are going, with Codex's SDK, lets teams focus on differentiation rather than infrastructure rewrites.

Backlog.md: Terminal Kanban Board for Managing Tasks with AI Agents — Alex Gavrilescu, Funstage

Nov 24, 2025 · 14:19

Alex Gavrilescu presents Backlog.md, an open-source CLI tool that stores tasks as Markdown files in Git repos, featuring a terminal Kanban board and MCP server for AI agents. He argues that breaking features into atomic Markdown tasks prevents agents from running out of context or implementing unwanted extras. The demo shows Claude creating a task from requirements, generating an implementation plan, and coding a move-mode feature—all via MCP tools. Gavrilescu emphasizes two review checkpoints (after task creation and after the plan) and notes that Backlog.md itself is 99% AI-written. The tool works cross-platform, requires no external APIs, and syncs across branches via Git.

Vision: Zero Bugs — Johann Schleier-Smith, Temporal

Nov 24, 2025 · 36:03

Johann Schleier-Smith, Technical Lead for AI at Temporal, argues that zero-bug software is achievable by applying aerospace-grade techniques like rigorous specification, modular design, and formal verification, now made practical by AI. He cites the Airbus A320's control software (no incidents from software in decades) and the Space Shuttle (1 error per 420,000 lines) as proof. Schleier-Smith demonstrates Dafne, a language that embeds proofs inline with code for automated verification, and shows how agentic coding can reduce high-assurance code costs by ~100× compared to traditional development. He concludes that agents generating verified code will overcome current quality limitations and trigger widespread adoption.

Infra that fixes itself, thanks to coding agents — Mahmoud Abdelwahab, Railway

Nov 24, 2025 · 18:08

Mahmoud Abdelwahab from Railway presents Railway Autofix, a system that uses Inngest for durable execution and OpenCode as a coding agent to automatically detect infrastructure issues and open pull requests to fix them. The system runs scheduled workflows to fetch project architecture, resource metrics, and HTTP performance data, then analyzes services exceeding thresholds. For affected services, it gathers additional context like logs and generates a detailed fix plan using an LLM. The coding agent then implements the plan and opens a PR with a summary of changes. The talk demonstrates how this shifts developers from manually investigating alerts to simply reviewing auto-generated fixes.

The Cure for the Vibe Coding Hangover — Corey J. Gallon, Rexmore

Nov 24, 2025 · 57:02

Corey J. Gallon presents a framework to cure the 'vibe coding hangover'—the despair of building unmaintainable demoware with AI coding agents. He argues that treating AI as a pure productivity tool leads to brittle code and dependency, so his framework centers on ten principles: AI engineering as accelerated learning, the architect-implementer boundary, slowing down to go fast, specification over prompt engineering, defining 'done' before implementing, feature atomicity, dependency-driven development, implementing one atomic feature at a time, context engineering, and 'make it work, make it right, make it fast.' The process splits into a planning phase (vision capture, feature identification, iterative specification, dependency analysis, implementation plan) and an implementation loop that uses multisensory feedback (visual, auditory, tactile) to validate each atomic feature. This framework has been used to build real applications such as litigation support for law firms and real-time appliance monitoring.

What Data from 20m Pull Requests Reveal About AI Transformation — Nick Arcolano, Jellyfish

Nov 24, 2025 · 17:57

Nick Arcolano, head of research at Jellyfish, presents data from 20 million pull requests across 200,000 engineers at 1,000 companies to reveal what AI transformation actually looks like. Median AI adoption of interactive coding tools (Copilot, Cursor, Claude Code) jumped from 22% in June 2024 to nearly 90% today, driving a 2x increase in PR throughput and 24% faster cycle times on average. Autonomous agents like Devin and Codex remain negligible—only 44% of companies have tried them, contributing less than 0.2% of PRs. PRs grow 18% larger with AI, but bug rates and reverts show no significant quality impact. Crucially, code architecture determines success: centralized and balanced repos see 4x throughput gains, while highly distributed architectures (many repos per engineer) show no correlation with AI adoption, likely due to cross-repo context limitations. Arcolano advises focusing on adoption first, then addressing architectural barriers to unlock productivity.

Z.ai GLM 4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

Nov 22, 2025 · 19:39

Yuxuan Zhang from zAI details the technical roadmap behind the GLM 4.6 open-source model series, which has surpassed 100 million downloads and tied for number one on the LMSYS Chatbot Arena alongside GPT-4o and Claude 3.5 Sonnet. The training pipeline uses 15 trillion tokens of pre-training data, followed by 7 trillion tokens of code and reasoning data, repo-level contexts at 32,000 tokens, and 100 billion tokens of long-context agent trajectories up to 128,000 tokens. Zhang introduces SLIME, a hybrid synchronous/asynchronous RL framework that decouples agent-environment interaction from GPU training to avoid bottlenecks. He explains why single-stage RL at 64,000 tokens outperforms multi-stage approaches for preserving long-context abilities, and shows that token-weighted loss converges faster than sequence-average loss for code RL. The multimodal GLM 4.5V model handles native resolution images and video with temporal index tokens, enabling GUI agent capabilities. Deployment is supported via vLLM and SGLang, with an API at z.ai and an open-source coding assistant.

How BlackRock Builds Custom Knowledge Apps at Scale — Vaibhav Page & Infant Vasanth, BlackRock

Aug 23, 2025 · 18:47

Infant Vasanth and Vaibhav Page of BlackRock explain how the world's largest asset manager scales custom AI knowledge apps for investment operations using a modular, Kubernetes-native framework. They detail challenges like prompt engineering for complex financial instruments, choosing LLM strategies (RAG, chain-of-thought), and deploying apps with cost control and compliance. Their sandbox lets domain experts rapidly iterate on extraction templates with inter-field dependencies and validations, while the app factory turns definitions into production apps. Key takeaways: invest in prompt engineering for domain experts, educate on LLM strategies, evaluate ROI, and design for human-in-the-loop in regulated environments. The framework compressed app development from eight months to days for use cases like setting up securities after IPOs or stock splits.

#define AI Engineer - Greg Brockman, OpenAI (ft. Jensen Huang)

Aug 10, 2025 · 41:05

Greg Brockman, co-founder and president of OpenAI, discusses his journey from Stripe's first CTO (growing from 40 to 250 people) to leading AI development, sharing insights on independent study (completing three years of high school math in one year), the engineering-research partnership at OpenAI (emphasizing technical humility), and scaling challenges like ChatGPT hitting 1 million users in five days and ImageGen reaching 100 million in five days. He explains how Codex transforms coding (low double-digit percent of internal PRs written by Codex, 24,000 PRs merged in public GitHub in one day) and introduces "vibe coding" as an empowerment mechanism. In response to Jensen Huang's questions, Brockman predicts a future with diverse AI infrastructure (homogeneous accelerators as default, purpose-built for specific workloads) and a menagerie of domain-specific agents leveraging distillation, driving 10x more economic activity. He identifies basic research as a renewed scaling bottleneck alongside compute, data, algorithms, power, and money.

On Engineering AI Systems that Endure The Bitter Lesson - Omar Khattab, DSPy & Databricks

Aug 6, 2025 · 19:12

Omar Khattab, research scientist at Databricks and creator of DSPy, argues that engineering AI systems that endure the rapid pace of model and technique changes requires decoupling high-level task definitions from low-level, swappable components. He reinterprets the Bitter Lesson—that scaling search and learning beats hand-coded domain knowledge—as a warning against premature optimization, which he equates to hard-coding at lower abstraction levels than justified. Khattab criticizes prompts as a poor programming abstraction that entangles task specs, formatting, and model-specific tricks, violating separation of concerns. Instead, he advocates for investing in three orthogonal concerns: natural language specs for things that can't be otherwise expressed, evals to capture true objectives, and code for reliable structure and control flow. DSPy realizes this philosophy through declarative signatures that define what an AI system should do, independent of which LLM, inference strategy (e.g., chain-of-thought, agent), or optimizer (e.g., reinforcement learning, prompt optimization) is used. The takeaway: avoid hand engineering at lower levels than current abstractions allow, and ride…

Vibe Coding with Confidence — Itamar Friedman, Qodo

Aug 6, 2025 · 20:55

Itamar Friedman, CEO of Qodo, argues that the command-line interface (CLI) is the future for achieving 'vibe coding with confidence' in enterprise software development. He contrasts vibe coding's greenfield success with the challenges of large codebases, where context, workflows, and multi-agent collaboration are essential. Friedman explains Qodo's multi-agent architecture, including tools for code review (Qodo Merge) and coverage (Qodo Cover), and announces the soft launch of a new CLI tool that lets developers create and pipe agents, supporting MCP and A2A protocols. He envisions a future 'swarm of agents' each specialized with credentials and best practices, enabling end-to-end workflows across the SDLC. The episode emphasizes moving from chat-based agents to command-driven multi-agent systems for reliable, high-quality code generation.

How to Improve your Vibe Coding — Ian Butler

Aug 3, 2025 · 7:30

Ian Butler (CEO of BizMyth) presents benchmark data showing that popular coding agents like Devin and Cursor have less than 10% true positive rates for finding bugs, with Cursor producing a 97% false positive rate across 100+ repos and 1,200+ issues. He argues that these agents overrun codebases with false positives, causing alert fatigue and bugs reaching production. To improve vibe coding, he recommends adding bug-focused rules (e.g., OWASP Top 10, explicit bug classes like auth bypasses), managing context by feeding diffs and preventing file summarization, and using thinking models which were significantly better at finding deeper bugs. However, even thinking models show high variability in which bugs they find across runs, indicating agents never holistically examine a file. The talk concludes with a plug for BizMyth.sh, which automates PRs and vulnerability scanning.

Vibes won't cut it — Chris Kelly, Augment Code

Aug 3, 2025 · 15:34

Chris Kelly of Augment Code argues that vibe coding—letting AI write code without review—fails for production systems, where every line of code carries maintenance burden and decisions about architecture, packages, and trade-offs. He contends that code generation is not the job; software engineering is changing code safely, requiring context that LLMs lack despite their human-like output. Claims that generating 30% of code against millions of existing lines leaves little wiggle room, and that complex systems have emergent failures that pattern matching can't fix. He advises documented standards, reproducible environments, easy testing, clear boundaries, and defined tasks—tools every engineer needs. Cites Jeff Atwood: 'The best code is no code at all.' Notes professional engineers are slowest to adopt AI, and suggests code review is the most important skill for an AI-assisted future.

Real World Development with GitHub Copilot and VS Code — Harald Kirschner, Christopher Harrison

Aug 3, 2025 · 1:19:33

Harald Kirschner demonstrates three stages of Vibe Coding with GitHub Copilot and VS Code—YOLO, Structured, and Spectrum—arguing that developers can move from rapid prototyping to maintainable, scalable AI-assisted development. He shows how to use Agent mode, auto-approve settings, and new workspace scaffolding to build a hydration tracking app with React Vite and Material Design without looking at code. The talk covers custom instructions, MCP servers (Playwright, Jest Pad, Perplexity), and custom modes like TDD that enforce test-driven development by writing failing tests first. Kirschner explains how to constrain tools for deterministic behavior using tool sets and recommends committing often, pausing AI when needed, and iterating on instructions. He also discusses spec-driven development, critiquing specs with AI, and the importance of well-structured codebases for AI productivity.

Shipping Products When You Don't Know What they Can Do — Ben Stein, Teammates

Jul 28, 2025 · 19:34

Ben Stein, founder of Teammates, argues that product management for AI agents requires abandoning traditional specs and embracing uncertainty. He describes how his team discovered emergent behavior when a customer asked to tag an agent in a Google Doc comment—an unplanned feature that worked. Stein advocates thinking in affordances rather than features, using evals as the new spec, and vibe coding to feel interactions before committing. He notes that bugs become ambiguous when behavior is probabilistic, and customer meetings shift from selling vision to co-inventing the future. The episode offers a field guide for product teams navigating boundless surface areas built on unpredictable LLMs.

Building the platform for agent coordination — Tom Moor, Linear

Jul 28, 2025 · 19:43

Tom Moor, Head of Engineering at Linear, explains how the company is evolving from an issue tracker into an operating system for engineering teams that treats AI agents as first-class teammates. He details Linear's pragmatic AI journey: starting with embeddings and PG vector, then moving to a hybrid search index using TurboPuffer and Cohere embeddings, leading to features like Product Intelligence (query rewriting, reranking, deterministic rules for suggestions), natural language filters, Slack-to-issue creation, and daily audio pulses. The core of the talk is Linear's agent platform, launched two weeks ago, where coding agents like CodeGen, Bucket, and Charlie integrate via OAuth, GraphQL, and new webhooks, allowing users to assign, mention, and interact with agents just like human teammates. Moor emphasizes best practices for builders: respond fast, inhabit the platform's language, move issues to 'in progress,' and clarify plans before acting, all while keeping interactions concise and value-adding. The episode argues that with this platform, engineering teams can build more, faster, and with higher quality by offloading grunt work to infinitely scalable cloud-based teammates.

Ship Production Software in Minutes, Not Months — Eno Reyes, Factory

Jul 25, 2025 · 16:06

Eno Reyes, cofounder and CTO of Factory, argues that AI agents can orchestrate the entire software development lifecycle, moving beyond vibe coding to agent-native development where enterprises delegate planning, coding, testing, and incident response to autonomous droids. He explains that AI tools are only as good as the context they receive—missing context from meetings, whiteboards, or Slack is the primary cause of failure, not LLM quality. Factory's droids search codebases, leverage organizational memory, and question unclear tasks before executing, from generating PRDs and tickets to creating runbooks and RCAs from sentry alerts. Reyes demonstrates how an agent can convert user transcripts and ad-hoc notes into a full feature plan, then break it into parallel tickets for multiple code droids. For incident response, droids pull logs, historical runbooks, and team discussions to produce mitigation plans in minutes, cutting response times in half and shifting from reactive to predictive operations. He emphasizes that the future belongs to engineers who manage agents—thinking clearly and communicating effectively—rather than those writing every line of code.

Software Development Agents: What Works and What Doesn't - Robert Brennan, OpenHands

Jul 25, 2025 · 16:46

Robert Brennan, creator of the open-source coding agent OpenHands (formerly OpenDevin), argues that while coding is going away, software engineering remains about critical thinking, not typing. He explains that agents like OpenHands operate via a loop between an LLM and tools like a diff-based editor, terminal, and web browser, all sandboxed in Docker. Brennan advises starting small with rote tasks like merge conflicts (99% success) or PR feedback, being specific about frameworks and files, and always reviewing AI-generated code to avoid tech debt. He shares that 90% of his code now goes through the agent, with 10% requiring manual intervention, and warns against auto-merging without human accountability—OpenHands now assigns PRs to the human who triggered them. Key use cases include fixing failing tests, infrastructure changes, database migrations, and greenfield internal apps where vibecoding is acceptable.

Devin 2.0 and the Future of SWE - Scott Wu, Cognition

Jul 25, 2025 · 16:13

Scott Wu of Cognition AI describes an AI 'Moore's Law' for coding agents, with capability doubling every 70 days, propelling Devin from simple migrations to autonomous project work. He traces Devin's evolution through tiers: first repetitive migrations using Playbooks for instruction-following, then isolated bugs with VM snapshots, then multi-file changes with codebase intelligence, and finally backlog killing with self-testing. Each tier required solving different bottlenecks—memory, language server integration, iterative planning, and confidence calibration. Wu argues that the interface and key capabilities shift every 2-3 months, and predicts another 16-64x improvement in the next year.

Does AI Actually Boost Developer Productivity? (100k Devs Study) - Yegor Denisov-Blanch, Stanford

Jul 23, 2025 · 18:12

Yegor Denisov-Blanch from Stanford presents findings from a study of nearly 100,000 developers across 600+ companies, showing AI boosts developer productivity by an average of 15-20%, but the effect varies widely. The study measures functionality delivered, not commits, and reveals that AI introduces significant rework—bug fixes from AI-generated code. Productivity gains depend on task complexity, codebase maturity (Greenfield vs. Brownfield), language popularity, and codebase size: low-complexity Greenfield tasks see 30-40% gains, while high-complexity Brownfield tasks see only 0-10%. AI can even decrease productivity for low-popularity languages like COBOL or Haskell, and context window limitations reduce performance on larger codebases. The study provides data-driven guidance for when AI helps or hinders, with a matrix showing 20% gains for low-complexity common languages and minimal gains for complex niche ones.

POC to PROD: Hard Lessons from 200+ Enterprise GenAI Deployments - Randall Hunt, Caylent

Jul 23, 2025 · 19:16

Randall Hunt from Caylent shares hard lessons from over 200 enterprise GenAI deployments, arguing that evals, embeddings, and prompt engineering matter far more than fine-tuning. He emphasizes that speed and UX are critical; a slow inference kills adoption, while techniques like generative UI and caching can mitigate latency. Hunt details real-world examples: using audio amplitude spectrographs for sports highlight reels, pooling multimodal embeddings for nature footage search, and noting that nurses prefer chat over voice bots in noisy hospitals. He reports zero regressions moving from Claude 3.7 to 4, and advises optimizing context and economics, such as leveraging Amazon Bedrock batch for 50% cost reduction. The talk underscores that knowing your end customer and minimizing irrelevant context are key to production success.

Books reimagined: AI to create new experiences for things you know — Lukasz Gandecki, TheBrain.pro

Jul 22, 2025 · 9:44

Łukasz Gandecki of TheBrain.pro presents Books Reimagined, an AI-powered reading companion that adds music, graphics, and conversational AI to books like '1984' and 'The Snow Queen', arguing that AI can create magical experiences by hiding itself behind a polished user interface. He started by vibe-coding a companion for a Trump reelection book to track characters, then iterated into a full multimedia reader. He demonstrates features including 100-millisecond voice response, natural language search via embeddings, and deep research that scans the entire book. Gandecki emphasizes that human curation of AI-generated music and graphics is essential for quality, and he is open-sourcing the player at bookgenius.net.

AI powered entomology: Lessons from millions of AI code reviews — Tomas Reimers, Graphite

Jul 22, 2025 · 10:21

Tomas Reimers, co-founder of Graphite, discusses the company's AI-powered code reviewer Diamond, arguing that while LLMs can effectively find bugs, they must be carefully prompted to avoid frustrating developers. After analyzing 10,000 comments from codebases, Graphite classified bugs into a quadrant based on what LLMs can catch versus what humans want to receive, identifying bugs, accidentally committed code, performance and security concerns, and documentation mismatches as high-value targets. Code cleanliness and best practice comments, though technically correct, are often unwelcome from AI. Graphite measures success via emoji reactions (less than 4% downvote rate) and the percentage of comments that lead to code changes, reaching 52% actionability in March—matching human-level effectiveness. The talk emphasizes continuous monitoring to ensure LLM comments stay within the desirable quadrant.

Top Ten Challenges to Reach AGI — Stephen Chin, Andreas Kollegger

Jul 22, 2025 · 4:23

Stephen Chin and Andreas Kollegger, curators of the GraphRAG track at AI Engineer World's Fair, use ten sci-fi memes to expose top challenges to AGI—from Memento's short-term memory as prompt engineering to Skynet's unintended consequences, the Matrix's agent simulation, Hal's trust issues, emotions as bug/feature, Frankenstein's creator obligations, Terminator's time travel, Star Wars' language nuance, the Borg's assimilation, and Deep Thought's right questions. They argue graphs and graph technology can solve some of these challenges, inviting the audience to the GraphRAG track to learn which ones. The episode delivers a playful yet pointed framework connecting pop culture to core AGI obstacles.

tldraw.computer - Steve Ruiz, tldraw

Jul 21, 2025 · 18:45

Steve Ruiz, founder and CEO of tldraw, demonstrates the company's AI experiments on their infinite canvas, including Make Real—which turns hand-drawn wireframes into working web apps using vision models—and tldraw computer, a visual programming environment where arrows and LLMs power a graph of connected nodes that can execute multi-step prompts, generate images and speech, and even run loops indefinitely. He also shows Draw Fast for real-time image generation and Teach, where Claude can draw and edit shapes on the canvas. The episode explains how tldraw's SDK (tldraw.dev) enables others to build custom canvas applications, and highlights the company's philosophy of 'shitty but amazing' rapid prototyping.

Design like Karpathy is watching — Zeke Sikelianos, Replicate

Jul 19, 2025 · 19:26

Zeke Sikelianos of Replicate analyzes Andrej Karpathy's experience building and deploying MenuGen, a vibe-coded web app that turns menu photos into images, to argue that LLMs are now the primary audience for developer tools. He details Replicate's specific failures—rate limiting and outdated docs that blocked Karpathy—and the fixes they implemented: adding LLMs.txt files for Markdown-friendly documentation, promoting curl commands as the LLM-friendly interface, and launching an MCP server built on OpenAPI schemas. Sikelianos advocates for 'boring technology' like SQL and 'good API hygiene' to keep payloads small and information-dense for LLM context windows. He also calls for better payment acceptance and documentation discipline, stressing that unblocking power users like Karpathy (whose CEO intervened) should not require a viral blog post.

On Curiosity — Sharif Shameem, Lexica

Jul 19, 2025 · 18:35

Sharif Shameem, founder of Lexica, argues that curiosity is the main force for pulling ideas from the future into the present, and that building and sharing demos is the best way to explore AI models' hidden capabilities. He recounts early GPT-3 demos from 2020-2021, when the model had a 2,000-token context window and cost $75 per million output tokens, showing how he built a JSX compiler in the browser, a shopping agent that parsed web pages, and a multi-step reasoning tool called MultiVAC. Shameem emphasizes that AI engineering is more like excavating than traditional engineering, and that researchers often don't know the full capabilities of their own models. He closes by invoking computing pioneer J.C.R. Licklider, arguing that today's AI engineers have a moral obligation to follow their curiosity and share what they discover.

RL for Autonomous Coding — Aakanksha Chowdhery, Reflection.ai

Jul 16, 2025 · 19:27

Aakanksha Chowdhery, CEO of Reflection AI and former lead researcher on PaLM and Gemini at Google, argues that reinforcement learning (RL) scaling is the next frontier for autonomous coding agents. She explains that inference-time techniques like majority voting and self-revision improve accuracy but require many samples (e.g., 10,000 for rare correct generations). RL training can learn to generate correct outputs directly, especially in verifiable domains like code with unit tests. Reflection AI aims to build superintelligence starting with autonomous coding, leveraging automated verification to design better reward functions. She notes challenges in scaling RL, including system complexity and reward hacking, but sees coding as ideal due to execution feedback.

Building a 10 person unicorn - Max Brodeur-Urbas, Gumloop

Jul 15, 2025 · 12:03

Max Brodeur-Urbas, founder of Gumloop, explains how his company scaled to millions in ARR as a team of two that raised a Series A and grew to only nine people, by being super picky in hiring, using product-led hiring where customers like those from Instacart and Webflow join the team, and requiring work trials such as hacking together in Airbnbs. They eliminate almost all meetings to give engineers deep focus time and automate every internal process with Gumloop itself, from customer research reports to chatbot monitoring. Culture-wise, they balance intense 45-minute shipping challenges with fun retreats and a public company handbook. Brodeur-Urbas argues that every hire must be a no-brainer, and small teams can outpace larger ones by avoiding meetings and leveraging AI tools.

Using OSS models to build AI apps with millions of users — Hassan El Mghari

Jul 15, 2025 · 18:47

Hassan El Mghari, a software engineer at Together.ai, explains how he builds open source AI apps that attract millions of users, including roomGPT.io (2.9 million users), restorePhotos.io (1.1 million), Blinkshot.io (1 million visitors), and LlamaCoder.io (1.4 million visitors). He details his simple four-step tech stack: user input, a single API call to an open source model on Together.ai, storing results in a database, and displaying output. His process emphasizes keeping ideas simple enough to describe in five words, spending 80% of time on UI, launching early, and incorporating the latest models like Fluxionel for virality. He advises naming apps with short, memorable names, making them free and open source to encourage sharing, and iterating rapidly—most apps fail, but the key is to keep building. He funds compute through sponsorships from Together.ai and partners like Neon and Clerk, which provide free services for his open source projects.

Prompt Engineering and AI Red Teaming — Sander Schulhoff, HackAPrompt/LearnPrompting

Jul 14, 2025 · 2:01:05

Sander Schulhoff, creator of Learn Prompting and HackAPrompt, argues prompt engineering remains vital despite claims of its demise, drawing on his systematic review of 1,500+ papers for 'The Prompt Report.' He covers advanced techniques including chain-of-thought, decomposition, ensembling, and few-shot prompting, noting that role prompting is ineffective for accuracy-based tasks and that example ordering can swing performance by 50%. He then explains AI red teaming, distinguishing jailbreaking from prompt injection, and warns that system prompts and guardrails cannot prevent attacks—even simple obfuscation like base64 or typos still works. He highlights the critical unsolved problem of agentic security, where agents with real-world actions are easily tricked, and introduces a live competition at the conference to gather more attack data. His key takeaway: AI security is fundamentally harder than classical cybersecurity because 'you cannot patch a brain.'

The emerging skillset of wielding coding agents — Beyang Liu, Sourcegraph / Amp

Jun 30, 2025 · 35:06

Beyang Liu, CTO and co-founder of Sourcegraph, argues that coding agents are a real and high-ceiling skill, contrary to skeptics like Jonathan Blow and Eric S. Raymond. He presents design decisions for the agentic era: agents should directly edit files instead of asking permission, UIs should be minimal (like Amp's bare-bones VS Code extension and CLI), and fixed pricing should give way to usage-based models. In a live demo, Liu uses Amp to implement a custom icon for the linear connector in Amp's own codebase, demonstrating agentic search and sub-agents. He shares power-user patterns, including writing long prompts, constructing feedback loops with Playwright and Storybook, and running multiple agents in parallel—as exemplified by Jeff Huntley using Amp to build a compiler while sleeping. Liu emphasizes that agents should enable more thorough code reviews, not replace human understanding.

To the moon! Navigating deep context in legacy code with Augment Agent — Forrest Brazeal, Matt Ball

Jun 27, 2025 · 15:24

Forrest Brazeal and Matt Ball demonstrate how Augment Agent helps developers navigate and modernize legacy codebases using the Apollo 11 guidance computer as an example. They show that Augment's chat mode can instantly explain the 1202 program alarm—indicating the computer was overloaded but could continue—by reading assembly code and web context. Then, using agent mode, they prompt the tool to write a Python implementation of the P65 vertical descent algorithm from the original assembly, run a simulator, and land the lunar module within expected parameters. The episode argues a three-step strategy for any legacy codebase: use chat to understand, write tests via agent, and then modernize modularly—proving AI agents can turn toil like Java 8-to-17 migrations into automated tasks.

Don’t get one-shotted: Use AI to test, review, merge, and deploy code — Tomas Reimers, Graphite

Jun 27, 2025 · 5:45

Tomas Reimers, CPO and co-founder of Graphite, argues that AI-generated code accelerates the inner loop (development) but bottlenecks the outer loop (review, test, merge, deploy), necessitating AI-native tooling across the entire workflow. He presents Graphite's AI code reviewer, Diamond, which achieves a 52% acceptance rate on its comments—exceeding the 45–50% rate for human comments—while only 4% of its comments are dismissed. Reimers emphasizes that AI tools must deeply understand the codebase, integrate with CI/testing pipelines, and provide high-signal, low-noise feedback to handle the increasing volume of AI-assisted code. He cites that 46% of GitHub code is now written by Copilot, and predicts the outer loop will break without AI-driven prioritization, review automation, and optimized deployment. The talk advocates for self-driving code review that lets developers focus on product features rather than manual processes.

[Full Workshop] Vibe Coding at Scale: Customizing AI Assistants for Enterprise Environments

Jun 27, 2025 · 1:20:38

Harold from the VS Code team demonstrates "Vibe Coding at Scale" at the AI Engineer World's Fair, presenting three stages—YOLO, structured, and spectrum vibes—for customizing AI assistants in enterprise environments. He live-codes a hydration tracking app using GitHub Copilot's agent mode, auto-approve, and new workspace scaffolding, then shows how to enforce design principles via Copilot instructions and reusable prompts. Harold introduces custom modes (e.g., TDD mode), tool sets, and MCP servers like Playwright and Perplexity for research and browser testing. He emphasizes iterating on instructions, committing often, and using spec-driven development to balance speed with reliability, arguing these techniques enable true flow-state collaboration even on complex codebases.

Unlocking AI Powered DevOps Within Your Organization — Jon Peck, GitHub

Jun 27, 2025 · 22:13

Jon Peck, a developer advocate at GitHub, explains how organizations can unlock AI-powered DevOps by integrating tools like GitHub Copilot into their workflows, achieving up to 1.5x feature throughput and 30% average improvement in successful builds. He emphasizes starting with brownfield (existing code) rather than greenfield, using agent mode for iterative planning and narrowed context, and codifying team best practices via copilot-instructions.md files. Peck also covers governance: safety, privacy, org-wide policies, and excluding sensitive files. Autonomous AI use cases include auto-generating PR descriptions, code review, and assigning issues to Copilot, always keeping a human in the loop with isolated branches. Finally, MCP servers allow agents to interact with GitHub and other systems, automating commits and PR creation.

Vibe Coding at Scale: Customizing AI Assistants for Enterprise Environments - Harald Kirshner,

Jun 27, 2025 · 15:26

Harald Kirshner, a Principal Product Manager at Microsoft working on VS Code and GitHub Copilot, argues that 'vibe coding' must evolve from ad-hoc experimentation to structured, spec-driven approaches for enterprise environments. He breaks vibe coding into three stages: YOLO (rapid prototyping with auto-accept and voice dictation), structured (using templates, custom instructions, and tools like MCP for consistency), and spec-driven (iterating from a plan for scale and reliability). Kirshner demonstrates VS Code Insiders' agent mode, new workspace flow, and auto-approve features, emphasizing that developers should commit often, review agent pauses, and leverage workspace instructions to maintain quality. He advocates for standardizing AI usage through team-defined prompts and modes, preserving developer autonomy while ensuring maintainable code even in complex, polyglot codebases.

Prompt Engineering is Dead — Nir Gazit, Traceloop

Jun 27, 2025 · 14:19

Nir Gazit, CEO of Traceloop, argues that manual prompt engineering is ineffective and demonstrates an automated, test-driven alternative. He improves a RAG-based documentation chatbot by 5x without hand-tuning prompts, using an LLM-as-a-judge evaluator with 20 example questions (each with 3 expected facts) and a CrewAI agent that iteratively optimizes the prompt based on evaluation scores. The initial score of 0.4 jumps to 0.9 after just two iterations. Gazit warns of overfitting to the small dataset and suggests splitting into train/test sets. The approach replaces manual tweaking with a 'gradient ascent' on evaluators, effectively vibe-engineering prompts.

The Eyes Are The (Context) Window to The Soul: How Windsurf Gets to Know You — Sam Fertig, Windsurf

Jun 27, 2025 · 13:37

Sam Fertig, Deployed Engineer at Windsurf, explains how the AI coding IDE achieves its uncanny ability to predict developer intent by focusing on relevant context rather than merely expanding context windows. He breaks context into two pillars: heuristics (user behavior like cursor position, open files, clipboard) and hard evidence (codebase, rules, memories). The core challenge is generating code that fits existing codebases, adheres to personal and organizational standards, and remains future-proof. Windsurf optimizes retrieval quality over quantity, using techniques like embedding search, reranking, and AST parsing to balance latency and scale. Fertig also addresses privacy, stating that while code is sent to servers, data is not stored or used for training.

Mastering Engineering Flow with Windsurf - Eashan Sinha, Windsurf

Jun 27, 2025 · 15:20

Eashan Sinha, a Deployed Engineer at Windsurf, explains how engineering teams can master agentic IDEs using Windsurf's Cascade agent. He argues that developers should treat AI as a teammate rather than a tool to abuse, emphasizing Windsurf's flow awareness—tracking user actions, edits, and terminal commands to infer intent. Sinha details Windsurf's evolution from copilots to agents, combining collaborative chat with autonomous tool calling, and introduces features like state-of-the-art context engine, workflows for deterministic steps, rules, memories, and simultaneous Cascades. He provides a practical four-step framework—explore, plan, build, test—for maximizing agentic flow, including using planning files, checkboxes, and automated test generation. The episode delivers a concrete guide to integrating AI into development without sacrificing quality, recorded at the AI Engineer World's Fair in San Francisco.

Supercharging developer workflow with Amazon Q Developer - Vikash Agrawal

Jun 10, 2025 · 13:22

Vikash Agrawal and Linda show how Amazon Q Developer can drive every stage of the software development lifecycle. They build a 2048 game with FastAPI and Poetry using Q's /dev command in the CLI, then add unit tests with /test, generate documentation with /doc, and integrate with GitHub for automated pull requests. After deploying to AWS, they use Q in CloudWatch to debug a runtime import error by analyzing logs and resource topology. The episode emphasizes that planning infrastructure and prompts upfront, along with security scans, are critical for production-ready AI-assisted development.

Why Bolt.new Won and Most DevTools AI Pivots Failed - Victoria Melnikova

Jun 10, 2025 · 5:45

Victoria Melnikova explains how StackBlitz's Bolt.new achieved explosive growth from near-death to over 100M ARR by avoiding common AI pivot traps. She outlines three mistakes: adding a chatbot UI, competing broadly against big AI labs, and waiting for perfect AI instead of building tactically. StackBlitz succeeded by identifying its unfair advantage—running full dev environments in the browser without containers or servers—and amplifying it with AI to let users describe apps and see them live instantly. This created a new category of vibe coding, not just an AI feature. Melnikova urges devtools founders to find their unique moat and reinvent workflows rather than bolt AI onto existing ones.

The Many Ends of Programming - Ray Myers

Jun 10, 2025 · 27:12

Ray Myers, Chief Architect at All Hands AI, argues that the future of software engineering is not predetermined and we have a say, categorizing competing visions into six distinct endgames: extreme completion, dev apocalypse (innovator's paradise), abstraction leap, uncharted waters, review economy, and infinite pile of garbage. He illustrates extreme completion with a real example where an AI agent generated a 48-file pull request from a Slack message, and abstraction leap via domain-specific languages (DSLs) shown in the Parcel paper’s 61 prompts producing a 220-line Lisbon interpreter. Myers warns that dev apocalypse is desirable but not yet feasible, while the review economy is a pit stop where manual review bottlenecks must be managed using Theory of Constraints. He cites contradictory studies: UpLevel found developers with coding assistants had higher bug rates, while GitHub Copilot claimed improved quality; he notes 2024 was the first year copy-pasted code exceeded refactored code, a red flag for tech debt. Ultimately, Myers urges deliberate choices: whether we want no programmers or everyone to be a programmer, and whether we prioritize quality over quantity.

The 4 Patterns of AI Native Development — Patrick Debois

Jun 4, 2025 · 14:11

Patrick Debois presents four patterns of AI-native development that redefine software engineering roles: from producer to manager (reviewing AI-generated code, managing agents), from implementation to intent (specifying requirements in markdown files), from delivery to discovery (rapid prototyping with tools like Lovable and Bolt), and from content creation to knowledge (capturing learnings for reuse). He argues these shifts mirror senior developer activities and are supported by new tools, such as the AI Native Dev landscape featuring roughly 300 tools. The talk emphasizes that AI changes not just coding speed but the entire workflow, reducing cognitive load through smarter reviews and intent-based coding.

The Coherence Trap: Why LLMs Feel Smart (But Aren’t Thinking) - Travis Frisinger

Jun 3, 2025 · 20:47

Travis Frisinger argues that large language models are not intelligent but coherent, introducing 'coherence reconstruction' as a mental model for understanding their utility. He explains that hallucinations are a feature, not a bug, as models fill gaps predictably to maintain pattern completion. Prompts act as force vectors navigating latent space, and the key to reliable outputs is a 'frame, generate, judge, iterate' loop. Frisinger presents a three-layer model: latent space, execution layer (tools and RAG), and conversational interface. He advises engineers to design for emergence rather than control, using dense context as anchors to steer generation, and to watch for breakdowns in tone as early signs of lost coherence.

Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier - Andrew Thomspson

Jun 3, 2025 · 25:09

Andrew Thompson, CTO of Orbital, introduces the concept of Prompt-Tax: the hidden cost of migrating prompts when upgrading AI models. He shares how his agentic product automates real estate due diligence, growing from <1B to 20B monthly tokens and zero to multiple seven-figure ARR over 18 months. Key tactics include optimizing for prompting over fine-tuning, using domain experts (ex-lawyers) to write prompts, and relying on vibes over formal evals. Thompson advocates 'betting on the model'—shipping new frontier models immediately and fixing regressions on the fly using progressive rollouts and rapid feedback loops. A clip from Demis Hassabis underscores the unique challenge of evolving tech stacks. The episode concludes with questions on whether evals or progressive delivery scale to manage Prompt-Tax.

Self Coding Agents — Colin Flaherty, Augment Code

Apr 21, 2025 · 17:23

Colin Flaherty, founding researcher at Augment Code, describes how the company built an AI coding agent that wrote over 90% of its own 20,000-line codebase with human supervision. The agent autonomously added third-party integrations like Google Search and Linear by searching its own codebase and even used its own Google Search tool to look up Linear API docs. It profiled and optimized itself, replacing synchronous file hashing with a process pool after adding print statements and running sub-copies. Flaherty argues that agent capabilities improve with better context engines, code execution environments, and test harnesses—adding test-driven iteration boosted a bug-fixing benchmark by 20% versus 4% from just upgrading the foundation model. He cautions that agents are general-purpose, not category-specific, and have different strengths than humans, making knowledge bases crucial for onboarding. With code becoming cheap, the focus shifts to product insights and design, while better tests enable greater autonomy and accelerate agent self-improvement.

AI Engineers: The Next Generation — Stefania Druga, Google Gemini

Apr 13, 2025 · 21:47

Stefania Druga, a research scientist on Google Gemini, presents Cognimates and its new AI copilot that teaches children to become AI engineers. Built on Scratch’s visual programming, Cognimates lets kids train custom models, program robots, and build games, shifting their perception from ‘AI is magic’ to understanding data and confidence levels. Druga’s design studies with 18 kids from 11 countries showed the copilot doubled programming time by supporting ideation, debugging, and creative agency—never giving answers unless stuck three times. The tool now integrates screenshot analysis and asset generation, with plans to embed the agent across the OS. Druga ties this to the EU AI Act’s mandate for AI literacy, arguing early, hands-on creation is the path to informed users.

Building AI Agents with Real ROI in the Enterprise SDLC: Bruno (Booking.com) & Beyang (Sourcegraph)

Apr 8, 2025 · 20:56

Beyang Liu, CTO of Sourcegraph, and Bruno Passos, Group PM at Booking.com, explain how their partnership built AI coding agents that deliver measurable ROI in enterprise software development. Booking.com, with 3,000 developers handling 250,000 merge requests and 2.5 million CI jobs yearly, used Sourcegraph's Cody and custom agents to automate code migration and review, cutting cycle times. They report that daily Cody users ship 30% more pull requests and are 30% faster, with lighter MRs. Agents like a GraphQL generator for a million-token schema and a code review tool with customizable rules emerged from joint hackathons. Bruno emphasizes education as critical: training turned skeptical developers into daily users, driving adoption and productivity gains. The goal is to shift-left compliance and reduce tech debt, making the SDLC self-healing.

How Windsurf writes 90% of your code with an Agentic IDE - Kevin Hou, Windsurf

Mar 11, 2025 · 20:50

Kevin Hou, head of product engineering at Codeium, introduces Windsurf as the first AI agent-powered editor, arguing that agents are the future of software development. The editor is built on three principles: trajectories, a unified timeline that lets the agent read the user's mind and execute commands like 'continue my work'; meta-learning, which auto-generates memories of user preferences and codebase context; and scaling with intelligence, removing legacy features like chat and @-mentions as models improve. Windsurf launched three months ago and has already generated 4.5 billion lines of code, with 90% of code now written via its Cascade agent. Hou emphasizes that the agent reduces human-in-the-loop effort by inferring context, running terminal commands in the user's environment, and dynamically retrieving documentation—allowing developers to contribute less input while getting more production-ready output.

WTF do people use Open Models for??

Feb 22, 2025 · 28:01

Eugene Cheah of Featherless.ai breaks down how individuals and enterprises actually use open-source AI models, based on platform data. DeepSeek R1 dominates individual usage, but Mistral Nemo 8B remains the top enterprise model due to production stickiness and Apache 2.0 licensing. Creative writing and roleplay account for 30–40% of all traffic, with over 60% of users in that segment being women; coding copilots and agents make up 20–30%, driven by 'vibe coding' and token-hungry workflows like Kline. RAG and ChatGPT clones represent 20%, while agentic workflows (10–20%) succeed with human-in-the-loop designs. Cheah advises enterprises to aim for 80% automation with escape hatches, and warns against chasing 100% reliability. He concludes by introducing Quirky, a post-transformer hybrid built for $100k.

The Model Isn’t Wrong—You’re Just Bad at Prompting

Feb 22, 2025 · 8:54

Dan from PromptHub argues that prompt engineering remains critical for improving LLM outputs, covering Chain of Thought, few-shot, and meta prompting techniques. Chain of Thought breaks problems into sub-problems and is built into reasoning models; few-shot prompting works best with just one or two diverse examples, but can degrade performance on reasoning models like O1 and R1. Meta prompting uses LLMs to write or refine prompts, with PromptHub offering model-specific enhancers. For reasoning models, Dan advises minimal prompting, encouraging more reasoning instead of few-shot, and avoiding instructing the model on how to reason. Free resources include PromptHub's templates, the AutoReason prompt, and the Prompt Engineering Substack.

Lets Build An Agent from Scratch — Kam Lasater

Feb 22, 2025 · 19:29

Kam Lasater demonstrates that an AI agent is simply an LLM paired with memory, planning, tools, and a while loop, and shows how to build one from scratch in incremental steps. He starts with a basic LLM call (step 0), adds a condition using an LLM as judge to check if an answer is complete (step 1), then incorporates a tool for web search via SERP AI (step 2). After refactoring for parallel tool calls (step 3), he introduces a to-do list for planning and tracking progress, along with a browse-web tool (step 4). The agent uses a to-do list to plan, execute tasks like searching and browsing, and then checks completion via LLM as judge. Lasater illustrates with examples: finding the average wind speed of a swallow, buying a hoodie near Times Square, and planning a date in Ardmore. He emphasizes that the simplest agent emerges naturally from combining these components, and encourages running and breaking the provided code.

Lessons from building GenAI based applications — Juan Peredo

Feb 22, 2025 · 33:13

Juan Peredo details the hidden complexities of building GenAI applications, from model hosting and cost control to output validation and observability. He compares local (Ollama) vs cloud hosting (Modal, SkyPilot) and warns that an agent processing 3,000 calls/day with OpenAI O1 costs nearly $300,000/month, while LLaMA 3.3 70B drops that to $50,000/month. He explains techniques to mitigate hallucinations—prompt engineering, guardrails, RAG, and fine-tuning—each with trade-offs like added latency or cost. Peredo advocates externalizing prompts via LangChain Hub for easy iteration and future-proofing, and illustrates agent design with parallel calls to reduce latency. Finally, he stresses observability using tools like LangSmith to debug probabilistic failures, such as an LLM failing on case sensitivity.

Optimizing LLMs in Insurance with DSPy: Jeronim Morina

Feb 16, 2025 · 19:29

Jeronim Morina from AXA Germany argues that AI engineers must stop relying on manual prompt tuning and instead adopt DSPy's automated optimization combined with first-principles thinking. He explains how his team built a customer-facing insurance chatbot by first crafting clear problem definitions, annotating evaluation data, and avoiding data leakage. After initial struggles with fragile prompts and overly complex error-handling code, they modularized their system into DSPy modules and created custom metrics for German text. Morina emphasizes that tools like DSPy require a steep learning curve and a solid baseline of hand-written prompts and basic evaluations before optimization. The episode calls for engineers to focus on real-world impact, measure everything, and break down problems into discrete, optimizable steps.

Claude plays Minecraft!

Feb 15, 2025 · 18:16

Derek from AWS demonstrates building Rocky, an autonomous Minecraft agent using Amazon Bedrock, Claude 3 Haiku, and serverless AWS services, proving agentic workflows enable AI to reason and act beyond chatbots. He explains the architecture: Minecraft server on ECS with Mineflare bot framework, agents for Bedrock with return of control, and prompt engineering for tasks like digging and building. In a live demo, Rocky jumps, finds players, digs a hole, and builds a double-decker couch from a chat command, inferring parameters from natural language. The code is open-source and built with CDK/CloudFormation, emphasizing that agentic AI can operate in complex 3D environments through tool orchestration and managed prompt design.

GitHub Next Explorations: Rahul Pandita

Feb 8, 2025 · 18:18

Rahul Pandita, a researcher at GitHub Next, presents two explorations: Copilot Next Edit Suggestions and Copilot Workspace. Next Edit Suggestions extends GhostX by suggesting multi-location edits beyond the cursor, such as updating method definitions and documentation when a parameter is added. Copilot Workspace addresses developers' pain points of getting started on tasks, trusting AI output, and retaining control by offering a one-click proposal, built-in runtime verification, and an iterative, collaborative environment. Pandita demonstrates updating a MonoSpace website to add Rust syntax highlighting via a GitHub issue, showing specification generation, planning, and terminal command synthesis. He emphasizes that GitHub Next explores uncertain futures rapidly, learning from experiments like Copilot (which started as a Next project) to shape AI-augmented software engineering without waiting decades.

[Full Workshop from Microsoft] Github Copilot - The World's Most Widely Adopted AI Developer Tool

Feb 7, 2025 · 1:19:45

This workshop from the AI Engineer World's Fair features GitHub's Christina Warren, Dave, Alex, and Harald presenting GitHub Copilot, the world's most widely adopted AI developer tool. They demonstrate Copilot's three interaction modes: ghost-text completions using GPT-3.5 for speed, inline chat (also GPT-3.5) for quick code edits, and the chat panel powered by GPT-4 Turbo for deeper conversations. Speakers emphasize prompt crafting—being specific, providing examples, and keeping relevant files open—to improve results. They show how to use slash commands (/fix, /explain, /test) and the new attach button for explicit context. The workshop uses GitHub Codespaces preconfigured with Python 3.11 and Copilot extensions; a coupon provides a 7-day free trial. Harald explains the trade-off between model quality and latency, and notes upcoming features like automatic agent delegation and workspace integration for cross-file edits.

[Full Workshop] How to add secure code interpreting in your AI app: Vasek Mlejnsky

Feb 6, 2025 · 1:48:16

Vasek Mlejnsky, CEO of E2B, demonstrates how to add secure AI code execution to any app using E2B’s Code Interpreter SDK alongside Anthropic's Sonnet 3.5 and Vercel’s AI SDK. The workshop builds an open-source version of Claude’s artifacts UI, running Python code in isolated Firecracker VMs that start in ~900ms and support custom environments via Dockerfiles. Vasek explains how E2B’s Jupyter-server-based sandbox returns stdout, stderr, runtime errors, and rich outputs like PNG charts, which are streamed to the front end using Vercel’s stream data. He covers security architecture—full root access in a VM that self-destructs on escape—and plans for AWS/GCP self-hosting and snapshot-based agent branching. The episode also addresses customizing sandboxes with private packages, mounting cloud storage (S3, GCS), and interpreting table previews as HTML or images.

GitHub Copilot: The World's Most Widely Adopted AI Developer Tool

Feb 6, 2025 · 29:49

GitHub Senior DevOps Advocate Dave Bernason demonstrates how GitHub Copilot has evolved from an AI code generator to a comprehensive developer assistant with chat, enterprise knowledge bases, pull request summaries, and third-party extensions, emphasizing that Copilot keeps developers in flow and requires human oversight. He shows Copilot Chat explaining code, refactoring, debugging, and generating unit tests. Copilot Enterprise adds Bing search, knowledge bases from Markdown files, and repo indexing for accurate answers, exemplified by updating a hard-coded sales tax function to use the Avalara API. Copilot Extensions integrate third-party tools like Octopus Deploy for deployment dashboards within chat. Bernason also covers prompt engineering tips—specificity improves suggestions—and highlights Copilot's support for any language, even COBOL, for modernization.

Accelerate your AI journey with Azure AI model catalog: Sharmila Chokalingam

Feb 6, 2025 · 23:14

Shubhi and Sharmila present the Azure AI model catalog as a platform offering over 1,600 models including GPT-4, Mistral, Llama, Cohere, and Phi3, with a standardized inference API enabling easy model swapping. They demonstrate deployment via serverless API (pay-per-token) and managed compute, emphasizing that customer prompts and completions are not shared with model providers or used for training. The platform includes model benchmarks, playground for RAG, and Prompt Flow for building generative AI apps with evaluation and variant comparison. Customer success stories include EY's EYQ chat adopted by 275,000 employees, CMA CGM's reduced response latency using Mistral, and Bridgestone's 30% reduction in forecasting errors with Nixtla's TimeGen.

The Rise of the AI Software Engineer: Jesse Han

Feb 5, 2025 · 3:48

Jesse Han, CEO of Morph Labs, announces the Morph Code Index and RiffCoder 7B model as milestones toward the personal AI software engineer. The Code Index is a neurosymbolic code database that enables semantic search over codebases, makes entire Git history visible to coding assistants, and transmutes code into training data. It uses static analysis, vector embeddings, and graph algorithms for state-of-the-art code search, with a query language guaranteeing precise results. RiffCoder 7B, trained on data from the Code Index, is the only open-source code editing model that runs on laptops and IDEs. Morph Labs is building a platform for managing personal AI software engineers, available via waitlist. The future of software should belong to everyone, and the tools are open-source and free.

Best Practices for Evaluating Large Language Model Applications with llmeval: Niklas Nielsen

Feb 5, 2025 · 9:33

Niklas Nielsen, CTO and co-founder of Log10, introduces llmeval, a command-line tool that enables teams to ship reliable LLM applications by evaluating and testing prompts and configurations. The tool initializes with four lines of code, uses Meta's Hydra for configurable test structures, and runs multiple samples (default 5) per test to assess stability. Nielsen demonstrates prompt engineering for a math problem, showing how stripping spaces or adding instructions like 'only return the answer' affects strict pass/fail results across Claude, GPT-4, and GPT-3.5. Advanced use cases include testing tools for Python code generation and model-based evaluation, where a larger model grades outputs on criteria like mermaid diagram quality. He notes pitfalls: models favor their own output and struggle with point scores. Log10 addresses this by bridging model-based and human feedback—collecting prior human ratings to train an 'auto-John' that pre-fills reviews for new completions.

Prompt Engineering Tactics: Dan Cleary

Feb 5, 2025 · 5:12

Dan Cleary, co-founder of PromptHub, presents three research-backed prompt engineering tactics to improve LLM output consistency and reduce hallucinations. Multipersona prompting, from University of Illinois, calls on multiple AI agents to collaborate on complex tasks like writing a book. The "according to" method, from Johns Hopkins, grounds prompts to a specific source (e.g., "according to Wikipedia") and can reduce hallucinations by up to 20%. The motion prompt, from Microsoft, adds emotional stimuli at the end of prompts, improving output accuracy by 8% to 115% depending on the task. These tactics are available as templates on PromptHub. Cleary emphasizes that even small changes in prompts can have outsized effects, crucial for maintaining user trust in AI-integrated products.

Scaling AI in Education: A Khanmigo case study: Shawn Jansepar

Feb 5, 2025 · 22:39

Shawn Jansepar, Khan Academy's Director of Engineering, details building Khanmigo, an AI tutor and teacher assistant on GPT-4 with OpenAI, arguing generative AI can democratize one-on-one tutoring. He describes a rapid prototyping culture that launched Khanmigo in three months via a company-wide hackathon, replacing traditional agile with a prototype-to-beta-to-launch framework. Technical challenges include math accuracy through a math agent and chain-of-thought prompting, refactoring prompts into a component architecture for testing, and managing scale using multiple models and dedicated Azure compute. Jansepar highlights ethical design: a Socratic tutor that avoids answers, teacher moderation, and a writing coach with revision history. Khanmigo now has 200,000 paid users, half in school districts, with teacher tools sponsored by Microsoft free to US teachers. Plans include releasing a math tutoring benchmark and evaluating smaller models like Phi-3.

The ROI of AI: Why you need Eval Framework - Beyang Liu

Dec 31, 2024 · 25:28

Beyang Liu, CTO of Sourcegraph, presents six evaluation frameworks for measuring the ROI of AI coding tools, arguing that no single metric suffices because AI ROI reduces to the intractable problem of developer productivity. He dismisses the 'roles eliminated' framework as irrelevant for engineering, noting that backlogs are never fully addressed. Instead, he highlights customer examples: A/B testing velocity at Palo Alto Networks showed 20-30% acceleration with Cody; time saved as a function of engagement lower-bounds gains (e.g., each code search saves ~2 minutes); and LITOS tracked value-add zones (building features, writing tests, reviewing code) instead of lines of code. For key initiatives like code migrations, ROI is measured by how many months the migration is pulled forward. Liu also advocates developer surveys bounded by a 5-25% productivity tool budget, and warns against the Mythical Man Month fallacy: preferring one 100x human developer over 100 mediocre AI agents.

The AI emperor has no DAUs why most devs still don't use code AI: Quinn Slack

Nov 20, 2024 · 18:45

Quinn Slack, CEO and cofounder of Sourcegraph, argues that despite massive hype, only about 5% of professional developers actually use Code AI tools, with total recurring revenue from Code AI sitting at roughly $300 million ARR—a fraction of Salesforce's $36 billion. He cites GitHub's 1.3 million paid Copilot subscribers and just 935,000 yearly active users receiving suggestions, revealing the gap between perception and reality. Slack warns that the entire AI ecosystem—foundation models, infra, and applications—risks collapse if usage doesn't grow, and most revenue in AI flows to NVIDIA and chip makers, not software. From building Cody, the number two Code AI product, he shares lessons: hype fools everyone, autocomplete is a freakishly good feature that spoils expectations, while chat and agents are harder to vet and adopt. He advises builders to use their own product daily, ignore customer demands for buzzwords like fine-tuning, and manually build explicit interactions before adding magic. Slack concludes that the industry must collectively dehype and focus on real daily active users to turn the potential into sustained enterprise revenue.

Productionizing GenAI Models – Lessons from the world's best AI teams: Lukas Biewald

Oct 23, 2024 · 22:36

Lukas Biewald, founder of Weights and Biases, shares lessons from productionizing GenAI models, emphasizing that while AI is easy to demo, it is hard to productionize. He illustrates this with a personal project building a custom Alexa-like device using LLaMA and Whisper, where accuracy improved from 0% to 98% through prompt engineering, switching to Mistral, and fine-tuning with QLoRA. Biewald argues that tracking experiments (including failures) is critical for reproducibility and collaboration, and that a robust evaluation framework—beyond 'testing by vibes'—is essential for iterating and shipping v2. He notes that 70% of the audience had LLM apps in production, yet many lack solid evaluations, and recommends starting with lightweight prototypes and incorporating end-user feedback.

Code Generation and Maintenance at Scale: Morgante Pell

Oct 17, 2024 · 18:54

Morgante Pell, founder of Grit, argues that AI agents for code generation must be built for modifying large existing codebases rather than generating new apps, and that combining static analysis with AI enables reliable migrations at scale. Grit has merged more PRs than any other company by focusing on supercharging top engineers, using GritQL to precisely find code via syntactic and semantic queries, and relying on compilers like TSC to catch errors that LLMs miss. They overcome slow enterprise builds (10 minutes for type checking) by precomputing in-memory indexes and using Firecracker to snapshot and fork environments for parallel agent execution, achieving PRs with only a few iterations. For editing, they developed a custom 'loose search and replace' format to avoid expensive full-file generation and LLM laziness. This approach allowed one customer to complete a multi-year OpenTelemetry migration in a week with under 100 developer hours. Morgante envisions future UIs that let engineers manage entire codebases like SimCity.

No more bad outputs with structured generation: Remi Louf

Oct 14, 2024 · 15:32

Rémi Louf, CEO of .txt and co-maintainer of Outlines, argues that structured generation—guiding LLMs to produce valid regex, JSON, or context-free grammar outputs—eliminates parsing errors and hallucinations while adding negligible overhead. Outlines masks tokens that violate the target structure, enabling Mistral 7B to achieve 99.9% valid JSON (vs. 17% without). It also accelerates inference: a chatty 50-token ChatGPT answer shrinks to 8 tokens, and structured generation boosts open models beyond GPT-4—Phi-3 Medium hits 96.5% on Berkeley function calling (GPT-4 gets 93.5%). With one shot matching eight shots in accuracy, Louf contends that most text is structured and that structured generation should be the default for non-chatbot workflows.

We accidentally made an AI platform: Jamie Turner

Oct 8, 2024 · 5:36

Jamie Turner, co-founder of Convex, explains how his company's backend platform accidentally became an AI platform because its reactive data flow paradigm—extending React's state reactivity to the server with subscribable queries and mutations—perfectly suits generative AI workflows. He describes Convex's architecture as seamlessly syncing state between backend steps and the application, enabling concurrent chains of tasks like automatic speech recognition, summarization, embedding generation, and finding related notes. Turner notes that post-ChatGPT, over 90% of Convex projects are generative AI. To support this shift, Convex added native vector indexes for schema fields and launched a startups program with discounts and exclusive forums. Upcoming high-level components will encapsulate state machines for sophisticated workflows. Turner presents Convex as a way for generative AI engineers to ship quickly and confidently.

Judging LLMs: Alex Volkov

Sep 9, 2024 · 18:39

Alex Volkov, as an LLM judge from 2034, humorously judges AI engineers on their development practices, emphasizing the importance of tracing, iterative prompt engineering, and robust evaluation pipelines. He finds Daniel guilty of deploying without logging, Sasha guilty of premature fine-tuning without prompt iteration, and Morgan guilty of ignoring AI news—commuted to attending ThursdAI. Francisco’s overreliance on programmatic evals leads to a legal loss, while Maxime is 'awesome' for using Weights & Biases Weave. Alex concludes with a primer on evaluation methods: programmatic, human-in-the-loop, and LLM-as-judge, stressing the need to validate validators and create custom criteria. He promotes Weave for tracing and evals, and ThursdAI for staying updated.

Pydantic is STILL all you need: Jason Liu

Sep 6, 2024 · 15:21

Jason Liu returns to the AI Engineer World's Fair to argue that Pydantic (and his library Instructor) is still all you need for structured output with LLMs. He shows that the core API—response_model, streaming with iterables, and partials for real-time validation—remains unchanged, now supporting Ollama, LlamaCPP, Anthropic, Gemini, and more. Liu demonstrates validators that enforce rules like uppercasing names or verifying receipt totals, reducing errors with automatic retries. He applies structured output to RAG: using a Search model with optional date ranges and source selection, and a Response model with follow-up questions and validated URLs. For extraction, he creates classifiers using Literal types, meeting summaries with action items, and even tables as Pandas DataFrames via custom type hints. Liu’s key takeaway is that one retry often suffices, and as models get faster and smarter, structured output makes LLMs compatible with classical programming—turning generative AI into generating data structures defined by the developer.

Git push get an AI API: Ryan Fox-Tyler

Aug 23, 2024 · 45:01

Ryan Fox-Tyler and Matt Johnson Pint from Hypermode present a hands-on workshop demonstrating how to build and iteratively improve AI features using their platform, focusing on a GitHub issue triage app. They first illustrate the process with a multiplayer game (Hypercategories) that uses AI for classification and scoring. Then they build a trend summary function using OpenAI GPT-4 to summarize repository issues, and a classify issue function with a Hugging Face DistilBERT model for labeling issues as bug, feature, or question. Finally, they add natural language search for similar issues by creating an embeddings model (MiniLM) and a Hypermode collection, enabling vector search without external databases. The workshop emphasizes incremental iteration, mixing AI models with traditional code, and using Hypermode's automatic GraphQL generation and observability to speed development.

Hypermode Launch: Kevin Van Gundy

Aug 23, 2024 · 5:03

Kevin Van Gundy, founder of Hypermode, argues that iteration velocity is the compound interest of software and key to succeeding with AI, and presents Hypermode as a runtime that makes AI approachable by integrating models and data into AI functions with minimal friction. Drawing from his experience at Vercel, he explains how rapid iteration helped them win against competitors, and applies the same philosophy to AI development. Hypermode removes common pain points: it offers one-request RAG with in-memory embedding and search, model comparison and fine-tuning export, and strong defaults that work with existing stacks. The platform de-risks getting it wrong by making model switching and prompt changes frictionless. Kevin invites listeners to a workshop to build a demo and offers $1000 in Hypermode credits to get started.

10x Development: LLMs For the working Programmer - Manuel Odendahl

Aug 21, 2024 · 1:14:46

Manuel Odendahl presents a workshop on using LLMs as a translation engine to achieve 10x development productivity, arguing that treating models as cultural rather than purely technological artifacts enables novel techniques like world simulation for code reviews and domain-specific language creation. He demonstrates concrete methods: regenerating responses, editing model outputs, clearing context frequently, and summarizing transcripts into actionable formats. Odendahl introduces his tool Prompto for managing code fragments and emphasizes that fundamental programming knowledge remains essential while API-specific details become less critical. The episode focuses on decomposing problems into language translation steps, using examples such as transforming meeting transcripts into GitHub issues and building applications by instructing the LLM to simulate itself as the app. Odendahl advocates for divergent thinking and human-centered output, noting that all generated artifacts should ultimately serve human understanding.

Building with Anthropic Claude: Prompt Workshop with Zack Witten

Aug 17, 2024 · 1:34:56

In this live prompt engineering workshop, Anthropic's Zack Witten and Jamie Neuwirth demonstrate techniques for improving Claude prompts in real time, using the Anthropic console. They show that XML tags clearly separate prompt sections, that placing instructions after information improves adherence, and that prefilling assistant responses with opening JSON or tags reliably forces JSON output without preamble. Witten advises using stop sequences and code to handle formatting rather than over-prompting, and emphasizes that few-shot examples—especially contrastive pairs with reasoning—drive more improvement than any other technique. The workshop covers role-playing multiple personas by routing via code, mitigating hallucinations by extracting quotes before summarization, and grading translations with chain-of-thought and fine-grained examples.

What's new from Anthropic and what's next: Alex Albert

Aug 5, 2024 · 13:38

Alex Albert, Head of Developer Relations at Anthropic, argues the industry is in a 'magic star icon phase'—tacking AI onto existing products—and urges rebuilding from the ground up. He announces Claude 3.5 Sonnet, which scores 64% on pull request evals (vs 38% for 3 Opus), with 200k context, vision improvements, and pricing at $3/M input and $15/M output tokens. New features include Artifacts for collaborative creation of code and React components from screenshots, Projects for team knowledge grounding, and an enhanced ToolUse API supporting hundreds of tools and structured JSON output. Future models (3.5 Haiku/Opus) will be smarter, cheaper, and faster; a steering API for interpretability allows clamping feature values to control outputs.

Unlocking Developer Productivity across CPU and GPU with MAX: Chris Lattner

Jul 25, 2024 · 18:33

Chris Lattner, CEO of Modular, presents MAX, a unified AI framework that accelerates Gen AI inference by combining CPU and GPU programming into a single Pythonic model, and Mojo, a new programming language that extends Python to systems programming with 100–1000x speedups. Lattner argues that current fragmentation across PyTorch, ONNX, TensorRT, and hardware-specific libraries slows innovation, and MAX replaces the entire stack—including cuDNN and Intel MKL—with a consistent, compiler-driven approach. He demonstrates that MAX's Int4/Int6 quantization achieves 5x faster performance than llama.cpp on cloud CPUs, and that its GPU matrix multiplication beats NVIDIA's cuBLAS by up to 30%. Mojo enables developers to write for loops and tokenizers (e.g., for LLaMA 3) in Python-like syntax without dropping to C++ or Rust. MAX is free and available now for CPU inference; GPU support launches in September with early access via Discord.

The Making of Devin by Cognition AI: Scott Wu

Jul 11, 2024 · 20:04

Scott Wu, co-founder and CEO of Cognition AI, demonstrates Devin, a fully autonomous AI software engineer that builds and deploys working applications from plain English instructions, sharing the early lessons and vision behind the product. Devin built a name game website for the AI Engineer World's Fair in minutes, iterating on feedback to hide names, add a streak counter, and deploy the final version. Cognition uses Devin internally for features like a search bar and dashboards; Wu notes that engineers typically spend 80–90% of time on implementation, but Devin frees them to focus on architecture and problem-solving, making them 5–10x more productive while increasing overall demand for software. He also highlights the UX challenges of agents, which require new interactions beyond text completion, and notes that access to Devin is expanding via a waitlist.

The Code AI Maturity Model and What It Means For You: Ado Kukic

Feb 13, 2024 · 7:58

Ado Kukic, Director of DevRel at Sourcegraph, presents the Code AI Maturity Model, a framework comparing AI coding assistant autonomy to SAE vehicle levels. He argues that 92% of developers now use Code AI tools, up from 1% a year ago, and predicts 99% of code will be written by AI in 5 years. The model defines six levels: Level 0 (manual coding), Level 1 (AI-generated snippets from developer intent), Level 2 (codebase-aware completions, e.g., suggesting Axios in a Node.js project), Level 3 (AI builds full features from high-level requirements, like adding authentication), Level 4 (AI proactively handles tasks, such as auto-fixing bugs via PRs), and Level 5 (AI handles the entire software lifecycle with minimal human oversight). The talk contextualizes each level with autonomous driving parallels, emphasizing that the industry is currently transitioning between Levels 1–3, with Level 5 representing full AI-led code generation.

How to Become an AI Engineer from a Fullstack Background - Reid Mayo

Feb 2, 2024 · 10:19

Reid Mayo presents a step-by-step syllabus to transition from fullstack engineer to AI engineer, covering generative AI foundations, prompt engineering, LangChain, fine-tuning, and cost-effective open-source model deployment. The syllabus starts with Cohere's LLM overview, then dives into prompt engineering via Elvis Seravia's guide and Learn Prompting org docs. It emphasizes LangChain as the glue layer for modular AI systems, with tutorials from Mayo Ocean. Evals are treated as software tests using OpenAI's cookbook. Fine-tuning is taught via OpenAI's cookbook and then open-source LLaMA 2, with a specific case study showing a $19 fine-tuned LLaMA 2 matching OpenAI's $24,000 model on a target task. The boot camp ends with advanced deep learning courses from FastAI and Hugging Face for further mastery.

GPT Web App Generator - 10,000 apps created in a month: Matija Sosic

Jan 26, 2024 · 9:23

Matija Sosic, CEO and co-founder of Wasp, demonstrates how his team built mage, a GPT-powered full-stack web app generator that produced over 10,000 applications in one month. Mage specializes in generating React, Node.js, Prisma, and Wasp apps from plain English descriptions, thanks to three key factors: specialization in full-stack web apps, leveraging Wasp's high-level declarative configuration to reduce boilerplate and errors, and an automated error-fixing phase. A typical app uses 25,000 to 60,000 tokens and costs 10 to 20 cents by combining GPT-3.5 for implementation and GPT-4 for planning, versus $1–$2 with GPT-4 alone. Mage serves as a highly customized cross-starter, with current limitations around lack of interactive debugging, which is planned next. Future directions include live debugging and potentially fine-tuning an LLM for Wasp.

Storyteller: Building Multi-modal Apps with TS & ModelFusion - Lars Grammel, PhD

Jan 23, 2024 · 7:30

Lars Grammel presents Storyteller, a kid's story generator built with TypeScript and his ModelFusion library, which orchestrates multiple AI models to turn voice input into narrated audio stories with images. The app transcribes speech via OpenAI Whisper in 1.5 seconds, generates a story outline with GPT-3 TurboInstruct in 4 seconds, and then runs title, image, and audio generation in parallel. For consistency, it uses GPT-4 with a low temperature to produce structured story passages, and streams partial results so narration can begin before the full story is ready—addressing the 1.5-minute full generation time. Voices are selected by embedding speaker descriptions and retrieving matching ElevenLabs voices filtered by gender. Image prompts are extracted from the story via GPT-4 and rendered with Stability AI's Stable Diffusion XL. The result is a responsive, multi-modal app that lets users listen while the server continues processing.

Open Questions for AI Engineering: Simon Willison

Nov 25, 2023 · 24:33

Simon Willison recaps the AI industry's past year—from ChatGPT's breakthrough to open-source local models—and poses key open questions for AI engineering. He argues that ChatGPT's chat interface, while popular, is a poor fit for advanced use, urging better UIs like his command-line tool LLM. He celebrates Meta's Llama release as a 'stable diffusion moment' for language models and highlights the rise of small, locally-run models such as Replit's 3B model, asking how small models can remain useful. On security, he warns that prompt injection remains unsolved after 13 months, limiting what can safely be built. He champions ChatGPT's Code Interpreter (which he dubs 'Coding Intern') as the most exciting tool, able to write and compile C code on a phone, and argues that LLMs flatten the learning curve, making programming accessible to more people. He concludes by urging the community to build tools that enable anyone to automate tedious tasks.

The Weekend AI Engineer: Hassan El Mghari

Nov 22, 2023 · 21:49

Hassan El Mghari shares how he built viral AI apps like RoomGPT and AI Commit, arguing that simple off-the-shelf APIs and focused weekend sprints can attract millions. He describes building 11 side projects in a year, which grew from 20,000 visitors to over 8.5 million unique visitors and 2.8 million sign-ups. Key projects include RoomGPT, which used ControlNet for room redesign and reached 6 million visitors, and AI Commit, an open-source CLI tool adopted by 30,000 developers. He emphasizes using tools like the Vercel AI SDK and v0.dev to accelerate development, making apps free and open source to drive growth, spending 80% of time on UI, and launching quickly with minimal fine-tuning.

Pragmatic AI with TypeChat: Daniel Rosenwasser

Nov 14, 2023 · 18:34

Daniel Rosenwasser, TypeScript program manager, introduces TypeChat, an experimental library that uses TypeScript type definitions to guide and validate unstructured LLM output into structured JSON for traditional apps. He demonstrates a coffee shop ordering system where types define the schema, showing how the library can handle ambiguous inputs like 'a purple gorilla' by including unknown text for recovery. TypeChat also generates programs as JSON using a fake language to safely script multi-step operations, avoiding sandboxing issues from real code. A Python prototype extends the same approach, with examples like CSV data manipulation via a class-based API. The library aims to make AI tools accessible to all engineers by leveraging types they already use.

Move Fast Break Nothing: Dedy Kredo

Nov 8, 2023 · 13:01

Dedy Kredo, co-founder and CPO of CodiumAI, argues that code generation needs a GAN-like architecture with a separate code integrity critic, and that behavior coverage is a more useful metric than code coverage. He demonstrates CodiumAI's IDE extensions (200,000 installs) on the open-source AutoScraper project (5,000 GitHub stars with no tests), automatically mapping behaviors, generating tests for happy paths and edge cases, and using reflection to auto-fix failures. The tool then enhances code with deep improvements (performance, security) and, via a PR assistant, catches a forgotten API key vulnerability. Kredo closes with a personal note about the Hamas attack on Israel, showing his co-founder on reserve duty while CodiumAI usage continues to rise.

The AI Evolution: Mario Rodriguez, GitHub

Nov 7, 2023 · 19:32

Mario Rodriguez, VP of Product at GitHub, recounts the history and future of GitHub Copilot, the first at-scale AI programmer. Copilot now serves over 20,000 organizations and 1 million-plus developers, generating over $100M in ARR, with 46% of code written via completions. He shares insider lessons: ghost text, low latency under 100 milliseconds, and prompt engineering were key to success. He warns that 'syntax is not software' and that global presence and offline/online scorecards are essential at scale. Looking ahead, Rodriguez envisions moving from procedures to goals and constraints, enabling AI to reason on code, and designing immersive UIs for human-AI collaboration. He concludes that GitHub has evolved from a version control system into an end-to-end platform infused with AI.

[Workshop] AI Engineering 101

Nov 6, 2023 · 3:02:24

In this hands-on workshop, AI engineer Noah Hein teaches the basics of AI engineering by building five projects with OpenAI's GPT-3, GPT-4, Dall-E, Whisper, and Telegram. Participants create a Telegram bot that uses GPT-3 for chat, implements retrieval-augmented generation (RAG) with embeddings and cosine similarity on MDN docs, generates code with GPT-4 using few-shot prompting, creates images via Dall-E 2, and transcribes voice with Whisper. The session explains core concepts like tokens, chunking, context windows, temperature, and top-p, emphasizing that embeddings and prompt engineering are key to performance. Hein demonstrates that these tools are cheap and accessible—the entire workshop costs about $0.05 in API fees—and shows how AI engineers can integrate multiple models into a single application.

Pydantic is all you need: Jason Liu

Nov 1, 2023 · 17:55

Jason Liu argues that Pydantic is the key to building reliable LLM applications by using structured prompting with OpenAI function calling. He introduces Instructor, a library that patches OpenAI's API to return Pydantic objects instead of raw JSON, ensuring type safety and validation. Liu demonstrates how Pydantic's 70 million downloads make it a trusted tool for defining data models with type hints, field validators, and even LLM-powered validators that catch errors like "don't say mean things" and retry via max retries. He shows concrete examples: decomposing user queries into structured search objects for RAG, generating a query plan DAG with parallel dependencies, extracting knowledge graphs for visualization, and verifying facts by requiring substring quotes from source text. Liu emphasizes that structuring prompts as code moves development from string manipulation to domain modeling, enabling cleaner, more maintainable systems that integrate easily with existing software.

Building AI For All: Amjad Masad & Michele Catasta

Oct 23, 2023 · 25:13

Amjad Masad and Michele Catasta announce Replit's AI for all, giving free AI-powered coding to millions of users. They detail the training of Replit Code v1.5, a 3.3B parameter model trained on 1 trillion tokens of code, which outperforms StarCoder 3B and approaches Code LLaMA 7B in Human Eval despite being half the size. The model is optimized for low-latency inference, generating over 200 tokens per second per GPU. Catasta explains the data pipeline using permissively licensed code from The Stack and Replit public repos, with five epochs of repeated high-quality data. The model is released open-source with a commercially permissive license. Partnerships with Glaive AI for instruction fine-tuning, Morph Labs for a novel fill-in-syntax-tree format, and Perplexity for fast model serving are also unveiled.

Principles for Prompt Engineering - Karina Nguyen (Claude Instant @ Anthropic)

Oct 20, 2023 · 55:31

Karina Nguyen, an engineer on Anthropic's Claude, presents principles for effective prompt engineering, treating it as a creative writing process requiring iteration. She advises using XML tags, placing instructions at the end of long prompts (improving accuracy), and decomposing questions to improve faithfulness over chain-of-thought. She covers reducing hallucinations by asking Claude to hedge or quote sources, and using self-consistency and contrastive examples for labeling. Nguyen explains generating evaluation datasets with Claude by splitting documents into multiple-choice questions, and notes that for long-document QA, asking at the end outperforms the beginning. She distinguishes Claude (larger, smarter) from Claude Instant (cheaper, faster, better at math and code), and predicts prompt engineering will remain essential for complex tasks.