GPT-3.5

Recsys Keynote: Improving Recommendation Systems & Search in the Age of LLMs - Eugene Yan, Amazon

Jul 16, 2025 · 20:54

Eugene Yan's keynote presents three innovations for recommendation systems: Semantic IDs, LLM-augmented data, and unified models. Kuaishou’s trainable multimodal Semantic IDs increased cold-start coverage by 3.6% and velocity by 3.5% by clustering content embeddings. Indeed used GPT-4 fine-tuning and distillation to filter bad job recommendations, reducing bad recs by 20% while boosting application rate 4% and cutting unsubscribes 5%. Spotify’s LLM-generated exploratory search queries drove a 9% increase in exploratory queries for new categories like podcasts. Netflix’s Unicorn unified ranker matched or exceeded specialized models across search and recommendations, while Etsy’s unified embeddings with a quality vector achieved a 2.6% sitewide conversion lift and 5% more search purchases.

The Coherence Trap: Why LLMs Feel Smart (But Aren’t Thinking) - Travis Frisinger

Jun 3, 2025 · 20:47

Travis Frisinger argues that large language models are not intelligent but coherent, introducing 'coherence reconstruction' as a mental model for understanding their utility. He explains that hallucinations are a feature, not a bug, as models fill gaps predictably to maintain pattern completion. Prompts act as force vectors navigating latent space, and the key to reliable outputs is a 'frame, generate, judge, iterate' loop. Frisinger presents a three-layer model: latent space, execution layer (tools and RAG), and conversational interface. He advises engineers to design for emergence rather than control, using dense context as anchors to steer generation, and to watch for breakdowns in tone as early signs of lost coherence.

Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier - Andrew Thomspson

Jun 3, 2025 · 25:09

Andrew Thompson, CTO of Orbital, introduces the concept of Prompt-Tax: the hidden cost of migrating prompts when upgrading AI models. He shares how his agentic product automates real estate due diligence, growing from <1B to 20B monthly tokens and zero to multiple seven-figure ARR over 18 months. Key tactics include optimizing for prompting over fine-tuning, using domain experts (ex-lawyers) to write prompts, and relying on vibes over formal evals. Thompson advocates 'betting on the model'—shipping new frontier models immediately and fixing regressions on the fly using progressive rollouts and rapid feedback loops. A clip from Demis Hassabis underscores the unique challenge of evolving tech stacks. The episode concludes with questions on whether evals or progressive delivery scale to manage Prompt-Tax.

[Full Workshop from Microsoft] Github Copilot - The World's Most Widely Adopted AI Developer Tool

Feb 7, 2025 · 1:19:45

This workshop from the AI Engineer World's Fair features GitHub's Christina Warren, Dave, Alex, and Harald presenting GitHub Copilot, the world's most widely adopted AI developer tool. They demonstrate Copilot's three interaction modes: ghost-text completions using GPT-3.5 for speed, inline chat (also GPT-3.5) for quick code edits, and the chat panel powered by GPT-4 Turbo for deeper conversations. Speakers emphasize prompt crafting—being specific, providing examples, and keeping relevant files open—to improve results. They show how to use slash commands (/fix, /explain, /test) and the new attach button for explicit context. The workshop uses GitHub Codespaces preconfigured with Python 3.11 and Copilot extensions; a coupon provides a 7-day free trial. Harald explains the trade-off between model quality and latency, and notes upcoming features like automatic agent delegation and workspace integration for cross-file edits.

No-code fine-tuning: Mark Hennings

Feb 5, 2025 · 9:27

Mark Hennings, creator of Entrypoint, argues that fine-tuning large language models is now accessible without code, offering faster and cheaper alternatives to prompt engineering: GPT-3.5 fine-tuned runs at 73ms per token vs GPT-4's 196ms, saving 88.6% in cost and cutting prompts by 90%. He explains that fine-tuning reduces prompt injection risks, enables team collaboration via training data, and only requires 20 examples to start. Hennings demonstrates Entrypoint's no-code UI that lets users import CSV data, structure fields with templating, and fine-tune GPT-3.5 Turbo, then iteratively improve models by feeding production feedback back into the dataset. He proposes a dev lifecycle: prototype with prompt engineering, use it to build a dataset, fine-tune, evaluate, deploy, and continuously refine.

Enhancing Quality and Security in CI: Gunjan Patel

Nov 27, 2024 · 18:27

Gunjan Patel, Director of Engineering at Palo Alto Networks, presents Ghost Pilot, an AI-powered CI pipeline that outsources boring software development tasks to enable self-evolving code. Unlike real-time Copilots, Ghost Pilot operates as a 'slow system' in CI, iteratively improving variable names and code comments, then generating unit tests by first listing edge cases and personalizing them via team context files. For security, it simulates three AI roles—Red Team engineer, developer, and engineering manager—who debate identified vulnerabilities, prioritize fixes by risk and effort, and propose changes with citations. Patel shares a reusable CI template with a Bring-Your-Own-LLM option, demonstrated by catching a logical Kubernetes bug missed by static analysis tools.

Iterating on LLM apps at scale Learnings from Discord: Ian Webster

Nov 22, 2024 · 18:26

Ian Webster, Senior Staff Engineer at Discord and maintainer of Promptfoo, shares how Discord built and scaled Klyde AI, a chatbot for 200 million users, focusing on evaluation and safety. He argues that evals should be treated as simple, deterministic unit tests that run locally, avoiding complex metrics, and that breaking the system into small, testable pieces (e.g., checking for lowercase output to enforce casual tone) achieves 80% of the goal with 1% of the work. Webster details how Discord mitigated risks like the 'grandma jailbreak' (which originated on Discord) by using an attacker LLM to generate adversarial inputs and a judge to refine them, exposing cracks in safeguards. He advocates for pre-deployment red teaming over live filtering, and describes using Promptfoo for risk assessment across brand, legal, and safety categories. The episode also covers prompt management via Git and Retool, routing with occasional GPT-4 responses to correct model drift, and the challenge of closing the feedback loop due to privacy constraints, relying instead on dogfooding and public examples.

Navigating RAG Optimization with an Evaluation Driven Compass: Atita Arora and Deanna Emery

Nov 12, 2024 · 18:14

Atita Arora (Qdrant) and Deanna Emery (Quotient AI) demonstrate how evaluation-driven optimization improves Retrieval Augmented Generation (RAG) systems, using Qdrant's vector database and Quotient's evaluation platform. They detail a 10-experiment pipeline on Qdrant documentation, starting with naive RAG and incrementally testing chunk sizes, embedding models, LLMs (Mistral, GPT-3.5), re-rankers (MixedBread, Cohere, ColBERT), and hybrid search. Key metrics—faithfulness (hallucination reduction), context relevance, and chunk relevance—guide decisions: increasing chunk size hurt faithfulness (0.76 to 0.72), while smaller chunks with larger retrieval windows improved it. Switching to GPT-3.5 lifted all metrics, but poor context relevance revealed retrieval issues. Adding Cohere re-ranking boosted faithfulness to 0.82, and hybrid search with re-ranking achieved 0.85, proving domain-specific terminology demands tailored search strategies. The episode stresses avoiding over-engineering without metrics, keeping evaluation datasets current, and using domain understanding for systematic RAG improvement.

What It Actually Takes to Deploy GenAI Applications to Enterprises: Arjun Bansal and Trey Doig

Nov 4, 2024 · 21:30

Trey Doig of Echo AI and Arjun Bansal of Log10 recount Echo AI's journey deploying a GenAI-native conversational intelligence platform for billion-dollar retail brands, focusing on the centrality of accuracy. Echo AI ingests all customer conversations, uses LLMs to surface insights at 100% coverage, but must overcome enterprise trust issues by achieving 95% accuracy within seven days. The platform relies on Log10's auto feedback system, which uses AI-based review to match human accuracy with model speed, yielding a 20 F1 point improvement in one use case. The episode details how Echo AI's solution engineers use Log10 to grade summarizations, catch hallucinations, and track model drift, turning human feedback into curated datasets for fine-tuning. Ultimately, the partnership demonstrates a path to self-improving LLM applications through iterative accuracy measurement and improvement.

Building and Scaling an AI Agent Swarm of low latency real time voice bots: Damien Murphy

Oct 8, 2024 · 1:07:23

Damien Murphy, Senior Applied Engineer at Deepgram, demonstrates building and scaling low-latency real-time voice bots using Deepgram's new voice agent API, which wraps speech-to-text, LLM, and text-to-speech into a single streaming endpoint. He shows a drive-thru ordering demo with function calling (add/remove items) using GPT-4o, achieving sub-second latency by co-locating components. Murphy explains scaling to millions of concurrent calls through regional Kubernetes clusters and multi-agent swarms (routing, booking, support agents) to reduce complexity and cost. He addresses endpointing challenges, VAD-based barge-in, and cost/quality trade-offs with hosted vs. self-hosted models, noting Deepgram offers 20x cheaper TTS than ElevenLabs and 50ms STT latency self-hosted. The talk emphasizes keeping agents simple, using smallest capable LLMs, and composability for reuse.

Pydantic is STILL all you need: Jason Liu

Sep 6, 2024 · 15:21

Jason Liu returns to the AI Engineer World's Fair to argue that Pydantic (and his library Instructor) is still all you need for structured output with LLMs. He shows that the core API—response_model, streaming with iterables, and partials for real-time validation—remains unchanged, now supporting Ollama, LlamaCPP, Anthropic, Gemini, and more. Liu demonstrates validators that enforce rules like uppercasing names or verifying receipt totals, reducing errors with automatic retries. He applies structured output to RAG: using a Search model with optional date ranges and source selection, and a Response model with follow-up questions and validated URLs. For extraction, he creates classifiers using Literal types, meeting summaries with action items, and even tables as Pandas DataFrames via custom type hints. Liu’s key takeaway is that one retry often suffices, and as models get faster and smarter, structured output makes LLMs compatible with classical programming—turning generative AI into generating data structures defined by the developer.

The Weekend AI Engineer: Hassan El Mghari

Nov 22, 2023 · 21:49

Hassan El Mghari shares how he built viral AI apps like RoomGPT and AI Commit, arguing that simple off-the-shelf APIs and focused weekend sprints can attract millions. He describes building 11 side projects in a year, which grew from 20,000 visitors to over 8.5 million unique visitors and 2.8 million sign-ups. Key projects include RoomGPT, which used ControlNet for room redesign and reached 6 million visitors, and AI Commit, an open-source CLI tool adopted by 30,000 developers. He emphasizes using tools like the Vercel AI SDK and v0.dev to accelerate development, making apps free and open source to drive growth, spending 80% of time on UI, and launching quickly with minimal fine-tuning.

AI Engineering 201: The Rest of the Owl

Nov 8, 2023 · 56:57

Charles Frye, instructor of the Full Stack LLM Bootcamp, presents essential patterns for building language user interfaces (LUIs), arguing that while RAG chatbots are the 'to-do list app' of AI engineering, structured outputs via function calling (e.g., OpenAI's JSON schema, Instructor library) improve robustness, and agents with memory (like generative agents or Voyager in Minecraft) represent the true AI frontier. He emphasizes the need for hybrid search combining vector and keyword retrieval (citing Vespa, Postgres, Redis), and warns that monitoring and evaluation are the hardest engineering challenges: monitoring user behavior, latency quantiles (especially 99th percentile), and costs must be paired with observability tools like Honeycomb or Gantry, while evaluation often requires iterated decomposition or using LLMs as evaluators (GPT-4 as 90th percentile crowd worker). The episode concludes that shipping to learn — starting with production data to generate tests — is the dominant engineering mindset, and that the field is still filling in the gaps between inference and full product value.