Redis

Why Your AI UX Is Broken (and It's Not the Model's Fault) — Mike Christensen, Ably

May 17, 2026 · 18:38

Mike Christensen, a staff engineer at Ably, argues that direct HTTP streaming (SSE) for AI chat apps breaks because it ties a response stream to a single connection, making resumability, multi-device sync, and live control mutually exclusive. He introduces durable sessions—a persistent, shared resource decoupled from any individual client or agent—built on Ably's Pub/Sub channels, which automatically handle reconnection, cross-tab synchronization, and concurrent multi-agent activity without complex plumbing. Christensen demonstrates with a live demo: a forced network disconnect that self-recovers, two tabs in perfect sync, two agents running in parallel without an orchestrator, and a handoff to a human agent who joins mid-conversation with full history. The episode concludes that treating the session as a durable shared resource unlocks resilient, multi-surface, and live-controllable AI experiences that the standard request-response model cannot support.

Scaling GitHub for your Agents — Sam Morrow, GitHub

Apr 27, 2026 · 20:35

Sam Morrow, GitHub's MCP server lead, details the architectural challenges and solutions for scaling a remote MCP server to 7 million weekly tool calls. He explains how tool proliferation degraded agent performance—LangChain's research confirmed more tools confuse agents—leading to innovations like tool sets and dynamic discovery, though 100% of users stuck with defaults. To reduce context, GitHub cut tool descriptions by 49% and trimmed output tokens by 75% on list pull requests. Security is addressed via OAuth 2.1 with PKCE and step-up auth; they rejected dynamic client registration to avoid unbounded app databases and rate-limiting issues. The stateless server uses Redis for session storage and builds a fresh server instance per request, enabling horizontal scaling without session affinity. Metrics include 11 million Docker downloads, 30,000 stars, 4,000 forks, and 126 contributors. Morrow predicts compositional tools and automatic server discovery will make thousands of tools the norm.

Building durable Agents with Workflow DevKit & AI SDK - Peter Wielander, Vercel

Jan 6, 2026 · 1:09:49

Peter Wielander from Vercel introduces the Workflow DevKit, an open-source library that adds durability, observability, and resumability to AI agents by wrapping them in a workflow pattern. He demonstrates converting a coding agent into a workflow-supported agent, showing how steps like LLM calls and tool executions become isolated, retryable units. The toolkit enables long-running agents that can sleep for days, resume streams after disconnection, and integrate human-in-the-loop via webhooks. Deployable on Vercel or any cloud, it provides built-in observability through a local UI. The Workflow DevKit is currently in beta with general availability targeted for January.

Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing - Neil Dwyer, Gabber

Jul 31, 2025 · 16:09

Neil Dwyer, CTO of Gabber, details how his startup serves real-time voice AI for under $1 per hour using open-source Orpheus TTS, LoRAs for emotive voice cloning, and vLLM with FP8 dynamic quantization to achieve 95-105 tokens per second on L40S GPUs. He explains the critical latency challenge of 'head of line silence' in Orpheus (600ms in default voices) and how fine-tuning LoRAs reduces it to ~100ms, fitting within a 1.5-second budget for real-time conversation. Gabber batches multiple LoRA generations per GPU and uses a consistent hash ring for load balancing across servers, enabling popular clones to be dynamically replicated. The talk argues that with current open-source tools, building affordable consumer voice AI is accessible to small teams.

Information Retrieval from the Ground Up - Philipp Krenn, Elastic

Jul 27, 2025 · 1:48:07

Philipp Krenn of Elastic demonstrates that vector search is only a feature of information retrieval, not its foundation, in this hands-on workshop on retrieval for RAG. He contrasts classic keyword search—using tokenization, stemming, stopword removal, and BM25 scoring with the inverted index—with sparse (Elser/Splade) and dense (OpenAI text-embedding-small) vector embeddings. Krenn explains why hybrid search, combining lexical and semantic methods via reciprocal rank fusion (RRF), outperforms any single approach, especially for brand or exact-match queries where keyword search remains superior. He also covers scoring normalization, chunking strategies, and the new `text_similarity_reranker` retriever, showing how Elasticsearch's retrievers API enables two-stage retrieval with a cross-encoder for re-ranking. The session emphasizes that retrieval quality depends on evaluating against a golden dataset or LLM-as-judge, and that 'it depends' is the correct answer for choosing between PgVector and dedicated search engines.

Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Jul 19, 2025 · 19:03

Keegan McCallum, Head of ML Infrastructure at Luma AI, details how the company's Dream Machine model scaled from 500 to 9,000 H100 GPUs within hours to handle 1 million users in four days, outpacing ChatGPT's initial growth. He explains that their initial Triton inference server setup was brittle and ill-suited for multi-GPU, multi-node video models, prompting a re-architecture to a custom serving stack on vanilla PyTorch. To solve work starvation across user tiers, they implemented an SLO-based aging system that ranks jobs by the percentage of their worst-case wait time elapsed. For managing dozens of model versions, they store immutable full Python environments and checkpoints in object storage, with a YAML file controlling active deployments and enabling zero-downtime rollouts across thousands of GPUs. McCallum also discusses partnerships with Nvidia, AMD, and Grok, and how Luma's broader mission is to build general multimodal intelligence that generates, understands, and operates in the physical world.

Memory Masterclass: Make Your AI Agents Remember What They Do! — Mark Bain, AIUS

Jun 27, 2025 · 51:25

Mark Bain, Vasilia Markovits, Alex Gilmore, and Daniel Chalev demonstrate that AI memory requires causal relationships and graph databases, not just vector similarity, to solve hallucinations and enable agentic workflows. Bain argues that memory is any data affecting change, and that attention, diffusion, and VAEs follow the same geometric principles as gravity and entropy. Alex shows Neo4j's MCP server storing semantic memory as entities and relationships retrieved across conversations. Vasilia demos Cognee building semantic graphs from GitHub data for agentic hiring decisions. Daniel presents Graphiti's domain-aware memory using custom Pydantic schemas to filter irrelevant facts. Bain introduces a GraphRAG chat arena that switches between memory solutions on a single Neo4j graph, testing different implementations for episodic and temporal recall.

How agents broke app-level infrastructure - Evan Boyle

Jun 3, 2025 · 13:32

Evan Boyle, founder of GenSX, argues that LLM-powered agents have broken assumptions about app-level infrastructure, as p1 latency jumps from milliseconds to seconds and workflows run minutes to hours. He explains that traditional serverless providers timeout after 5 minutes, lack native streaming, and force brittle Rube Goldberg machines on top of queues like SQS or tools like Airflow. Boyle presents an open-source library that separates API and compute layers using Redis streams for resumability, enabling users to refresh pages or navigate away without losing progress. He demonstrates components with built-in retries, caching, and tracing, and a workflow that automates Hacker News analysis. Key architectural lessons include starting simple but planning for long-running agents, keeping compute and API planes separate, and leaning on Redis for resumability.

AI Engineering 201: The Rest of the Owl

Nov 8, 2023 · 56:57

Charles Frye, instructor of the Full Stack LLM Bootcamp, presents essential patterns for building language user interfaces (LUIs), arguing that while RAG chatbots are the 'to-do list app' of AI engineering, structured outputs via function calling (e.g., OpenAI's JSON schema, Instructor library) improve robustness, and agents with memory (like generative agents or Voyager in Minecraft) represent the true AI frontier. He emphasizes the need for hybrid search combining vector and keyword retrieval (citing Vespa, Postgres, Redis), and warns that monitoring and evaluation are the hardest engineering challenges: monitoring user behavior, latency quantiles (especially 99th percentile), and costs must be paired with observability tools like Honeycomb or Gantry, while evaluation often requires iterated decomposition or using LLMs as evaluators (GPT-4 as 90th percentile crowd worker). The episode concludes that shipping to learn — starting with production data to generate tests — is the dominant engineering mindset, and that the field is still filling in the gaps between inference and full product value.