Google Sheets

The Future Is Domain-Specific Agents - Justin Schroeder, StandardAgents

Jun 29, 2026 · 30:38

Justin Schroeder of StandardAgents argues that domain-specific agents—small, focused AI agents each limited to a single domain—will outperform general-purpose agents that rely on accumulating tools and context via inheritance, and that composition of such agents is the key to building practical, cost-effective AI systems. He defines an agent as deterministic software harnessing non-deterministic model outputs, then critiques the current approach of piling tools and skills into a single agent (inheritance), proposing instead a coordinator agent that delegates to many specialized sub-agents (composition), each with its own minimal context, system prompt, and tools. He claims these domain-specific agents achieve over 80% token efficiency, enable use of cheaper small models (e.g., DeepSeek V4 Flash is 137x cheaper than a frontier model), improve safety by limiting capabilities, and scale easily. Schroeder predicts rapid adoption through 2026-2027, with multi-agent orchestration becoming mainstream and tokens no longer getting cheaper—up 29% when adjusted for IQ. He describes an ideal agent architecture with hooks, rules, and recursive sub-agents, and invites listeners to try…

Agents for Everything Else — swyx

May 1, 2026 · 14:10

swyx, co-founder of AI Engineer Conferences, details how his team of nine full‑time people uses coding agents like Devin and Town to run the conference, moving from a non‑AI stack to agents that handle everything from Figma‑to‑website conversion and schedule management to buying a lobster for the event. He claims these agents save “yak shaving” and increase human productivity because employees enjoy faster feedback cycles, leading to more work like animations and polish. The team now manages a 1,000‑person conference with plans to handle 6,000 in San Francisco without growing headcount. Swyx also advocates for “agents for everything else,” predicting that the primary user of software will shift from humans to bots, making APIs and MCPs more important than dashboards. He urges attendees to adopt agents and “prescribe” them to skeptical coworkers.

[Full Workshop] Building Metrics that actually work — David Karam, Pi Labs (fmr Google Search)

Jul 29, 2025 · 40:28

Achim and David Karam, former Google Search leaders now at Pi Labs, argue that building reliable AI evaluations requires a multi-signal scoring system rather than single LLM-as-judge metrics. They walk through designing custom metrics starting with simple correlated signals, using a copilot to generate and iterate on dimensions like 'includes insights' or 'title length,' then calibrating them against ground truth thumbs up/down data from users. The workshop demonstrates applying this to a meeting summarizer, integrating the scoring system into Google Sheets to score 120 examples and produce a confusion matrix showing alignment with human feedback. They also show how to use the same scoring system for model comparison (e.g., GPT-1.5 vs 2.5) and online best-of-N sampling, which boosts response quality by generating multiple responses and selecting the highest-scored one.

Shipping Products When You Don't Know What they Can Do — Ben Stein, Teammates

Jul 28, 2025 · 19:34

Ben Stein, founder of Teammates, argues that product management for AI agents requires abandoning traditional specs and embracing uncertainty. He describes how his team discovered emergent behavior when a customer asked to tag an agent in a Google Doc comment—an unplanned feature that worked. Stein advocates thinking in affordances rather than features, using evals as the new spec, and vibe coding to feel interactions before committing. He notes that bugs become ambiguous when behavior is probabilistic, and customer meetings shift from selling vision to co-inventing the future. The episode offers a field guide for product teams navigating boundless surface areas built on unpredictable LLMs.

How to build world-class AI products — Sarah Sachs (AI lead @ Notion) & Carlos Esteban (Braintrust)

Jun 27, 2025 · 1:43:46

Sarah Sachs (AI lead at Notion) and Carlos Esteban (Braintrust) explain that building great AI products requires 10% prompting and 90% evals and observability, with Notion AI using Braintrust to iterate on prompts and models. Sachs details their cycle: curate small datasets, tie them to scoring functions (LLM-as-a-judge with per-sample prompts and heuristic checks), run evals before shipping, and use production logs to catch regressions. She notes Notion AI supports 100M+ users, switches models in under a day, and 60% of enterprise users are non-English—requiring multilingual eval rigor. Carlos and Doug then walk through Braintrust's framework: tasks (prompts, tools, agents), datasets, and scores (0-1). They demonstrate offline evals via playground and SDK, online scoring in production, user feedback capture, human review setup, and remote evals to bridge complex code with the playground. The workshop covers moving from pre-prod evals to production monitoring, closing the feedback loop by adding underperforming spans to datasets.

Building AI Agents that actually automate Knowledge Work - Jerry Liu, LlamaIndex

Jun 24, 2025 · 17:57

Jerry Liu, CEO of LlamaIndex, argues that building AI agents which actually automate knowledge work requires a combination of really good tools and carefully tailored agent reasoning, moving beyond naive RAG to a 'document toolbox' that includes parsing, extraction, indexing, and manipulation of complex unstructured data like PDFs, Excel, and PowerPoints. He introduces two main agent categories: assistive agents (chat-based, human-in-the-loop, unconstrained reasoning) and automation agents (batch-processing, constrained workflows, less human oversight). A key new capability is an Excel agent that uses reinforcement learning to learn a semantic map of unnormalized spreadsheets, achieving 95% accuracy on data transformation, surpassing the 75% baseline of LLMs with code interpreter and even human baselines of 90%. Real-world examples include financial due diligence combining automation (inhaling financial data) and assistant interfaces (analyst co-pilot), enterprise search with specialized agentic RAG, and technical data sheet injection for a global electronics company that reduced weeks of manual work to an automated extraction pipeline. Liu emphasizes that the right preprocessing…

Beyond APIs: How AI Web Agents Are Automating the "Long Tail" of Knowledge Work

Feb 22, 2025 · 17:44

Arjun and Bhavani present Rtrvr.ai, a universal AI web agent using a text-based approach to autonomously perform tasks across multiple browser tabs at under a penny per page. They argue text-based reduces hallucination compared to vision-based agents like OpenAI's operator, and being a Chrome extension avoids password sharing and accesses logged-in content. Features include deep research navigating multiple pages, dynamic function calling for any third-party API, and graph generation. Rtrvr extracts structured data, performs actions like clicking dropdowns, and processes tabs simultaneously, automating the long tail of knowledge work including market research, LinkedIn automation, and WhatsApp messaging. They claim a distributed subtask approach lowers failure rates and envision collaborative dataset construction, e.g., aggregating local government events.

The Age of the Agent: Flo Crivello

Oct 24, 2023 · 14:49

Flo Crivello, founder and CEO of Lindy, argues that AI agents will democratize business impact, enabling a 25-year-old to match the Coca-Cola company. He draws a parallel to media, where YouTube removed gatekeepers and created outliers like MrBeast and Ryan's World. Lindy, an AI personal assistant, automates calendar, email, and meeting prep—saving the average US manager 15 hours weekly, worth $459B annually. A story reveals Lindy's hallucinations: it invented the nonexistent word "periquity." Crivello envisions a "society of Lindys" collaborating, citing Lindy's own agent team that builds integrations with Slack and Twilio. He concludes that the only future skill will be wielding AI leverage, as AI CMOs, designers, and engineers will work for anyone.