Zapier

Leadership in AI Assisted Engineering – Justin Reock, DX (acq. Atlassian)

Dec 19, 2025 · 18:11

Justin Reock, Deputy CTO at DX (acquired by Atlassian), argues that AI's impact on engineering productivity varies wildly and that leaders must move beyond top-down mandates to focus on psychological safety, measurement of actual outcomes, and targeted integration across the SDLC. He presents data showing a 2.6% average increase in change confidence but extreme variability across companies, with some seeing 20% drops. Emphasizes that writing code is rarely the bottleneck; instead, leaders should identify and fix bottlenecks like context switching, citing Morgan Stanley's DevGenAI saving 300,000 hours annually by converting legacy code specs and Zapier reducing engineer onboarding to two weeks via AI agents. Introduces DX's AI Measurement Framework covering utilization, impact, and cost, and stresses trust-building through system prompt feedback loops and temperature settings. The episode delivers actionable guidance on measuring AI's true impact and enabling engineers through education, time to learn, and creative unblocking of usage.

Your Support Team Should Ship Code – Lisa Orr, Zapier

Dec 16, 2025 · 16:06

Lisa Orr of Zapier explains how the company empowers its support team to ship code using AI, tackling the app erosion caused by 8,000+ third-party integrations. Initially, they built an API playground that failed due to lack of workflow embedding, then pivoted to MCP tools in IDEs, but the slow Diagnosis tool led to an asynchronous agent. The resulting Scout Agent autonomously categorizes tickets, assesses fixability, generates merge requests, and enables rapid iteration in GitLab. Today, Scout drives 40% of support's integration fixes, doubling individual velocity from 1-2 to 3-4 tickets per week. Orr highlights support's three superpowers: closest to customer pain, real-time troubleshooting, and best at validation, with some team members becoming engineers.

A2A & MCP Workshop: Automating Business Processes with LLMs — Damien Murphy, Bench

Jul 26, 2025 · 1:23:14

Damien Murphy presents A2A and MCP protocols for building multi-agent systems that automate business processes from webhooks, using a host agent that delegates tasks to sub-agents (Slack, GitHub, Bench) via A2A, with each sub-agent using MCP tools from Zapier or internal APIs. He demonstrates processing a meeting transcript to create a GitHub issue, send a Slack message, and research attendees, emphasizing that A2A handles remote agent discovery and opaqueness while MCP provides standardized third-party tool access. Murphy highlights benefits like context isolation—sub-agents absorb large tool outputs and keep the host's context small—and parallel processing, but notes limitations: A2A's early stage, MCP's silent failures (e.g., Zapier Slack missing channels), the non-determinism of LLM orchestration, and the challenge of prompt caching costs. He argues that A2A is best for third-party agents where complexity is hidden, while MCP is useful for extensible tool integration, but if you control the tools or agents, direct function calls are simpler and more reliable.

Structuring a modern AI team — Denys Linkov, Wisedocs

Jul 24, 2025 · 17:40

Denys Linkov, who leads ML at Wisedocs, argues that building a modern AI team hinges on identifying your company's bottleneck—shipping features, acquiring users, or scalability—rather than reflexively hiring AI researchers. He introduces Ampere's Wager: trading your entire domain-savvy team for five top-lab researchers is usually a losing bet. For early-stage AI strategy, generalists who blend model training, serving, and business acumen outperform specialists; Linkov lived this in 2021 building a custom MLOps platform for a conversational AI startup and again in 2024 using advanced open-source tools for medical record processing. He stresses reskilling existing teams through weekly learning cadences and moving domain experts from giving feedback to writing evaluations. Hiring should hold context and act on it, verifying trends like 'don't hire juniors' against YC's AI school drawing 2,000 young people.

Monetizing AI — Alvaro Morales, Orb

Jul 23, 2025 · 18:18

Alvaro Morales, CEO of Orb, argues that traditional subscription-based SaaS pricing fails for AI products due to rapid cost fluctuations, margin pressures, and customer demand for clear ROI. He presents three frameworks: decide between direct monetization (add-on, bundled) or indirect (free to drive upsells) using examples like GitHub Copilot, Notion AI, and Expedia; select a value metric from resource-based tokens (Vercel's v0) to task-based (Zapier) to outcome-based pricing (Intercom's Fin at $0.99 per resolved ticket); and embrace continual experimentation, as AI pricing cycles shrink from years to months. To reduce pricing guesswork, Orb built Simulations, a tool that lets companies backtest alternate pricing strategies on real usage data from closed betas before launch, helping teams like Replit and Perplexity optimize revenue without risking customer trust.

The rise of the agentic economy on the shoulders of MCP — Jan Curn, Apify

Jul 18, 2025 · 18:08

Jan Curn, founder of Apify, argues that MCP (Model Context Protocol) enables a future agentic economy where AI agents autonomously discover and purchase tools from other agents or businesses (B2A/A2A). He explains that Apify's marketplace of 5,000 Actors (Docker-based tools) now integrates with MCP, allowing agents to dynamically discover and call any Actor via tool discovery — a key MCP feature. Curn demonstrates this with Claude Desktop, where an agent uses Apify's MCP server to find a venue, scrape Twitter, and even fill a form via a nested MCP server from Browserbase, all without prior configuration. He notes that Apify pays creators over $250,000 monthly, with total Actor revenue exceeding $1.5M/month, and that any developer can publish an Actor to monetize their tools instantly across the ecosystem. The talk closes with open questions about reliability, trust, and whether autonomous agent interaction can lead to AGI.

Turning Fails into Features: Zapier’s Hard-Won Eval Lessons — Rafal Willinski, Vitor Balocco, Zapier

Jun 30, 2025 · 16:15

Zapier AI Tech Lead Rafal Willinski and Staff Engineer Vitor Balocco explain how Zapier's evaluation system turns agent failures into targeted improvements through a data flywheel. They detail collecting explicit feedback at critical moments and mining implicit signals like testing behavior, cursing, and user follow-ups. The pair advocates building unit test evals for specific failure modes, then trajectory evals and LLM-as-judge with rubrics to avoid overfitting and capture multi-turn criteria. They share that over-indexing on unit tests hurt model benchmarking, and reasoning models can compare model runs, revealing differences like Claude as a decisive executor versus Gemini's yapping. Ultimately, they argue that the goal is user satisfaction, so A/B testing on a small traffic fraction is the ultimate verification.

Beyond Conversation: Why Documents Transform Natural Language into Code - Filip Kozera

Jun 10, 2025 · 10:57

Filip Kozera argues that chat-based interfaces like ChatGPT create ephemeral, polluted context windows and lack forced clarity, making them poor for specifying complex systems, while documents naturally force rigor and structured thinking. He advocates shifting from conversational brainstorming to document-driven workflows, where humans precisely articulate intent for AI agents. Kozera introduces background agents that run asynchronously, triggered by events like emails or meetings, and surface only when needing human approval, turning the human role into one of 'swiping left or right' to approve or edit outputs. He envisions a future where humans manage swarms of such agents, first adopted by prosumers and later by enterprises, with taste and intent becoming critical for imbuing personal brand into agent outputs. The episode presents a clear progression from chat-based brainstorming to structured document-driven agent systems.

How Zapier Builds AI Products and Features with the Help of Braintrust: Ankur Goyal & Olmo Maldonado

Nov 7, 2024 · 14:59

In this talk, Olmo Maldonado (Senior AI Engineer at Zapier) and Ankur Goyal (CEO of Braintrust) explain how Zapier uses Braintrust's evaluation and observability platform to build and improve AI features like AI Zap Builder and Copilot. They moved from 7 manual unit tests to over 800 automated evals, improving accuracy by nearly 300%. Braintrust's tracing capabilities let them dissect Copilot's multi-tool agent performance, leading them to adopt GPT-4 Turbo for the message router despite latency trade-offs. When switching to GPT-4.0 caused a regression below 80% scores, they diagnosed the issue via evals—GPT-4.0 was ignoring system prompts—and fixed it by relaxing prompt engineering and adjusting tool choices, recovering accuracy and cutting response time from 14 to 3 seconds. The episode details how eval-driven development and observability enable rapid iteration and reliable AI products at scale.