BigQuery

Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop

May 7, 2026 · 50:25

Zubin Koticha and Danny Gollapalli of Raindrop argue that agent observability must shift from evals to production monitoring because agent failures are non-deterministic and unbounded. They break down explicit signals (tool error rate, latency, cost) and implicit signals (user frustration, refusals, task failure) detected by trained classifiers and regex, emphasizing that aggregate patterns even from imperfect regex are valuable. Experiments let teams ship changes to a percentage of users and compare semantic signal rates, with statistical relevance often reached after a few hundred events. Self-diagnostics—a simple tool and system prompt—enable agents to report their own failures, capability gaps, and even self-correction behavior, as demonstrated in a live coding agent demo where a disabled write tool caused the agent to use bash and then report the bypass. The episode covers alerting, trace visualization, data export to BigQuery/Snowflake, and the challenge of managing fast-paced experimentation at scale.

How we hacked YC Spring 2025 batch’s AI agents — Rene Brandel, Casco

Jul 30, 2025 · 17:33

Rene Brandel, CEO of Casco, explains how his team hacked seven of sixteen YC Spring 2025 batch AI agents within 30 minutes each, revealing three critical security flaws: cross-user data access via IDOR, arbitrary code execution through code tools, and server-side request forgery (SSRF) from tool endpoints. They extracted personal data by exploiting missing authorization checks, overwrote security controls by writing malicious files into code sandboxes, and stole Git credentials by manipulating a database-schema tool. Brandel emphasizes that agent security extends beyond LLM prompt injection, advises treating agents as users with proper authentication and authorization, and warns against rolling custom code sandboxes, recommending out-of-the-box solutions like E2B. The talk concludes with a Q&A on extracting system prompts and the dangers of local coding agents.

MCP is all you need — Samuel Colvin, Pydantic

Jul 18, 2025 · 15:24

Samuel Colvin, creator of Pydantic, argues that MCP (Model Context Protocol) is all you need for agent-to-agent communication, simplifying what many are overcomplicating. He explains that MCP's tool calling primitive offers advantages over OpenAPI, such as dynamic tools, logging, and sampling—a mechanism for MCP servers to request LLM calls through the client, enabling nested agentic workflows. Colvin demonstrates a research agent using Pydantic AI that queries the BigQuery public PyPI dataset via an MCP server over stdio, showing how sampling allows the server to generate SQL itself while keeping the main agent's context lean. The demo logs progress to Logfire, an observability platform, and outputs human-readable results. He emphasizes that MCP’s standard I/O mode and extensibility make it ideal for autonomous agents beyond its original desktop coding use case.