Raindrop

Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop

May 7, 2026 · 50:25

Zubin Koticha and Danny Gollapalli of Raindrop argue that agent observability must shift from evals to production monitoring because agent failures are non-deterministic and unbounded. They break down explicit signals (tool error rate, latency, cost) and implicit signals (user frustration, refusals, task failure) detected by trained classifiers and regex, emphasizing that aggregate patterns even from imperfect regex are valuable. Experiments let teams ship changes to a percentage of users and compare semantic signal rates, with statistical relevance often reached after a few hundred events. Self-diagnostics—a simple tool and system prompt—enable agents to report their own failures, capability gaps, and even self-correction behavior, as demonstrated in a live coding agent demo where a disabled write tool caused the agent to use bash and then report the bypass. The episode covers alerting, trace visualization, data export to BigQuery/Snowflake, and the challenge of managing fast-paced experimentation at scale.

Building AI Products That Actually Work — Ben Hylak (Raindrop), Sid Bendre (Oleve)

Jul 24, 2025 · 18:42

Ben Hylak (Raindrop) and Sid Bendre (Oleve) argue that building reliable AI products requires iterative real-world signals over traditional evals. Ben debunks eval myths—evals don't measure product quality, LLM-as-judge fails, and production evals are costly—and stresses tracking explicit signals (thumbs up/down, copy rate) and implicit signals (refusals, frustration) to identify issues. Sid introduces Trellis, a framework for scaling viral AI apps that uses discretization—breaking infinite output into intent buckets—prioritization by volume times negative sentiment times achievable delta, and recursive refinement. Starting with an MVP, teams classify user intents, convert them into semi-deterministic workflows, then repeatedly drill into sub-intents to engineer repeatable, attributable magic. Oleve's approach, powered by Raindrop, has scaled six viral products to $6M ARR profitably with four people.