PromptFu

Case Study + Deep Dive: Telemedicine Support Agents with LangGraph/MCP - Dan Mason

Jun 22, 2025 · 1:56:13

Dan Mason of Stride presents a case study on building autonomous agents for telemedicine support, replacing a human-driven button-pushing workflow with LangGraph, Claude, MCP, and a Node.js/React/MongoDB stack, achieving roughly 10X capacity increase. The LLM-driven virtual operations associate ("Ava") assesses patient messages, updates state with anchors and scheduled messages, and passes proposals to an evaluator agent that scores confidence and complexity, escalating to humans below 75% confidence. Mason explains how treatment "blueprints" in Google Docs are read directly by the LLM instead of using RAG, enabling new treatments to be added without writing code. He details the eval system using LLM-as-a-judge via PromptFu and retry logic for tool-calling errors, and discusses trade-offs around confidence scoring, prompt caching, and model selection (Claude for steerability). The team includes two software engineers, one designer, and Mason, who wrote most LangGraph code with AI assistance (Kline).

Your Evals Are Meaningless (And Here’s How to Fix Them)

Feb 22, 2025 · 18:50

In this AI Engineering Summit talk, HoneyHive co-founder exposes why most LLM evaluations are meaningless due to criteria drift—where evaluator criteria misalign with user needs—and dataset drift, where test cases don't reflect real-world queries. He argues that static evaluation frameworks from tools like LangChain or Ffragas fail because they measure generic metrics rather than business-specific relevance, citing an e-commerce recommendation system that looked perfect in testing but broke in production. The fix is a three-step iterative alignment process: align evaluators with domain experts via continuous critique and few-shot examples; keep datasets alive by logging production underperformance and flowing those cases back into the test bank; and track alignment over time using F1 scores for binary judgments. Practical advice includes customizing the LLM evaluator prompt, starting with 20 domain expert examples in spreadsheets, and avoiding templated metrics. The speaker emphasizes that evals must evolve continuously, just like the LLM application itself, or they become meaningless.