Hex

CIAM for AI: Authn/Authz for Agents — Michael Grinich, CEO of WorkOS

Jul 21, 2025 · 20:13

Michael Grinich, CEO of WorkOS, argues that AI agents need first-class identity and access management, distinct from human or machine-to-machine auth. He identifies key challenges: headless login, least privilege for non-deterministic systems, and compliance tracking. Grinich presents four emerging patterns—persona shadowing, delegation chains, capability tokens, and human-in-the-loop escalation—and references standards like OAuth, UMA, GNAP, OIDC for agents, and verifiable credentials. He predicts a shift from 95% human traffic to 95% agent traffic, calling for middleware trust boundaries and urgent collaboration on agent identity standards.

Hiring & Building an AI Engineering Team: Dr. Bryan Bischof

Dec 31, 2024 · 29:07

Dr. Bryan Bischof, Head of AI at Hex, argues that building an AI engineering team requires hiring based on product stage—starting with full-stack engineers and data profiles, then adding designers and later MLEs—while avoiding the 'mythical man month' trap in early AI products. He advocates for data intuition over LeetCode in interviews, using a take-home data exercise to assess candidates' ability to extract meaning from data and give feedback. Key attributes he looks for are curiosity, urgency, and product-mindedness, noting that enthusiasm alone is insufficient. Bischof also recommends working directly with domain experts to model AI behavior and suggests a centralized AI platform team to support multiple product teams, rather than having each team build AI infrastructure independently.

Real ROI: Lessons from Enterprises that have already succeeded with LLMs at Scale: Raza Habib

Dec 31, 2024 · 20:01

Raza Habib, CEO of Humanloop, shares lessons from enterprises like Duolingo, Filevine, and Ironclad that have achieved real ROI with LLMs at scale. He argues that success hinges on centering domain experts (e.g., Duolingo's linguists do all prompt engineering), needing less ML expertise than expected, and breaking down evaluation into small, testable components rather than chasing a single metric. Habib emphasizes collecting end-user feedback (acceptance rates, edits, thumbs up/down) and building tooling for logging, regression testing, and team collaboration. Filevine doubled revenue by launching six LLM-powered products; Ironclad's open-source Rivet logging tool enabled agents to auto-negotiate 50% of contracts. The talk covers how to optimize the four-component chain (base model, prompt, data selection, function calling) and why simpler systems will win as models improve.

Lessons From A Year Building With LLMs

Jul 19, 2024 · 35:21

The six authors of the O'Reilly article "Lessons From A Year Building With LLMs" — Bryan Bischof, Jason Liu, Hamel Husain, Eugene Yan, Shreya Shankar, and Swix — argue that the model itself is not a moat and that success comes from continuous improvement centered on evals and data. They stress that AI engineers should treat models as SaaS, quickly swapping for better ones, and focus on product and user interactions. The talk warns against toxic practices like prematurely hiring ML engineers without data or blindly adopting tools, advocating instead for deliberate eval practice and data literacy. On tactics, they compare LLM-as-judge (quick to prototype) vs fine-tuned evaluators (more precise and faster), and emphasize looking at real user data regularly with automated guardrails. Ultimately, they conclude that going from demo to production requires sustained investment in infrastructure and evaluation, echoing MLOps lessons from a decade ago.

The AI Pivot: With Chris White of Prefect & Bryan Bischof of Hex

Nov 7, 2023 · 35:16

Chris White (CTO of Prefect) and Bryan Bischof (Head of AI at Hex) detail how their non-AI startups successfully pivoted to integrate AI, arguing that ruthless prioritization and deep product integration are key. White explains Prefect built the open-source Marvin project to learn from LLM experimenters, while adding AI features like error summaries to its core orchestration product, but had to restrain over-enthusiasm from engineers. Bischof describes Hex's Magic feature as an augmentation, not a separate product, and explains they built their own evaluation system but chose not to build a vector database, and killed a promising feature called Crystal Ball to avoid splitting the product experience. They both emphasize machine-to-machine interfaces and typed outputs, and caution that the work is often tedious data engineering. Their hot takes: White says stop building chat interfaces, AI is a tool; Bischof warns that the journey is boring but worth it.