Bryan Bischof

Hiring & Building an AI Engineering Team: Dr. Bryan Bischof

Dec 31, 2024 · 29:07

Dr. Bryan Bischof, Head of AI at Hex, argues that building an AI engineering team requires hiring based on product stage—starting with full-stack engineers and data profiles, then adding designers and later MLEs—while avoiding the 'mythical man month' trap in early AI products. He advocates for data intuition over LeetCode in interviews, using a take-home data exercise to assess candidates' ability to extract meaning from data and give feedback. Key attributes he looks for are curiosity, urgency, and product-mindedness, noting that enthusiasm alone is insufficient. Bischof also recommends working directly with domain experts to model AI behavior and suggests a centralized AI platform team to support multiple product teams, rather than having each team build AI infrastructure independently.

Lessons From A Year Building With LLMs

Jul 19, 2024 · 35:21

The six authors of the O'Reilly article "Lessons From A Year Building With LLMs" — Bryan Bischof, Jason Liu, Hamel Husain, Eugene Yan, Shreya Shankar, and Swix — argue that the model itself is not a moat and that success comes from continuous improvement centered on evals and data. They stress that AI engineers should treat models as SaaS, quickly swapping for better ones, and focus on product and user interactions. The talk warns against toxic practices like prematurely hiring ML engineers without data or blindly adopting tools, advocating instead for deliberate eval practice and data literacy. On tactics, they compare LLM-as-judge (quick to prototype) vs fine-tuned evaluators (more precise and faster), and emphasize looking at real user data regularly with automated guardrails. Ultimately, they conclude that going from demo to production requires sustained investment in infrastructure and evaluation, echoing MLOps lessons from a decade ago.

The AI Pivot: With Chris White of Prefect & Bryan Bischof of Hex

Nov 7, 2023 · 35:16

Chris White (CTO of Prefect) and Bryan Bischof (Head of AI at Hex) detail how their non-AI startups successfully pivoted to integrate AI, arguing that ruthless prioritization and deep product integration are key. White explains Prefect built the open-source Marvin project to learn from LLM experimenters, while adding AI features like error summaries to its core orchestration product, but had to restrain over-enthusiasm from engineers. Bischof describes Hex's Magic feature as an augmentation, not a separate product, and explains they built their own evaluation system but chose not to build a vector database, and killed a promising feature called Crystal Ball to avoid splitting the product experience. They both emphasize machine-to-machine interfaces and typed outputs, and caution that the work is often tedious data engineering. Their hot takes: White says stop building chat interfaces, AI is a tool; Bischof warns that the journey is boring but worth it.