Nano

Think You Can Build a Game with AI? Think Again! - Danielle An & David Hoe, Meta

Jul 8, 2026 · 18:00

Meta engineers Danielle An and David Hoe share insights from leading AI-driven game creation at Meta, arguing that while anyone can now prompt a basic game with AI, truly standout games require aesthetic cohesion, playtesting, and runtime LLMs that make NPCs dynamic and unscripted. They demonstrate how anchoring an LLM on a single image (like a QR code) can unify art style and gameplay, and show a multiplayer game where LLM-driven NPCs compete with unique personalities. The key shift is from linear, costly game development to parallel, rapid iteration cycles enabled by AI tools. They also highlight challenges: managing indeterminism across the entire stack (from user prompts to platform ranking), debugging agentic systems, and ensuring content safety with runtime generation. The takeaway: whether you're a beginner, intermediate, or advanced creator, the AI gaming frontier is at day zero, with open problems in token economics and scalable safety.

Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI

Apr 10, 2026 · 40:51

Mahmoud Mabrouk, co-founder of Agenta AI, demonstrates how to build calibrated LLM-as-a-judge evaluators using the GEPA prompt optimization algorithm, arguing that miscalibrated evals are worse than none. He walks through a practical workflow for a customer support agent using the TaoBench airline dataset, covering metric design, data annotation, and GEPA-based optimization. The seed judge achieved 61% accuracy; after optimization, accuracy rose to 74% with reduced bias, though the judge still struggled to fully learn the complex policy. Mabrouk shares key lessons: start with a seed prompt biased toward compliance, use larger models for refinement, overfit to training data first, and beware of high token costs.

How to build world-class AI products — Sarah Sachs (AI lead @ Notion) & Carlos Esteban (Braintrust)

Jun 27, 2025 · 1:43:46

Sarah Sachs (AI lead at Notion) and Carlos Esteban (Braintrust) explain that building great AI products requires 10% prompting and 90% evals and observability, with Notion AI using Braintrust to iterate on prompts and models. Sachs details their cycle: curate small datasets, tie them to scoring functions (LLM-as-a-judge with per-sample prompts and heuristic checks), run evals before shipping, and use production logs to catch regressions. She notes Notion AI supports 100M+ users, switches models in under a day, and 60% of enterprise users are non-English—requiring multilingual eval rigor. Carlos and Doug then walk through Braintrust's framework: tasks (prompts, tools, agents), datasets, and scores (0-1). They demonstrate offline evals via playground and SDK, online scoring in production, user feedback capture, human review setup, and remote evals to bridge complex code with the playground. The workshop covers moving from pre-prod evals to production monitoring, closing the feedback loop by adding underperforming spans to datasets.