outlines

No more bad outputs with structured generation: Remi Louf

Oct 14, 2024 · 15:32

Rémi Louf, CEO of .txt and co-maintainer of Outlines, argues that structured generation—guiding LLMs to produce valid regex, JSON, or context-free grammar outputs—eliminates parsing errors and hallucinations while adding negligible overhead. Outlines masks tokens that violate the target structure, enabling Mistral 7B to achieve 99.9% valid JSON (vs. 17% without). It also accelerates inference: a chatty 50-token ChatGPT answer shrinks to 8 tokens, and structured generation boosts open models beyond GPT-4—Phi-3 Medium hits 96.5% on Berkeley function calling (GPT-4 gets 93.5%). With one shot matching eight shots in accuracy, Louf contends that most text is structured and that structured generation should be the default for non-chatbot workflows.

Pydantic is STILL all you need: Jason Liu

Sep 6, 2024 · 15:21

Jason Liu returns to the AI Engineer World's Fair to argue that Pydantic (and his library Instructor) is still all you need for structured output with LLMs. He shows that the core API—response_model, streaming with iterables, and partials for real-time validation—remains unchanged, now supporting Ollama, LlamaCPP, Anthropic, Gemini, and more. Liu demonstrates validators that enforce rules like uppercasing names or verifying receipt totals, reducing errors with automatic retries. He applies structured output to RAG: using a Search model with optional date ranges and source selection, and a Response model with follow-up questions and validated URLs. For extraction, he creates classifiers using Literal types, meeting summaries with action items, and even tables as Pandas DataFrames via custom type hints. Liu’s key takeaway is that one retry often suffices, and as models get faster and smarter, structured output makes LLMs compatible with classical programming—turning generative AI into generating data structures defined by the developer.