Tinker

Let LLMs Wander: Engineering RL Environments — Stefano Fiorucci

Apr 8, 2026 · 40:35

Stefano Fiorucci demonstrates how to build Reinforcement Learning environments for language models using the open-source Verifiers library, arguing that training small models with verifiable rewards can surpass large closed models on specific tasks. He maps classic RL concepts to LLMs, introduces Verifiers components for single-turn, multi-turn, and tool environments, and then walks through an experiment where he takes LiquidAI's LLM 2 — a small open model — and transforms it into a tic-tac-toe master via supervised fine-tuning and GRPO-based reinforcement learning. After training, the model dominates random opponents and draws 85% of games against optimal ones, eventually outperforming GPT-5 Mini against identical optimal opponents. Fiorucci shares practical lessons: large batch sizes (≥256) ensure stable training, hidden biases in opponent algorithms can skew results, and starting from a base model (not a reasoning model) avoids truncated thinking traces. He concludes that if you can define a clear reward signal, you can build an environment and train a small specialized model to beat a large closed model at a fraction of the cost.

Jack Morris: Stuffing Context is not Memory, Updating Weights is

Dec 29, 2025 · 1:02:44

Jack Morris argues that large language models fail at niche, long-tail knowledge tasks, such as optimizing AMD GPU kernels or answering private company queries, because they rely on context windows and RAG, which suffer from quadratic self-attention costs and context rot. He advocates for a third paradigm—training knowledge directly into model weights—using synthetic data generation (e.g., synthetic continued pretraining from Stanford) to expand small datasets and parameter-efficient methods like LoRA or memory layers to avoid catastrophic forgetting. Morris demonstrates that full fine-tuning on a 3M 10-K report causes the model to only regurgitate exact sentences, whereas generating diverse synthetic question-answer pairs enables better generalization. He notes that RL-based fine-tuning (e.g., GRPO) can achieve improvements with as few as 14 parameters, while memory layers offer the best trade-off between learning and forgetting. The episode also explores temporal information handling, federated learning resurgence, and the practical decision boundary between RAG and weight-based injection based on data freshness and volume.