Thinking Machines

Jack Morris: Stuffing Context is not Memory, Updating Weights is

Dec 29, 2025 · 1:02:44

Jack Morris argues that large language models fail at niche, long-tail knowledge tasks, such as optimizing AMD GPU kernels or answering private company queries, because they rely on context windows and RAG, which suffer from quadratic self-attention costs and context rot. He advocates for a third paradigm—training knowledge directly into model weights—using synthetic data generation (e.g., synthetic continued pretraining from Stanford) to expand small datasets and parameter-efficient methods like LoRA or memory layers to avoid catastrophic forgetting. Morris demonstrates that full fine-tuning on a 3M 10-K report causes the model to only regurgitate exact sentences, whereas generating diverse synthetic question-answer pairs enables better generalization. He notes that RL-based fine-tuning (e.g., GRPO) can achieve improvements with as few as 14 parameters, while memory layers offer the best trade-off between learning and forgetting. The episode also explores temporal information handling, federated learning resurgence, and the practical decision boundary between RAG and weight-based injection based on data freshness and volume.

State of Startups and AI 2025 - Sarah Guo, Conviction

Aug 2, 2025 · 23:52

Sarah Guo of Conviction argues that AI's value creation is massive and early, with companies like Cursor reaching $100M ARR in 12 months and Harvey exceeding $70M ARR. She predicts that by end of 2026, AI agents will ship code directly to production, voice AI will replace text for most business communication, and inference costs will drop below a cent per million tokens. Reasoning is a new scaling vector unlocking higher-stakes use cases, and agent startups have increased 50% in the last year. Multimodal models from HeyGen and Eleven are already rocketing past $50M ARR. The model market is more competitive than ever, with GPT-4 costs falling from $30 to $2 per million tokens in 18 months and open-source like DeepSeek competing. Guo advises builders to focus on thick wrappers around LLMs, leveraging domain and workflow knowledge, and warns against generic text boxes: 'The prompt is a bug, not a feature.' Execution, not first-mover advantage, is the moat.

Frontier Feud: Anthropic, Google DeepMind, Meta FAIR, Thinking Machines — Barr Yaron, Amplify

Apr 19, 2025 · 22:26

Teams from Anthropic, Google DeepMind, Meta FAIR, and Thinking Machines compete in a Frontier Feud game hosted by Barr Yaron at the AI Engineer Summit 2025. Surveyed AI engineers name Ilya Sutskever as the most influential AI researcher, with Andrej Karpathy, Jeff Hinton, and Yann LeCun also on the board. Cost is the top consideration when choosing a model, followed by latency, eval scores, and open vs. closed source. The buzzword everyone is tired of hearing is AGI, with RAG and prompt engineering trailing. In the fast money round, Cursor tops favorite AI tools, 'Attention Is All You Need' wins most influential paper, and hardware failure is the biggest 2 a.m. nightmare. The winning team, Rocco's Basilisk, takes home a massive llama and other prizes.