Mastercard

Jack Morris: Stuffing Context is not Memory, Updating Weights is

Dec 29, 2025 · 1:02:44

Jack Morris argues that large language models fail at niche, long-tail knowledge tasks, such as optimizing AMD GPU kernels or answering private company queries, because they rely on context windows and RAG, which suffer from quadratic self-attention costs and context rot. He advocates for a third paradigm—training knowledge directly into model weights—using synthetic data generation (e.g., synthetic continued pretraining from Stanford) to expand small datasets and parameter-efficient methods like LoRA or memory layers to avoid catastrophic forgetting. Morris demonstrates that full fine-tuning on a 3M 10-K report causes the model to only regurgitate exact sentences, whereas generating diverse synthetic question-answer pairs enables better generalization. He notes that RL-based fine-tuning (e.g., GRPO) can achieve improvements with as few as 14 parameters, while memory layers offer the best trade-off between learning and forgetting. The episode also explores temporal information handling, federated learning resurgence, and the practical decision boundary between RAG and weight-based injection based on data freshness and volume.

Navigating Challenges and Technical Debt in LLMs Deployment: Ahmed Menshawy

Dec 31, 2024 · 16:15

Ahmed Menshawy, VP of AI Engineering at Mastercard, argues that deploying LLMs requires overcoming massive technical debt, with over 95% of the work outside the ML code. He details how Mastercard boosted fraud detection by 300% using LLMs while adhering to seven responsible AI principles. The episode covers RAG as a solution for hallucination, attribution, and access control, noting that 80% of organizational data is unstructured and 71% of companies struggle to manage it. Menshawy highlights that GPU compute for inference now exceeds training, and that fine-tuning open-source models enables joint training of retriever and generator. He also cautions against AGI hype, quoting Ada Lovelace and a Nature article urging focus on current AI risks.