Weaviate

Jack Morris: Stuffing Context is not Memory, Updating Weights is

Dec 29, 2025 · 1:02:44

Jack Morris argues that large language models fail at niche, long-tail knowledge tasks, such as optimizing AMD GPU kernels or answering private company queries, because they rely on context windows and RAG, which suffer from quadratic self-attention costs and context rot. He advocates for a third paradigm—training knowledge directly into model weights—using synthetic data generation (e.g., synthetic continued pretraining from Stanford) to expand small datasets and parameter-efficient methods like LoRA or memory layers to avoid catastrophic forgetting. Morris demonstrates that full fine-tuning on a 3M 10-K report causes the model to only regurgitate exact sentences, whereas generating diverse synthetic question-answer pairs enables better generalization. He notes that RL-based fine-tuning (e.g., GRPO) can achieve improvements with as few as 14 parameters, while memory layers offer the best trade-off between learning and forgetting. The episode also explores temporal information handling, federated learning resurgence, and the practical decision boundary between RAG and weight-based injection based on data freshness and volume.

Building efficient hybrid context query for LLM grounding: Simrat Hanspal

Feb 5, 2025 · 11:35

Simrat Hanspal from Hasura explains how to build efficient hybrid context queries for LLM grounding using Hasura's GraphQL API, demonstrated via an e-commerce product search use case. The talk covers three query types: semantic search (e.g., 'essential oils for relaxation'), structured search (e.g., 'products less than $500'), and hybrid queries combining both (e.g., 'essential oil diffusers between $500 to $1,000'). Hanspal shows how Hasura unifies relational (Postgres) and vector (Weaviate) databases into a single GraphQL API, auto-vectorizing records via events. A critical security demonstration highlights how role-based permissions (e.g., 'Product Search Bot' role with only select access) prevent malicious insert mutations like adding a fake product with ID 7001. The approach enables secure, dynamic data retrieval for Retrieval-Augmented Generation (RAG) pipelines without building separate APIs.

Building Production-Ready RAG Applications: Jerry Liu

Nov 15, 2023 · 18:35

Jerry Liu, CEO of LlamaIndex, explains how to productionize Retrieval Augmented Generation (RAG) systems by moving beyond naive implementations. He identifies key challenges: low retrieval precision causing hallucination, low recall from insufficient top-K, and lost-in-the-middle problems. Liu advocates starting with 'table stakes' improvements like tuning chunk sizes (showing optimal values per dataset), adding metadata filters (e.g., year=2021 for SEC 10Q queries), and hybrid search. More advanced techniques include 'small-to-big retrieval', embedding smaller chunks for precision then expanding windows for synthesis, and using reranking to improve recall. Finally, he explores agent architectures where each document becomes a tool for summarization or QA, and fine-tuning—generating synthetic query datasets from raw text to fine-tune embeddings, or distilling GPT-4's chain-of-thought into GPT-3.5 Turbo for better reasoning.

AI Engineering 201: The Rest of the Owl

Nov 8, 2023 · 56:57

Charles Frye, instructor of the Full Stack LLM Bootcamp, presents essential patterns for building language user interfaces (LUIs), arguing that while RAG chatbots are the 'to-do list app' of AI engineering, structured outputs via function calling (e.g., OpenAI's JSON schema, Instructor library) improve robustness, and agents with memory (like generative agents or Voyager in Minecraft) represent the true AI frontier. He emphasizes the need for hybrid search combining vector and keyword retrieval (citing Vespa, Postgres, Redis), and warns that monitoring and evaluation are the hardest engineering challenges: monitoring user behavior, latency quantiles (especially 99th percentile), and costs must be paired with observability tools like Honeycomb or Gantry, while evaluation often requires iterated decomposition or using LLMs as evaluators (GPT-4 as 90th percentile crowd worker). The episode concludes that shipping to learn — starting with production data to generate tests — is the dominant engineering mindset, and that the field is still filling in the gaps between inference and full product value.