Kaggle

Agentic Evaluations at Scale, For Everybody — Nicholas Kang & Michael Aaron, Google DeepMind

May 25, 2026 · 20:03

Nicholas Kang and Michael Aaron from Google DeepMind's Kaggle team argue that AI evaluations are broken due to being scattered, stale, and lacking transparency, citing a competing lab publishing inflated results by using custom compaction settings. They introduce four solutions: hackathons to channel community expertise, a standardized agent exam that returned 500+ submissions in its first week without promotion, a Game Arena where models play poker, chess, and werewolf for an ELO rating that cannot saturate, and an open benchmarks platform. A wastewater treatment plant engineer in Turkey built a novel safety benchmark from 20 years of field experience. They note that on SWE-Bench Pro, six frontier models land within a couple of percentage points, but the harness shifts performance by 22%, complicating comparisons. Challenges include high cost (400,000 poker hands for statistical significance), maintaining community engagement, and dealing with fast model deprecation cycles.

Building a Chess Coach — Anant Dole and Asbjorn Steinskog, Take Take Take

May 13, 2026 · 18:22

Anant Dole and Asbjørn Steinskog of Take Take Take, Magnus Carlsen's chess app, built an AI chess coach that keeps LLMs as translators rather than reasoners. Stockfish evaluates positions, tactical and positional detectors extract forks, pins, and structural weaknesses, and the LLM only converts those structured signals into English—preventing hallucination. They target sub-3-second latency using Gemini Flash. When a user flags bad commentary, it posts to Slack and injects into a running Claude Code channel via MCP. Claude investigates, modifies prompts or detectors, regenerates commentary, and asks clarifying questions. They run automated evals across 16 scenarios: Gemini Flash at 75%, Claude thinking below 60%, GPT-5 Mini lower. Their key insight: separate data pipeline from language generation, and close the loop with autonomous agents.

Gemma 4 Deep Dive — Cassidy Hardin, Researcher, Google DeepMind

Apr 27, 2026 · 19:03

Cassidy Hardin, a researcher at Google DeepMind, details the Gemma 4 family of open-source models, claiming they set a new precedent for small-scale performance. The family includes two on-device effective models (E2B and E4B) with Per Layer Embeddings (PLE) stored in flash memory, and two larger models: a 26B mixture-of-experts (MoE) with 128 total experts (8 active) and a 31B dense model that ranked #3 on the LM arena leaderboard, outperforming models 20x its size. Architectural innovations include interleaved local/global attention (5:1 ratio, sliding windows of 512/1024 tokens), grouped query attention (8 queries per key-value in global layers), and native multimodal support with variable aspect ratios and resolutions for vision encoders (550M params for large, 150M for small) plus a 305M parameter conformer for audio. All models are released under Apache 2.0 license, available for self-hosting on Hugging Face and Ollama, or cloud deployment on Vertex AI.

[Full Workshop] Reinforcement Learning, Kernels, Reasoning, Quantization & Agents — Daniel Han

Jul 19, 2025 · 2:42:28

Daniel Han of Unsloth presents a technical workshop covering reinforcement learning (RL), kernels, reasoning, quantization, and agents, arguing that RL with verifiable rewards (RLVR) is the key to unlocking LLM capabilities beyond supervised fine-tuning. He explains why open-source models plateaued after September 2024 until DeepSeek-R1 showed that RL can elicit reasoning, and breaks down PPO, GRPO, and the REINFORCE algorithm, emphasizing that GRPO removes the value model for efficiency. Han details how reward functions—not algorithms—are the hardest part, with examples like distance-based scoring for math. He demonstrates a free Colab notebook training a base model to reason, and shows that dynamic quantization can shrink models like DeepSeek-R1 from 730 GB to 140 GB with only ~1% accuracy loss, arguing that GPUs may stop getting faster after FP4 precision.

RAG Evaluation Is Broken! Here's Why (And How to Fix It) - Yuval Belfer and Niv Granot

Jun 3, 2025 · 10:58

Yuval Belfer and Niv Granot of AI21 Labs argue that current RAG evaluation is broken because benchmarks rely on local questions with answers contained in single chunks, failing to reflect real-world messy data. They demonstrate that standard RAG pipelines—like those from LangChain and LlamaIndex—achieve only 5-11% accuracy on aggregative questions about 22 FIFA World Cup documents, such as 'Which team has won the most times?' Their proposed fix converts unstructured corpuses into structured SQL databases by clustering documents, inferring schemas (e.g., year, winner, top scorer), and using text-to-SQL at inference. This structured RAG approach handles counting and max/min queries that standard chunking cannot, though it requires homogeneous data and careful normalization to avoid ambiguity (e.g., West Germany vs. Germany). They caution that RAG is not one-size-fits-all and that existing benchmarks miss these critical use cases.

LLM Quality Optimization Bootcamp: Thierry Moreau and Pedro Torruella

Feb 8, 2025 · 53:05

Thierry Moreau of OctoAI demonstrates how to fine-tune Llama 3 8B on a PII redaction task using OpenPipe and OctoAI, achieving 47% better accuracy and a 200x cost reduction (from $30 to $0.15 per million tokens) compared to GPT-4 Turbo. He explains that fine-tuning should follow prompt engineering and RAG, and works best for specialized tasks like function calling. The talk walks through building a fine-tuning dataset from the PI Masking 200k dataset, using OpenPipe to train a LoRA for $40, deploying it on OctoAI, and evaluating it to show the fine-tuned model scores 0.97 accuracy versus GPT-4’s 0.68. Moreau emphasizes that this continuous deployment cycle requires monitoring data drift and retraining, but tools like OpenPipe and OctoAI make it accessible even for teams without deep ML expertise.

LLM Scientific Reasoning: How to Make AI Capable of Nobel Prize Discoveries: Hubert Misztela

Sep 23, 2024 · 20:00

Hubert Misztela, an AI researcher at Novartis, argues that LLMs require more than naive RAG to achieve scientific reasoning capable of Nobel-level discoveries. He uses the 1990s petunia flower experiment — where three separate biological phenomena (including RNA interference) went unexplained for eight years until their common cause was found — as a benchmark. Misztela classifies question complexity from one-to-one to multi-needle problems and demonstrates that reasoning before retrieval (e.g., routing, GraphRAG) and after retrieval (e.g., relevance classifiers) improves hypothesis generation. His experiments show that strict prompting and a relevance classifier that evaluates each paper's contribution to advancing a hypothesis can recover the correct RNA/DNA link without post-discovery knowledge. He concludes that harder problems demand reasoning steps and that brute-force checking with LLMs may outperform embedding distances.