LM Arena

Sovereign Escape Velocity: Ownership w Open Models — Gus Martins, & Ian Ballantyne, Google DeepMind

Jun 10, 2026 · 20:52

Gus Martins and Ian Ballantyne of Google DeepMind introduce Gemma 4, a family of open-weight models that deliver high quality per parameter, enabling deployment on a single GPU or even a phone. They argue that the models' efficiency — a 31B model rivals those twenty times larger — and the shift to Apache 2.0 licensing remove barriers for sovereign institutions like those in Ukraine, Bulgaria, and Brazil. Ian demonstrates multi-agent translation running locally on an M4 Mac, showcasing ownership and control over agentic workloads.

Gemma 4 Deep Dive — Cassidy Hardin, Researcher, Google DeepMind

Apr 27, 2026 · 19:03

Cassidy Hardin, a researcher at Google DeepMind, details the Gemma 4 family of open-source models, claiming they set a new precedent for small-scale performance. The family includes two on-device effective models (E2B and E4B) with Per Layer Embeddings (PLE) stored in flash memory, and two larger models: a 26B mixture-of-experts (MoE) with 128 total experts (8 active) and a 31B dense model that ranked #3 on the LM arena leaderboard, outperforming models 20x its size. Architectural innovations include interleaved local/global attention (5:1 ratio, sliding windows of 512/1024 tokens), grouped query attention (8 queries per key-value in global layers), and native multimodal support with variable aspect ratios and resolutions for vision encoders (550M params for large, 150M for small) plus a 305M parameter conformer for audio. All models are released under Apache 2.0 license, available for self-hosting on Hugging Face and Ollama, or cloud deployment on Vertex AI.

Gemma, DeepMind's Family of Open Models — Omar Sanseviero, Google DeepMind

Apr 20, 2026 · 15:26

Omar Sanseviero presents Gemma 4, Google DeepMind's latest family of open models, which range from 2B to 32B parameters and introduce a novel per-layer embedding (E2B) architecture optimized for on-device inference. The models feature multimodal understanding (images, video, audio), multilingual support across 140+ languages, and are released under an Apache 2 license. Within a week, Gemma 4 reached 10 million downloads, contributing to over 500 million total downloads for the Gemma family and 100,000 community-derived models. Sanseviero highlights official variants like Shield Gemma for content safety and MedGemma for medical tasks, as well as community efforts such as AI Singapore's Southeast Asian language models and Sarvam's sovereign AI initiative for India. He emphasizes real-world applications including cancer therapy pathway discovery and fully offline agentic tasks on phones and Raspberry Pis, arguing that open models are rapidly enabling high-performance, private, and customizable AI across diverse use cases.

The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Jun 3, 2025 · 11:20

Darius Emrani exposes how AI benchmarks are rigged, showing that xAI cherry-picked Grok-3 comparisons, OpenAI funded FrontierMath for privileged access, and Meta submitted 27 Llama-4 variants to LM Arena optimizing style over substance. Citing Goodhart's Law, he argues that when benchmarks target billions in investment, they cease to measure real capability—Andrej Karpathy admits he doesn't know which metrics to trust. Emrani provides a 5-step framework to build use-case-specific evaluations, emphasizing that 39% of score variance comes from writing style. He advocates for apple-to-apple comparisons, open-source test sets, and style-controlled metrics, concluding that teams should stop chasing leaderboards and instead iterate on real production data to ship reliable AI.