Hugging Face

Gemma 4 Deep Dive — Cassidy Hardin, Researcher, Google DeepMind

Apr 27, 2026 · 19:03

Cassidy Hardin, a researcher at Google DeepMind, details the Gemma 4 family of open-source models, claiming they set a new precedent for small-scale performance. The family includes two on-device effective models (E2B and E4B) with Per Layer Embeddings (PLE) stored in flash memory, and two larger models: a 26B mixture-of-experts (MoE) with 128 total experts (8 active) and a 31B dense model that ranked #3 on the LM arena leaderboard, outperforming models 20x its size. Architectural innovations include interleaved local/global attention (5:1 ratio, sliding windows of 512/1024 tokens), grouped query attention (8 queries per key-value in global layers), and native multimodal support with variable aspect ratios and resolutions for vision encoders (550M params for large, 150M for small) plus a 305M parameter conformer for audio. All models are released under Apache 2.0 license, available for self-hosting on Hugging Face and Ollama, or cloud deployment on Vertex AI.

Z.ai GLM 4.6: What We Learned From 100 Million Open Source Downloads — Yuxuan Zhang, Z.ai

Nov 22, 2025 · 19:39

Yuxuan Zhang from zAI details the technical roadmap behind the GLM 4.6 open-source model series, which has surpassed 100 million downloads and tied for number one on the LMSYS Chatbot Arena alongside GPT-4o and Claude 3.5 Sonnet. The training pipeline uses 15 trillion tokens of pre-training data, followed by 7 trillion tokens of code and reasoning data, repo-level contexts at 32,000 tokens, and 100 billion tokens of long-context agent trajectories up to 128,000 tokens. Zhang introduces SLIME, a hybrid synchronous/asynchronous RL framework that decouples agent-environment interaction from GPU training to avoid bottlenecks. He explains why single-stage RL at 64,000 tokens outperforms multi-stage approaches for preserving long-context abilities, and shows that token-weighted loss converges faster than sequence-average loss for code RL. The multimodal GLM 4.5V model handles native resolution images and video with temporal index tokens, enabling GUI agent capabilities. Deployment is supported via vLLM and SGLang, with an API at z.ai and an open-source coding assistant.

Structuring a modern AI team — Denys Linkov, Wisedocs

Jul 24, 2025 · 17:40

Denys Linkov, who leads ML at Wisedocs, argues that building a modern AI team hinges on identifying your company's bottleneck—shipping features, acquiring users, or scalability—rather than reflexively hiring AI researchers. He introduces Ampere's Wager: trading your entire domain-savvy team for five top-lab researchers is usually a losing bet. For early-stage AI strategy, generalists who blend model training, serving, and business acumen outperform specialists; Linkov lived this in 2021 building a custom MLOps platform for a conversational AI startup and again in 2024 using advanced open-source tools for medical record processing. He stresses reskilling existing teams through weekly learning cadences and moving domain experts from giving feedback to writing evaluations. Hiring should hold context and act on it, verifying trends like 'don't hire juniors' against YC's AI school drawing 2,000 young people.

Going beyond RAG: Extended Mind Transformers - Phoebe Klett

Sep 11, 2024 · 16:04

Phoebe Klett presents Extended Mind Transformers (EMT), a modification to transformer attention that lets models retrieve relevant memory tokens during generation without fine-tuning. EMT uses relative position embeddings (RoPE in LLaMA, ALiBi in MPT) to enable zero-shot generalization. On a counterfactual retrieval benchmark up to 16K tokens, EMT outperforms fine-tuned models and, combined with RAG, surpasses GPT-4. It provides granular citations by showing exactly which memory tokens were attended to, and reduces hallucinations via active learning: when token-level entropy signals uncertainty, the model retrieves more memories. EMT is open-sourced on Hugging Face and GitHub with configurable parameters like stride length, top K, similarity masking, and unknown token elimination.