From 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents — Cormac Brick, Google
May 20, 2026 · 21:01
Google's Cormac Brick explains how developers can build on-device AI agents using either system-level Gemini Nano via AI Core or app-level custom models via LiteRT-LM. He demonstrates a skill harness built on Gemma 4 that enables agentic tasks like restaurant roulette, running fully on-device with JavaScript UI. For fine-tuned tiny models, Function Gemma at 270M parameters improved from 46% to over 90% accuracy on eight of ten app-intent functions after synthetic data fine-tuning. The talk also covers the Eloquent transcription app, which chains two Gemma 3-based models (ASR and text polishing) under a few hundred million parameters for offline use. Key trade-offs are latency, privacy, and customization versus system integration effort.
