TLMs: Tiny LLMs and Agents on Edge Devices with LiteRT-LM — Cormac Brick, Google
May 3, 2026 · 1:20:58
Cormac Brick from Google AI Edge explains how Tiny LLMs (sub-1B parameters) and on-device agent skills are making edge AI practical. He details LiteRT-LM, an open-source runtime that runs Gemma models on Android, iOS, and embedded systems, achieving over 1,000 tokens/s on high-end phones. Agent skills use progressive disclosure—loading skill details on demand—enabling reliable tool calling on 2B-4B models. For app deployment, fine-tuning boosts tiny model reliability by 20-40 points (e.g., Function Gemma 270M hit 86% accuracy on 10 functions). Synthetic data workflows and modular design (ASR + text polishing) power real apps like AI Edge Eloquent, which runs entirely offline. Safety is managed through system-level checkers and narrow functional scope for tiny models.
