PaLM

RL for Autonomous Coding — Aakanksha Chowdhery, Reflection.ai

Jul 16, 2025 · 19:27

Aakanksha Chowdhery, CEO of Reflection AI and former lead researcher on PaLM and Gemini at Google, argues that reinforcement learning (RL) scaling is the next frontier for autonomous coding agents. She explains that inference-time techniques like majority voting and self-revision improve accuracy but require many samples (e.g., 10,000 for rare correct generations). RL training can learn to generate correct outputs directly, especially in verifiable domains like code with unit tests. Reflection AI aims to build superintelligence starting with autonomous coding, leveraging automated verification to design better reward functions. She notes challenges in scaling RL, including system complexity and reward hacking, but sees coding as ideal due to execution feedback.

A Practical Guide to Efficient AI: Shelby Heinecke

Nov 18, 2024 · 17:45

Shelby Heinecke, who leads an AI research team at Salesforce, presents five orthogonal dimensions for making AI models efficient: efficient architecture selection, pre-training, fine-tuning, inference, and prompting. She highlights the power of small models like Phi-3 (3.8B parameters outperforming a 7B model), mobile LLM (350M parameters on par with 7B after fine-tuning), and Octopus (2B fine-tuned Gemma exceeding GPT-4 on Android tasks). For efficient inference, she explains post-training quantization, showing 4-bit quantization nearly halves memory usage without performance loss (e.g., LLaMA models), but warns 3-bit can degrade quality. She recommends frameworks like LLaMA CBP and ONNX Runtime for quantization and introduces her team's open-source Mobile AI Bench for evaluating quantized models, including an iOS app to measure latency and battery drain. The central claim is that deploying AI in constrained environments—cloud, on-prem, or edge—demands efficiency, and these practical techniques bridge the gap from demo to production.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.