State Space Models for Realtime Multimodal Intelligence: Karan Goel
Oct 29, 2024 · 14:26
Karan Goel, founder of Cartesia, argues that state space models (SSMs) are key to real-time multimodal intelligence, offering cheaper, faster, and higher-quality alternatives to transformers for streaming applications like conversational voice and on-device assistants. He contrasts batch intelligence (cloud APIs) with streaming intelligence needed for low-latency tasks, emphasizing that SSMs compress information linearly rather than storing all tokens, enabling efficient long-context processing. Cartesia’s voice generation model achieves instant latency in the data center and is being optimized for on-device Mac and desktop deployment. Goel asserts that compression helps long-context tasks (e.g., 24-hour security footage analysis) more than retrieval, and that SSMs now match transformer quality while scaling better for multimodal data.
