Storyteller: Building Multi-modal Apps with TS & ModelFusion - Lars Grammel, PhD
Jan 23, 2024 · 7:30
Lars Grammel presents Storyteller, a kid's story generator built with TypeScript and his ModelFusion library, which orchestrates multiple AI models to turn voice input into narrated audio stories with images. The app transcribes speech via OpenAI Whisper in 1.5 seconds, generates a story outline with GPT-3 TurboInstruct in 4 seconds, and then runs title, image, and audio generation in parallel. For consistency, it uses GPT-4 with a low temperature to produce structured story passages, and streams partial results so narration can begin before the full story is ready—addressing the 1.5-minute full generation time. Voices are selected by embedding speaker descriptions and retrieving matching ElevenLabs voices filtered by gender. Image prompts are extracted from the story via GPT-4 and rendered with Stability AI's Stable Diffusion XL. The result is a responsive, multi-modal app that lets users listen while the server continues processing.
