SD3

Customized, production ready inference with open source models: Dmytro (Dima) Dzhulgakov

Feb 16, 2025 · 18:55

Dmytro (Dima) Dzhulgakov, co-founder and CTO of Fireworks AI, argues that open source models are the future for production Gen AI applications, and Fireworks provides a platform to make them customized and production-ready. He explains that while proprietary models like GPT-4 are powerful, they are too large, expensive, and slow for many use cases, whereas open models like Llama or Gemma can be fine-tuned for specific domains to achieve better quality at up to 10x speed and lower cost. The main challenges of using open models—complex setup, performance tuning, and production scaling—are addressed by Fireworks' custom serving stack, which achieves the fastest long-prompt inference and serves SDXL fastest among providers, handling over 150 billion tokens per day. The platform supports fine-tuning and serving thousands of LoRA adapters on the same GPU with serverless pay-per-token pricing. Dzhulgakov also highlights the emerging architecture of compound AI systems, where function calling—exemplified by the open-source Fire Function model—connects LLMs to external tools and knowledge sources, enabling agentic applications like stock querying and chart generation. The episode…

Making Open Models 10x faster and better for Modern Application Innovation: Dmytro (Dima) Dzhulgakov

Oct 9, 2024 · 18:55

Dmytro Dzhulgakov, CTO of Fireworks AI, argues that open-source models are the future for GenAI applications because they offer lower latency, lower cost, and domain adaptability compared to proprietary models. He explains that open models can be up to 10x faster for narrow domains and cut costs significantly, using examples like fine-tuned Llama 3 for function calling. Fireworks addresses the challenges of setup, optimization, and production readiness with a custom serving stack that delivers the fastest inference for long prompts and image generation (e.g., SDXL). He highlights FireFunction V2, an open-source model for function calling that combines chat and tool use, and notes that Fireworks serves over 150 billion tokens per day for companies like Quora and Cursor. The talk emphasizes that platforms like Fireworks enable developers to start with serverless inference, fine-tune models, and scale to enterprise-grade deployment with dedicated hardware.