AMD

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Jul 19, 2025 · 19:03

Keegan McCallum, Head of ML Infrastructure at Luma AI, details how the company's Dream Machine model scaled from 500 to 9,000 H100 GPUs within hours to handle 1 million users in four days, outpacing ChatGPT's initial growth. He explains that their initial Triton inference server setup was brittle and ill-suited for multi-GPU, multi-node video models, prompting a re-architecture to a custom serving stack on vanilla PyTorch. To solve work starvation across user tiers, they implemented an SLO-based aging system that ranks jobs by the percentage of their worst-case wait time elapsed. For managing dozens of model versions, they store immutable full Python environments and checkpoints in object storage, with a YAML file controlling active deployments and enabling zero-downtime rollouts across thousands of GPUs. McCallum also discusses partnerships with Nvidia, AMD, and Grok, and how Luma's broader mission is to build general multimodal intelligence that generates, understands, and operates in the physical world.

ComfyUI Full Workshop — first workshop from ComfyAnonymous himself!

Jul 19, 2025 · 51:25

ComfyAnonymous and Yedrick Kosinski present ComfyUI, an open-source node-based canvas for generative AI that has become the top 150 most popular GitHub repos with 78,000 stars and 3 to 4 million active users. Started as a personal project in January 2023, it now supports image, video, audio, 3D, and text models, with 22,000 custom nodes from 3,000 developers. A key feature is embedding full workflows into generated files for easy sharing. The team discusses its story, including ComfyAnonymous's time at Stability AI and founding ComfyOrg, and addresses questions on CFG, VAEs, LoRAs, control nets, API nodes for remote generation, and future plans for cloud inference and improved onboarding.

Foundry Local: Cutting-Edge AI experiences on device with ONNX Runtime/Olive — Emma Ning, Microsoft

Jun 27, 2025 · 22:52

Emma Ning, Principal PM at Microsoft, presents Foundry Local, a platform for building cross-platform on-device AI applications using ONNX Runtime and Olive. She argues local AI is essential for low-bandwidth, privacy-sensitive, cost-efficient, and real-time latency scenarios, noting that modern hardware and optimized models like Phi-4-mini and DeepSeek make it viable. Ning demonstrates Foundry Local's CLI, model benchmarking (Qwen 1.5B at 90 tok/s vs. Phi-4-mini with richer output), and a cross-platform document summarization app built with JavaScript SDK, running identically on Windows and macOS. She also previews a local agent with MCP servers (file system and OCR) that extracts receipt totals using Phi-4-mini. Customer testimonial from Sava (CEO of Pieces) highlights improved memory management and tokens-per-second, while another partner underscores ease of install and hybrid cloud-local solutions.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.