H100

An AI Agent Became the #1 Contributor in OpenAI's Hiring Challenge — Zhengyao Jiang, Weco

Jul 16, 2026 · 16:16

Zhengyao Jiang, CEO of Weco AI, details how their autonomous agent Aiden became the top contributor in OpenAI's Parameter Golf competition, setting 7 leaderboard records—more than twice any human's 3. Over 22 days, Aiden ran 1,300 experiments on a single H100 node, using just 4% of total compute while contributing 15% of records and achieving a 28% leaderboard hit rate (six times the community average). The agent excelled at finding and implementing ideas from human researchers, combining them (e.g., gated attention from a paper with a tokenizer improvement) to produce breakthrough results. Jiang argues that auto-research does not replace human creativity but shifts it upward: humans design evals and code abstractions that bias the agent's search, a new craft he compares to training neural networks. The talk reveals that Aiden's H-index in the community was 10 versus the next human's 7, proving autonomous agents can produce work that others build on.

Sovereign Escape Velocity: Ownership w Open Models — Gus Martins, & Ian Ballantyne, Google DeepMind

Jun 10, 2026 · 20:52

Gus Martins and Ian Ballantyne of Google DeepMind introduce Gemma 4, a family of open-weight models that deliver high quality per parameter, enabling deployment on a single GPU or even a phone. They argue that the models' efficiency — a 31B model rivals those twenty times larger — and the shift to Apache 2.0 licensing remove barriers for sovereign institutions like those in Ukraine, Bulgaria, and Brazil. Ian demonstrates multi-agent translation running locally on an M4 Mac, showcasing ownership and control over agentic workloads.

GPU Cloud Deployment Without Leaving Your IDE — Audry Hsu, RunPod

Jun 9, 2026 · 20:19

Audry Hsu of RunPod introduces Flash, a Python SDK that deploys GPU cloud functions from a developer's IDE with a single decorator, eliminating the slow iteration cycle of commits, Docker builds, and server allocation. She demonstrates hot reload, swapping Stable Diffusion XL Turbo for DreamShaper instantly, and a pipeline that chains Qwen 3 for prompt generation, DreamShaper for image rendering, and Nano Banana 2 for photo composition. RunPod's serverless H100 pricing is $0.00116 per second, charged only during active inference. Hsu recommends starting with pods for experimentation and switching to serverless when scaling to hundreds of workers across data centers.

Road to 5 Million Tokens: Breaking Barriers in Long Context Training — Max Ryabinin, Together AI

Jun 8, 2026 · 15:50

Max Ryabinin from Together AI presents their research on extending transformer context length to 5 million tokens using Untied Ulysses, which cuts activation memory by reusing buffers across attention head iterations. The talk walks through a stack of techniques including fully sharded data parallelism, DeepSpeed Ulysses context parallelism for an 8x activation reduction, activation checkpointing for another 8x, CPU offloading of transformer block inputs, and chunked sequence training. Even with these, training a LLaMA 3B model with 3 million tokens fits on an 8xH100 node, but 5 million requires Untied Ulysses. Instead of allocating one large buffer per attention head group, it chunks heads further and reuses buffers across iterations, cutting activation memory with negligible throughput impact. At both 8B and 32B scale, results match the most memory-optimized transformer training baselines while pushing sequence length 25% further than prior Ulysses implementations.

Your Coding Agent Should Do AI System Engineering — Ben Burtenshaw, Hugging Face

May 21, 2026 · 18:25

Ben Burtenshaw from Hugging Face demonstrates how coding agents can take on AI systems engineering tasks—writing CUDA kernels, fine-tuning models, and running multi-agent research labs—by leveraging skills and the Hugging Face Hub. He shows a 1.88x speedup on H100s with an RMSNorm kernel written by Claude Code, and a fine-tuned Qwen3 0.6B achieving 35% on LiveCodeBench. Skills compress years of specialization into hours by turning zero-shot tasks into few-shot workflows. For multi-agent research, a Planner generates hypotheses from papers, Workers implement them as training scripts, and a Reporter monitors results via the open-source Trackio dashboard, with all jobs running on Hub compute. The key is exposing open primitives like kernels, Trackio, and HF jobs as agent-controllable tools.

Gemma, DeepMind's Family of Open Models — Omar Sanseviero, Google DeepMind

Apr 20, 2026 · 15:26

Omar Sanseviero presents Gemma 4, Google DeepMind's latest family of open models, which range from 2B to 32B parameters and introduce a novel per-layer embedding (E2B) architecture optimized for on-device inference. The models feature multimodal understanding (images, video, audio), multilingual support across 140+ languages, and are released under an Apache 2 license. Within a week, Gemma 4 reached 10 million downloads, contributing to over 500 million total downloads for the Gemma family and 100,000 community-derived models. Sanseviero highlights official variants like Shield Gemma for content safety and MedGemma for medical tasks, as well as community efforts such as AI Singapore's Southeast Asian language models and Sarvam's sovereign AI initiative for India. He emphasizes real-world applications including cancer therapy pathway discovery and fully offline agentic tasks on phones and Raspberry Pis, arguing that open models are rapidly enabling high-performance, private, and customizable AI across diverse use cases.

AI Kernel Generation: What's working, what's not, what's next – Natalie Serrino, Gimlet Labs

Dec 17, 2025 · 19:15

Natalie Serrino, cofounder of Gimlet Labs, presents how AI-generated kernels can automatically speed up custom PyTorch code by up to 24% on Apple M4 hardware using the Metal framework, with a 40% speedup from kernel fusion. The agentic system iterates through compilation, execution, correctness, and optimization, but faces challenges like validation of floating-point results and reliable benchmarking. Successes include rewriting average pool 1D as a convolution for 80% improvement, while failures occur on heavily optimized ops like matrix multiply. A real-world audio encoder model saw 70% faster inference on RTX 6000 Blackwell via six custom fused kernels. Serrino emphasizes that AI is best for rapidly searching optimizations and porting code to new hardware, not for surpassing human experts on novel algorithms.

Why We Don’t Need More Data Centers - Dr. Jasper Zhang, Hyperbolic

Aug 1, 2025 · 14:18

Dr. Jasper Zhang, CEO of Hyperbolic, argues that the AI industry's focus on building more data centers is inefficient because GPUs already sit idle 80% of the time. Instead, he advocates for a GPU marketplace that aggregates supply from fragmented providers, cutting costs by 50–75%—Hyperbolic offers H100s at $0.99/hour versus $2–$11 from traditional clouds. His HyperDOS software connects any Kubernetes cluster in 5 minutes, enabling dynamic rental and release of GPUs. He illustrates with a startup that can reduce compute costs from $43.8 million to $6.9 million by flexibly scaling capacity and selling idle GPUs, effectively boosting productivity 6x on the same budget. Zhang contends that smarter allocation, not just more data centers, is key to sustainable and democratized AI infrastructure.

Hacking the Inference Pareto Frontier - Kyle Kranen, NVIDIA

Aug 1, 2025 · 20:25

Kyle Kranen, architect of NVIDIA Dynamo, explains how to break the inference Pareto frontier by manipulating trade-offs between quality, latency, and cost using a toolkit of techniques. Disaggregation separates prefill and decode phases, achieving up to 2× tokens per second per GPU at fixed latency for LLaMA 70B on 16 H100s. Smart routing maximizes KV cache hits, asymptotically reducing prefill work as deployments scale. Structure from agentic workloads, like inference-time scaling, enables smaller models (e.g., 8B queried 3–4 times) to match larger models' quality at lower cost. KV manipulation offloads caches during tool calls (e.g., 30-second delays) to avoid re-prefill. Dynamism adjusts worker specialization and autoscales prefill/decode ratios in real time to handle shifting user distributions, ensuring disaggregation reaches maximum potential.

Serving Voice AI at $1/hr: Open-source, LoRAs, Latency, Load Balancing - Neil Dwyer, Gabber

Jul 31, 2025 · 16:09

Neil Dwyer, CTO of Gabber, details how his startup serves real-time voice AI for under $1 per hour using open-source Orpheus TTS, LoRAs for emotive voice cloning, and vLLM with FP8 dynamic quantization to achieve 95-105 tokens per second on L40S GPUs. He explains the critical latency challenge of 'head of line silence' in Orpheus (600ms in default voices) and how fine-tuning LoRAs reduces it to ~100ms, fitting within a 1.5-second budget for real-time conversation. Gabber batches multiple LoRA generations per GPU and uses a consistent hash ring for load balancing across servers, enabling popular clones to be dynamically replicated. The talk argues that with current open-source tools, building affordable consumer voice AI is accessible to small teams.

The Unofficial Guide to Apple’s Private Cloud Compute - Jmo, CONFSEC

Jul 30, 2025 · 20:36

Jonathan Mortensen explains Apple's Private Cloud Compute (PCC), a paradigm shift in confidential computing that delivers cryptographically provable privacy for AI inference on remote servers. He details Apple's five requirements—stateless computation, enforceable guarantees, non-targetability, no privileged runtime access, and verifiable transparency—and six technical components, including oblivious HTTP, blind signatures, secure enclave, remote attestation, transparency logs, and secure boot. The talk highlights how remote attestation allows an iPhone to verify server software and tie encryption to that verified state, while the transparency log enables public auditing of all deployed binaries. Mortensen notes trade-offs: complete trust in Apple's supply chain, higher latency and compute costs, and no support for custom models or third-party developers. He points to alternative tools like TPMs, SigStore, and confidential VMs for non-Apple environments, and concludes that Apple's approach is being adopted by Azure AI and Meta.

What every AI engineer needs to know about GPUs — Charles Frye, Modal

Jul 20, 2025 · 19:52

Charles Frye of Modal explains that AI engineers need to understand GPU hardware constraints to optimize inference, arguing GPUs embrace bandwidth over latency and that Tensor Cores for low-precision matrix-matrix multiplication are the key resource. He describes how GPUs achieve 16,000+ parallel threads per cycle on H100, and notes Patterson’s Law: bandwidth improves at the square of latency. The main insight: arithmetic intensity favors N² operations per N memory loads, so matrix-matrix operations are efficient while matrix-vector is wasteful. Frye demonstrates that running a small 8B model 1,000 times on the same prompt matches GPT-4 quality, and that multi-token prediction and multi-sample query become nearly free because Tensor Cores handle expanded batches as matrix-matrix multiplications. He recommends using smaller models that fit on a single GPU and scaling via multiple generations.

Trends Across the AI Frontier — George Cameron, ArtificialAnalysis.ai

Jul 8, 2025 · 17:52

George Cameron of Artificial Analysis presents multiple frontiers—reasoning, open-weights, cost, speed—across the AI stack, arguing that trade-offs between intelligence, latency, and expense are critical for building applications. Reasoning models like O4 mini high use an order of magnitude more output tokens (72M vs GPT-4.1's 7M) and take over 40 seconds per response versus 4.7 seconds, impacting agentic workflows where 30 sequential calls multiply latency. The open-weights gap has nearly closed, with China-based labs like DeepSeek R1 and Alibaba's Qwen 3 leading. O3 cost roughly $2,000 to run the intelligence index, while GPT-4.1 nano is over 500 times cheaper. Output speeds have jumped from GPT-4's 40 tokens/s in 2023 to over 1,000 on a B200 accelerator. Despite efficiency gains, demand for compute will keep rising due to larger models, reasoning’s extra tokens, and multi-step agents.

Optimizing inference for voice models in production - Philip Kiely, Baseten

Jul 1, 2025 · 15:13

Philip Kiely of Baseten shows how open-source TTS models like Orpheus TTS, built on a LLaMA 3.2 3B backbone, can be optimized for production inference using LLM tooling such as TensorRT-LLM and FP8 quantization, achieving time-to-first-byte (TTFB) under 150 milliseconds and supporting 16–24 simultaneous streams on a single half-H100 GPU. He argues that because TTS models are architecturally similar to LLMs, techniques like dynamic batching, KV-cache quantization, and Torch Compile on the audio decoder can dramatically reduce latency and increase concurrency. He emphasizes that for real-time voice agents, the goal is not raw tokens per second (real-time requires only 83 TPS for Orpheus) but low TTFB and high throughput to minimize GPU spend. However, Kiely warns that non-runtime factors—such as client code using sequential requests without session reuse or sending traffic to distant data centers—can easily add back the milliseconds saved at the model level, and that infrastructure connecting listening, thinking, and talking pipelines is often the dominant source of latency.

How fast are LLM inference engines anyway? — Charles Frye, Modal

Jun 27, 2025 · 16:07

Charles Frye presents benchmarks from hundreds of runs on Modal comparing open-source inference engines VLM, SGLang, and TensorRTLM across models like Qwen 3 and Gemma 27B, arguing open weights models have caught up to proprietary ones, making self-hosting viable. He shows, for example, that Qwen 3 (MoE) on VLM achieves ~1 request/sec with 128 input tokens and 1024 output tokens, while switching to a RAG-like workload (1024 in, 128 out) yields a 4x throughput improvement. Frye warns that optimizing for context over reasoning can improve latency without sacrificing quality, and notes that the engines' out-of-the-box performance varies by model—e.g., SGLang underperforms VLM on Gemma due to less optimization. He also highlights the gap between prefill (parallel) and decode (autoregressive) speeds, which a rationalist would expect from transformer architecture. The benchmarks, available at modal.com/llmalmanac, aim to help engineers choose hardware and engines, with contributions welcome for optimized configs like TensorRTLM's knobs.

How to Build Your Own AI Data Center in 2025 — Paul Gilbert, Arista Networks

Apr 27, 2025 · 23:00

Paul Gilbert, tech lead at Arista Networks, explains the key considerations for building AI data center networks in 2025, emphasizing the stark differences from traditional enterprise networks. He details the backend GPU network (eight 400G ports per H100 server, no over-subscription), the front-end storage network (calmer, 100-200G), and the need for lossless Ethernet with ECN and PFC flow control to prevent packet drops during synchronized GPU bursts. Gilbert highlights power challenges (10.2kW per GPU server, requiring 100-200kW water-cooled racks) and the importance of telemetry like RDMA error monitoring and an AI agent that correlates GPU and network issues. He also covers advanced load balancing (cluster-aware, up to 93% utilization) and smart system upgrades without downtime, while noting the upcoming Ultra Ethernet Consortium (v1.0 in 2025) that shifts congestion control to NICs.

Keynote: The AI developer experience doesn't have to suck – why and how we built Modal

Feb 22, 2025 · 21:38

Eric Bernhardson, CEO of Modal, explains why and how his company replaced Kubernetes and Docker with a custom container system to deliver sub-second cold starts for AI developers. Modal turns any Python function into a serverless function with a decorator, runs on thousands of H100s, and fans out to 10,000 parallel calls. To achieve fast startup, Modal built content-addressable storage for deduplication, lazy file loading with prefetching, and uses gVisor for CPU memory snapshotting, cutting Stable Diffusion startup to seconds. The company built its own scheduler and file system, and uses mixed integer programming to manage a global GPU pool across cloud vendors. Customers like Suno use Modal for AI-generated music inference. Modal offers $30/month free credits.

Accelerating Mixture of Experts Training With Rail Optimized InfiniBand Networking in Crusoe Cloud

Feb 12, 2025 · 17:45

Ievgen Bakulenko, product manager at Crusoe Cloud, explains how their rail-optimized InfiniBand networking accelerates training for sparse mixture of experts models. By leveraging NVIDIA's PXN feature, which allows GPUs to communicate across different rails using the internal NVSwitch in a single hop, Crusoe achieves a 50% improvement in synthetic benchmark latency and bandwidth for both small and large messages. In a real-world test fine-tuning the Mixtral model (8 feed-forward blocks, 7 billion parameters) on 240 H100 GPUs, this topology reduced training time by 14%, directly lowering cost and time-to-train. Bakulenko also outlines Crusoe's AI cloud platform, its climate-aligned mission using stranded energy, and its focus on easy-to-use infrastructure for AI engineers.

System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis

Feb 11, 2025 · 18:29

Dylan Patel of SemiAnalysis breaks down the inference challenges for next-generation frontier models like GPT-4 (1.8 trillion parameters) and upcoming models trained on 100,000+ GPU clusters. He emphasizes that prefill (prompt processing) is compute-intensive while decode (token generation) is memory bandwidth-intensive, creating a systems problem where serving 64 users at 30 tokens/second requires 60 terabytes/second of memory bandwidth. Patel details engineering strategies such as continuous batching to improve batch utilization by 10-100x, disaggregated prefill to isolate noisy neighbors and maintain time-to-first-token SLAs, and context caching (like Google's) to cache KV cache on CPU/storage instead of GPU memory, dramatically reducing prefill costs. He warns that open-source tools like LLaMA.cpp lack these optimizations, making high-performance serving of models like LLaMA 405b infeasible without libraries like vLLM or TensorRT-LLM. On scaling, Patel notes that 100,000 GPU clusters (e.g., Microsoft's Arizona data center consuming 150 MW) face reliability issues — optical transceivers fail every five minutes — and straggler chips (silicon lottery) can degrade training…

Fine tune 20 Llama Models in 5 Minutes: Santosh Radha

Feb 9, 2025 · 6:26

Santosh Radha, Head of Product/Research at Agnostiq, demonstrates Covalent, an open-source platform that lets users fine-tune and deploy hundreds of Llama models directly from Python without Kubernetes or Docker. By adding a single decorator to Python functions, users specify GPU requirements (e.g., H100 with 48 GB, 18-hour limit) and run them on remote compute, paying only for actual usage (e.g., 87 cents for 6 minutes on an L14, 11 cents on a V100). Covalent supports job submission, inference endpoints with custom autoscaling (e.g., scale to 10 GPUs at 9 AM daily), and automated workflows for training, evaluation, and deployment. Radha shows a workflow that iterates over 20 models, fine-tunes each, evaluates accuracy, sorts, and deploys the best—all from a Jupyter notebook with a single dispatch call. The talk, recorded at the AI Engineer World's Fair, emphasizes eliminating infrastructure overhead for accelerated compute.

Insights from Snorkel AI running Azure AI Infrastructure: Humza Iqbal and Lachlan Ainley

Feb 8, 2025 · 20:46

Humza Iqbal of Snorkel AI explains how the company uses Azure AI infrastructure powered by NVIDIA GPUs to fine-tune foundation models for enterprise customers, achieving better performance per dollar by switching from A100s to H100s. He details their distributed training stack (PyTorch, Horovod, NFS) and lessons learned such as balancing node count for batch size and monitoring GPU utilization to avoid networking or data-loading bottlenecks. A cost comparison found two H100s outperformed four A100s on both training and inference, enabling faster iteration through more synthetic data. Azure's dedicated VMs, reliable NFS throughput, and flexible capacity allowed Snorkel to scale experiments from single-node to dozens of GPUs. Future work includes programmatic preference signals and multimodal retrieval algorithms, all planned on Azure.

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

Jan 1, 2025 · 33:39

NVIDIA solutions architect Mark Moyou explains that LLM inference differs fundamentally from standard deep learning deployment, requiring careful management of KV Cache, attention mechanisms, and GPU memory to control cost. He details how tokens are processed: prefill computes attention across the entire prompt, then generation produces one token at a time, with KV Cache storing key-value pairs to avoid recomputation. Llama's 32 attention heads and FP8 quantization (halving memory with near-identical accuracy) are cited as key optimizations. Moyou emphasizes measuring time to first token, inter-token latency, and input/output sequence length distributions to size inference engines. He presents NVIDIA's TRT-LLM (model compilation for LLMs) and Triton inference server as tools to maximize throughput, and discusses how query patterns like long-input-short-output or short-input-long-output impact GPU utilization and deployment cost.

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

Sep 13, 2024 · 1:40:01

Philip Kiely and Pankaj Gupta of Baseten lead a workshop on TensorRT-LLM, NVIDIA's high-performance inference framework for LLMs, arguing its use delivers best-in-class throughput and latency on NVIDIA GPUs. They explain that TensorRT-LLM optimizes computational graphs via plugin kernels and in-flight batching, achieving 216 tokens per second and 180ms time to first token on Mistral 7B. The workshop demonstrates building an engine for TinyLlama 1.1B, including FP8 quantization that reduced engine size from 2 GB to 1.2 GB with minimal quality loss. They benchmark a deployed model, showing 7,000 total tokens per second at batch size 64 on an A10G. The presenters compare TensorRT-LLM favorably to VLLM for high-throughput production use and introduce Truss, Baseten's open-source packaging tool, alongside their managed platform for automatic scaling and fast cold starts.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.