Ray

Your LLM Stack Is a 2008 Database With Better Marketing — Lovina Dmello, NVIDIA

Jul 20, 2026 · 20:36

NVIDIA's Lovina Dmello argues that production ML security failures stem from infrastructure misconfigurations, not exotic AI attacks, citing 2023 research finding thousands of Ray clusters exposed because authentication was off by default. An audit of 50 systems showed 78% had critical mistakes, always the same three: overprivileged accounts, flat networks, and exposed secrets or model weights. She presents a maturity model tied to overhead budgets—basic controls under 8%, selective isolation 10–20%, real-time detection 15–30%—and insists the field needs deployable defenses, not new attack-defence pairs. Dmello concludes that LLM stacks should be secured like 2008 databases: lock down access, segment networks, protect data at rest.

When Agents Meet Physical Data: The Other Physics of Agent Harnesses - Dmitry Petrov, DataChain

Jul 20, 2026 · 27:33

Dmitry Petrov of DataChain argues that AI agents fail on unstructured physical data because their intuitions assume cheap recompute, while large-scale video, sensor, and robot data requires a different harness. He cites Anthropic's finding that agents achieve only 21% accuracy on data projects without specific data harnesses, and OpenAI's need for six layers of context even on structured data. Petrov demonstrates DataChain, an open-source Python framework that uses Pydantic schemas to turn messy binary files into queryable databases, an execution engine for distributed processing, incremental checkpoints to avoid recomputing on failure, and a knowledge base of datasets and source code so agents can answer follow-up questions in seconds instead of reprocessing terabytes. In a live demo with Claude Code analyzing 90 dashcam videos, the harness took 24 minutes to extract 100,000 object records, then instantly answered 'how many clips have people?' without rerunning inference. Petrov emphasizes that the key is organizing metadata into star schemas and sharing data lineage across teammates so no one pays the compute cost twice.

Dream Machine: Scaling to 1m users in 4 days — Keegan McCallum, Luma AI

Jul 19, 2025 · 19:03

Keegan McCallum, Head of ML Infrastructure at Luma AI, details how the company's Dream Machine model scaled from 500 to 9,000 H100 GPUs within hours to handle 1 million users in four days, outpacing ChatGPT's initial growth. He explains that their initial Triton inference server setup was brittle and ill-suited for multi-GPU, multi-node video models, prompting a re-architecture to a custom serving stack on vanilla PyTorch. To solve work starvation across user tiers, they implemented an SLO-based aging system that ranks jobs by the percentage of their worst-case wait time elapsed. For managing dozens of model versions, they store immutable full Python environments and checkpoints in object storage, with a YAML file controlling active deployments and enabling zero-downtime rollouts across thousands of GPUs. McCallum also discusses partnerships with Nvidia, AMD, and Grok, and how Luma's broader mission is to build general multimodal intelligence that generates, understands, and operates in the physical world.

Domain adaptation and fine-tuning for domain-specific LLMs: Abi Aryan

Nov 14, 2023 · 25:09

Abi Aryan's talk covers domain adaptation and fine-tuning for large language models, contrasting prompting, RAGs, and three fine-tuning methods: adaptive, behavioral, and parameter-efficient. Adaptive fine-tuning adds small adapter modules (0.15% of parameters) for new domains like biochemical engineering; behavioral fine-tuning optimizes label space for a single task; and parameter-efficient methods like LoRA and QLoRA reduce model size via low-rank adaptation and four-bit precision, ideal for low-resource devices. Aryan emphasizes data quality—deduplication reduces memorization—and practical tips: batch size of 32 or 64, starting with 100 epochs, using Adam optimizer, gradient checkpointing for memory savings, and in-context learning with dynamic examples to handle drift. Evaluation should combine metric-based (Bleu, Rouge), tool-based (Weights & Biases), model-based, and human-in-the-loop approaches, though full pipeline considerations (data collection, base model choice, storage) are critical for robust applications.

[Workshop] AI Engineering 201: Inference

Nov 7, 2023 · 1:43:16

Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads.