GPT-4 Turbo

System Design for Next-Gen Frontier Models — Dylan Patel, SemiAnalysis

Feb 11, 2025 · 18:29

Dylan Patel of SemiAnalysis breaks down the inference challenges for next-generation frontier models like GPT-4 (1.8 trillion parameters) and upcoming models trained on 100,000+ GPU clusters. He emphasizes that prefill (prompt processing) is compute-intensive while decode (token generation) is memory bandwidth-intensive, creating a systems problem where serving 64 users at 30 tokens/second requires 60 terabytes/second of memory bandwidth. Patel details engineering strategies such as continuous batching to improve batch utilization by 10-100x, disaggregated prefill to isolate noisy neighbors and maintain time-to-first-token SLAs, and context caching (like Google's) to cache KV cache on CPU/storage instead of GPU memory, dramatically reducing prefill costs. He warns that open-source tools like LLaMA.cpp lack these optimizations, making high-performance serving of models like LLaMA 405b infeasible without libraries like vLLM or TensorRT-LLM. On scaling, Patel notes that 100,000 GPU clusters (e.g., Microsoft's Arizona data center consuming 150 MW) face reliability issues — optical transceivers fail every five minutes — and straggler chips (silicon lottery) can degrade training…

[Full Workshop from Microsoft] Github Copilot - The World's Most Widely Adopted AI Developer Tool

Feb 7, 2025 · 1:19:45

This workshop from the AI Engineer World's Fair features GitHub's Christina Warren, Dave, Alex, and Harald presenting GitHub Copilot, the world's most widely adopted AI developer tool. They demonstrate Copilot's three interaction modes: ghost-text completions using GPT-3.5 for speed, inline chat (also GPT-3.5) for quick code edits, and the chat panel powered by GPT-4 Turbo for deeper conversations. Speakers emphasize prompt crafting—being specific, providing examples, and keeping relevant files open—to improve results. They show how to use slash commands (/fix, /explain, /test) and the new attach button for explicit context. The workshop uses GitHub Codespaces preconfigured with Python 3.11 and Copilot extensions; a coupon provides a 7-day free trial. Harald explains the trade-off between model quality and latency, and notes upcoming features like automatic agent delegation and workspace integration for cross-file edits.

Multi model multimodal and multi agent innovations in Azure AI: Cedric Vidal

Feb 6, 2025 · 28:56

Cedric Vidal, Principal AI Advocate at Microsoft, demonstrates Azure AI's multi-model, multimodal, and multi-agent capabilities in a session packed with live demos. He shows how GPT-4 Omni mixes text and vision to read handwritten French menus and translate them, and diagnoses infrastructure damage from photos for energy and insurance industries. A new video translation service lets him speak German, Spanish, Italian, and Japanese in his own voice while preserving tone, such as whispering or yelling. The Azure AI model catalog now offers 1,600 models, including serverless deployment options, and Phi-3 Vision, a small 3.8B parameter model running locally in a browser via WebGPU. He also demonstrates code interpreter analyzing a GPX file from a kitesurfing session, plotting a map with turn markers, and GitHub Workspaces preview generating a Java GUI from Python code.

Creating and scaling your own custom copilots with Azure AI Studio: Hanchi Wang

Feb 6, 2025 · 24:21

Hanchi Wang, Software Engineer Lead at Azure AI, introduces Azure AI Studio and Promptflow for creating and scaling custom copilots, focusing on tracing, evaluation, and monitoring. He demonstrates a chatbot app that uses the Assistant API with a sales data insight tool (natural language to SQL) and a code interpreter. With Promptflow's trace decorator, developers capture inputs, outputs, and LLM interactions, viewable in a local UI and shareable via Azure AI Studio. For evaluation, he shows synthetic test data sets, content safety evaluators, and custom evaluators like execution time, error rate, and SQL similarity, comparing models such as GPT-4 Turbo, Mistral large, and Phi-3. In production, monitoring dashboards in Application Insights track model duration, token usage (prompt vs. completion), and failure rates, enabling engineers to optimize performance and cost.

How Zapier Builds AI Products and Features with the Help of Braintrust: Ankur Goyal & Olmo Maldonado

Nov 7, 2024 · 14:59

In this talk, Olmo Maldonado (Senior AI Engineer at Zapier) and Ankur Goyal (CEO of Braintrust) explain how Zapier uses Braintrust's evaluation and observability platform to build and improve AI features like AI Zap Builder and Copilot. They moved from 7 manual unit tests to over 800 automated evals, improving accuracy by nearly 300%. Braintrust's tracing capabilities let them dissect Copilot's multi-tool agent performance, leading them to adopt GPT-4 Turbo for the message router despite latency trade-offs. When switching to GPT-4.0 caused a regression below 80% scores, they diagnosed the issue via evals—GPT-4.0 was ignoring system prompts—and fixed it by relaxing prompt engineering and adjusting tool choices, recovering accuracy and cutting response time from 14 to 3 seconds. The episode details how eval-driven development and observability enable rapid iteration and reliable AI products at scale.

Building State of the Art Open Weights Tool Use: The Command R Family: Sandra Kublik

Aug 26, 2024 · 15:03

Sandra Kublik of Cohere presents the Command R family of open-weight models optimized for retrieval-augmented generation and tool use. Released in March 2024, Command R and Command R+ achieved 150,000 Hugging Face downloads within two weeks and now serve nearly 500,000 developers. The models overcome RAG challenges such as prompt sensitivity and citation accuracy through post-training, delivering fine-grained citations and low hallucination. Cohere open-sourced a toolkit UI with plug-and-play components for RAG and tool use, supporting cloud, local, and Hugging Face access. The new Multi-Step API enables sequential reasoning with automatic retry and reflection. Command R+ matches GPT-4 Turbo and Claude Opus on complex reasoning while being three to five times cheaper, positioning it as a scalable enterprise solution.

From Text to Vision to Voice Exploring Multimodality with Open AI: Romain Huet

Jul 10, 2024 · 23:39

OpenAI's Romain Huet demonstrates GPT-4o's real-time audio, vision, and screen-sharing capabilities at the AI Engineer World's Fair, showing how the omni-model achieves human-like conversational latency, understands emotion, and can be interrupted naturally. He walks through live demos of ChatGPT Desktop recognizing drawings, reading book pages, and pair-coding a responsive travel app using Tailwind CSS. Huet also previews Sora's text-to-video generation and OpenAI's unreleased Voice Engine, which can clone a voice from seconds of audio and translate it into multiple languages while preserving the original speaker's tone. He outlines OpenAI's four focus areas: advancing textual intelligence, faster/cheaper models (GPT-4o is twice as fast as GPT-4 Turbo at half the price), model customization via fine-tuning, and enabling multimodal agents. The talk emphasizes that today's models are the dumbest they'll ever be and urges developers to build for a future where AI works across text, voice, and video.