Groq

[Full Workshop] Llama 3 at 1,000 tok/s on the SambaNova AI Platform

Feb 7, 2025 · 1:00:58

Michelle Matern and Petro Milan of SambaNova present their full-stack AI platform, built on the SN40L RDU chip with a three-tiered memory architecture capable of storing up to 5 trillion parameters. They demonstrate Samba1 Composition of Experts (CoE), a trillion-parameter model combining 92 expert models behind a single endpoint, and show Llama-3-8B achieving 1,000 tokens per second with a time-to-first-token of 0.09 seconds and total inference time of 0.65 seconds—far exceeding GPU-based providers. The workshop includes a hands-on basic inference call using LangChain and SambaStudio API, and a RAG-based Q&A system for enterprise search that integrates Unstructured for document loading, E5-large-v2 embeddings, ChromaDB vector store, and the high-speed Llama-3 endpoint. Attendees learn to configure prompts with special Llama-3 tags, set chunk size and overlap, and optionally run embeddings on SambaNova's RDU hardware for faster processing.

Breaking AI's 1-GHz Barrier: Sunny Madra (Groq)

Oct 10, 2024 · 20:11

Sunny Madra (Groq) argues that LLM inference speed is undergoing a transformation akin to the 1-GHz microprocessor breakthrough, with Groq increasing LLaMA 3 8B speed by over 50% in just two months. He draws parallels to the industrial revolution: just as mass production replaced bespoke manufacturing, AI now enables 1,000 outputs in a minute where previously one designer produced one per day. Specific applications like Globe.Engineer plan a trip in seconds by processing 10,000 input tokens per second. Madra envisions LLMs as future OS cores, enabling instantaneous decision-making, universal natural-language interfaces, and multi-agent collaboration that allows smaller models to compete with larger ones. Edge AI projects like hyperspace.ai distribute unused GPU compute, and personalized education (citing Khan's two-sigma effect) becomes practical with cheap inference. He also highlights automated data science agents (Pioneer) that run endlessly, discovering correlations like productivity decline after certain performance reviews. The episode underscores that speed unlocks predictive analytics, context-aware systems, and enhanced security against AI-powered scams.