Falcon

Lessons from Trillion Token Deployments at Fortune 500s — Alessandro Cappelli, Adaptive ML

May 12, 2026 · 18:35

Alessandro Cappelli, co-founder of Adaptive ML, argues that 95% of GenAI pilots fail to reach production because they rely on proprietary models or instruction fine-tuning, which lack systematic feedback integration. Reinforcement learning (RL) is the only post-training technique that mathematically incorporates defects, business metrics, and production signals to continuously improve models. RL enables smaller, cheaper, faster models that enterprises can own—critical for scaling use cases like AT&T’s transcript summarization or Manulife’s agents. For agents, RL naturally fits because it was designed for environments; synthetic data is generated as a byproduct of environment training, not a prerequisite. Reward signals come from business KPIs (e.g., containment rate) or LLM judges defined by human rubrics in hours, not weeks. Adaptive ML’s Adaptive Engine abstracts away RL complexity (orchestrating four models for PPO) and provides pre-built recipes to industrialize model deployment.

[Full Workshop] Llama 3 at 1,000 tok/s on the SambaNova AI Platform

Feb 7, 2025 · 1:00:58

Michelle Matern and Petro Milan of SambaNova present their full-stack AI platform, built on the SN40L RDU chip with a three-tiered memory architecture capable of storing up to 5 trillion parameters. They demonstrate Samba1 Composition of Experts (CoE), a trillion-parameter model combining 92 expert models behind a single endpoint, and show Llama-3-8B achieving 1,000 tokens per second with a time-to-first-token of 0.09 seconds and total inference time of 0.65 seconds—far exceeding GPU-based providers. The workshop includes a hands-on basic inference call using LangChain and SambaStudio API, and a RAG-based Q&A system for enterprise search that integrates Unstructured for document loading, E5-large-v2 embeddings, ChromaDB vector store, and the high-speed Llama-3 endpoint. Attendees learn to configure prompts with special Llama-3 tags, set chunk size and overlap, and optionally run embeddings on SambaNova's RDU hardware for faster processing.

How to evaluate a model for your use case: Emmanuel Turlay

Feb 5, 2025 · 7:32

Emmanuel Turlay, CEO of Sematic, explains why evaluating large language models for specific use cases is more difficult than traditional ML evaluation, as metrics like BLEU and ROUGE and benchmarks like GLUE do not measure task-specific performance. He advocates using another LLM as a grader, describing a workflow where a scoring prompt with grading criteria is fed to a model like GPT-4 (best but costly) or FLAN-T5 (good speed–accuracy trade-off) to numerically score outputs. Turlay demonstrates with a politeness evaluation for email closings and introduces AirTrain, a platform that lets users upload datasets, compare models, and visualize metric distributions to make data-driven LLM selections. The episode argues that teams must build custom evaluation procedures rather than relying on generic benchmarks, treating evaluation as a test suite in the ML development pipeline.

Build enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform

Sep 11, 2024 · 54:34

SambaNova’s Michelle Matern and Petro Milan present their full-stack AI platform, demonstrating how the SN40L RDU chip enables Llama 3 inference at 1,000 tokens per second. They introduce Samba-1, a composition of 92 expert models behind a single endpoint, and benchmark it against GPT-3.5 and GPT-4 on enterprise tasks like information extraction and text-to-SQL. The workshop then builds a RAG-based Q&A system using LangChain, Unstructured, E5-large-v2 embeddings, ChromaDB, and Llama-3-8B-Instruct at 1,000 tokens per second. Attendees set up the environment, load documents, and run inference with real-time metrics showing time to first token of 0.09 seconds and total inference time of 0.65 seconds.

Harnessing the Power of LLMs Locally: Mithun Hunsur

Nov 22, 2023 · 17:09

Mithun Hunsur presents llm.rs, a Rust library for running large language models locally, arguing it gives developers ownership, lower latency, and privacy compared to cloud APIs. He explains how quantization makes inference viable on consumer hardware, and shows that llm.rs supports architectures like LLaMA and Falcon through a unified interface. Practical code examples demonstrate customization, and community projects like LocalAI and LLMchain illustrate real-world use. Hunsur shares his own date-extraction pipeline, fine-tuning a small model with GPT-3 data to replace expensive cloud calls. He also cautions about hardware requirements, trade-offs between speed and quality, and ecosystem churn from rapid innovation.