ChromaDB

The 100-Tool Agent Is a Trap - Sohail Shaikh & Ankush Rastogi, Prosodica

Jun 28, 2026 · 28:27

Sohail Shaikh and Ankush Rastogi of Prosodica argue that the common 'Fat Agent' architecture—loading every tool schema into every prompt—causes latency, cost, and accuracy collapses as tool catalogs grow. Their benchmarks show tool selection accuracy dropping from 78% at 10 tools to 13.6% at 741 tools, while their Semantic Tool Router pattern stays above 83% across the same range. The router uses Just-in-Time Context Injection, embedding tool descriptions offline, retrieving 3–5 relevant tools per query at runtime, and injecting only those schemas into the model call. This reduces input tokens by up to 99% (from 127k to ~1k tokens) and cuts time-to-first-token latency by up to 90% in high-tool-density scenarios. They present a production checklist and emphasize that routing pays off once beyond 50 tools, with K=5 as a strong default, but warn of router misses and weak descriptions needing fallbacks and monitoring.

[Full Workshop] Llama 3 at 1,000 tok/s on the SambaNova AI Platform

Feb 7, 2025 · 1:00:58

Michelle Matern and Petro Milan of SambaNova present their full-stack AI platform, built on the SN40L RDU chip with a three-tiered memory architecture capable of storing up to 5 trillion parameters. They demonstrate Samba1 Composition of Experts (CoE), a trillion-parameter model combining 92 expert models behind a single endpoint, and show Llama-3-8B achieving 1,000 tokens per second with a time-to-first-token of 0.09 seconds and total inference time of 0.65 seconds—far exceeding GPU-based providers. The workshop includes a hands-on basic inference call using LangChain and SambaStudio API, and a RAG-based Q&A system for enterprise search that integrates Unstructured for document loading, E5-large-v2 embeddings, ChromaDB vector store, and the high-speed Llama-3 endpoint. Attendees learn to configure prompts with special Llama-3 tags, set chunk size and overlap, and optionally run embeddings on SambaNova's RDU hardware for faster processing.

Build enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform

Sep 11, 2024 · 54:34

SambaNova’s Michelle Matern and Petro Milan present their full-stack AI platform, demonstrating how the SN40L RDU chip enables Llama 3 inference at 1,000 tokens per second. They introduce Samba-1, a composition of 92 expert models behind a single endpoint, and benchmark it against GPT-3.5 and GPT-4 on enterprise tasks like information extraction and text-to-SQL. The workshop then builds a RAG-based Q&A system using LangChain, Unstructured, E5-large-v2 embeddings, ChromaDB, and Llama-3-8B-Instruct at 1,000 tokens per second. Attendees set up the environment, load documents, and run inference with real-time metrics showing time to first token of 0.09 seconds and total inference time of 0.65 seconds.