TensorRT

OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack

Jun 28, 2026 · 24:36

Lech Kalinowski presents Vault, a dual-display handheld AI terminal built on an ESP32-S3 that runs a local LLM and OpenClaw agents without cloud reliance. The device pairs a fast OLED live surface with a bistable e-paper content display, offering four modes — shell, assist, control, and an LLM-native RPG — all powered by a single lithium-polymer cell. The backend serves GPT-oss 120b via NVIDIA TensorRT-LLM, keeping inference off the microcontroller. Kalinowski shares engineering war stories: blown OLEDs from unstable power supplies, software I2C issues, and noisy encoders. The RPG mode generates four worlds with NPCs and narrative state tracked by the LLM, not dice or HP. Built in three months with 130 commits, the terminal is intentionally distraction-free, targeting quiet spaces where users want text-first AI interaction.

Accelerating AI on Edge — Chintan Parikh and Weiyi Wang, Google DeepMind

May 5, 2026 · 23:58

Chintan Parikh and Weiyi Wang from Google DeepMind present Gemma 4E edge models (2B and 4B) and the LiteRT framework for on-device AI, arguing that edge AI delivers latency, privacy, offline capability, and cost savings. The Gemma 4E models introduce agentic capabilities including built-in function calling, structured JSON output, and chain-of-thought reasoning, all optimized for hardware-native support across CPUs, GPUs, and NPUs. LiteRT supports cross-platform deployment on Android, iOS, macOS, Linux, Windows, and IoT devices like Raspberry Pi, with a CLI tool and AI Edge portal for benchmarking. Performance benchmarks show up to 56 tokens per second on iOS, 30x speedups with NPU acceleration, and 35x faster than Llama on mobile. The Gallery app demonstrates on-device skills such as Wikipedia querying, mood tracking, photo-to-music generation, and voice agents, with open-source code and a Hugging Face repository. Q&A addresses use cases like local security camera face recognition, LiteRT vs TensorRT on Orin, multi-agent architectures, and audio model support.

The End of Awkward AI Transcriptions - Travis Bartley and Myungjong Kim

Jun 3, 2025 · 16:24

NVIDIA speech AI researchers Travis Bartley, Myungjong Kim, and Jae-hun detail how their Fast Conformer backbone powers both streaming (Parakeet) and multitask (Canary) models, achieving top-5 Hugging Face ASR leaderboard rankings by prioritizing customization over one-size-fits-all. They explain CTC/TDT for low-latency streaming and attention encoder-decoder for high-accuracy multitask models, with Subformer enabling unified speaker diarization and target-speaker ASR. Training uses open-source and proprietary data with pseudo-labeling via the Nemo toolkit. Deployment on NVIDIA Riva NIM with TensorRT optimization supports low-latency streaming and offline processing. Customization includes fine-tuning acoustic models, language models, punctuation, and inverse text normalization for domain-specific terms like medical and food ordering.

Mastering LLM Inference Optimization From Theory to Cost Effective Deployment: Mark Moyou

Jan 1, 2025 · 33:39

NVIDIA solutions architect Mark Moyou explains that LLM inference differs fundamentally from standard deep learning deployment, requiring careful management of KV Cache, attention mechanisms, and GPU memory to control cost. He details how tokens are processed: prefill computes attention across the entire prompt, then generation produces one token at a time, with KV Cache storing key-value pairs to avoid recomputation. Llama's 32 attention heads and FP8 quantization (halving memory with near-identical accuracy) are cited as key optimizations. Moyou emphasizes measuring time to first token, inter-token latency, and input/output sequence length distributions to size inference engines. He presents NVIDIA's TRT-LLM (model compilation for LLMs) and Triton inference server as tools to maximize throughput, and discusses how query patterns like long-input-short-output or short-input-long-output impact GPU utilization and deployment cost.

From model weights to API endpoint with TensorRT LLM: Philip Kiely and Pankaj Gupta

Sep 13, 2024 · 1:40:01

Philip Kiely and Pankaj Gupta of Baseten lead a workshop on TensorRT-LLM, NVIDIA's high-performance inference framework for LLMs, arguing its use delivers best-in-class throughput and latency on NVIDIA GPUs. They explain that TensorRT-LLM optimizes computational graphs via plugin kernels and in-flight batching, achieving 216 tokens per second and 180ms time to first token on Mistral 7B. The workshop demonstrates building an engine for TinyLlama 1.1B, including FP8 quantization that reduced engine size from 2 GB to 1.2 GB with minimal quality loss. They benchmark a deployed model, showing 7,000 total tokens per second at batch size 64 on an A10G. The presenters compare TensorRT-LLM favorably to VLLM for high-throughput production use and introduce Truss, Baseten's open-source packaging tool, alongside their managed platform for automatic scaling and fast cold starts.

Unlocking Developer Productivity across CPU and GPU with MAX: Chris Lattner

Jul 25, 2024 · 18:33

Chris Lattner, CEO of Modular, presents MAX, a unified AI framework that accelerates Gen AI inference by combining CPU and GPU programming into a single Pythonic model, and Mojo, a new programming language that extends Python to systems programming with 100–1000x speedups. Lattner argues that current fragmentation across PyTorch, ONNX, TensorRT, and hardware-specific libraries slows innovation, and MAX replaces the entire stack—including cuDNN and Intel MKL—with a consistent, compiler-driven approach. He demonstrates that MAX's Int4/Int6 quantization achieves 5x faster performance than llama.cpp on cloud CPUs, and that its GPU matrix multiplication beats NVIDIA's cuBLAS by up to 30%. Mojo enables developers to write for loops and tokenizers (e.g., for LLaMA 3) in Python-like syntax without dropping to C++ or Rust. MAX is free and available now for CPU inference; GPU support launches in September with early access via Discord.