GPT-2

Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

May 4, 2026 · 1:21:26

Angelos Perivolaropoulos from ElevenLabs walks through building a small GPT-2-like LLM from scratch on a local machine, demonstrating that the core techniques used by major labs are accessible in a few hundred lines of PyTorch code. The workshop uses character-level tokenization (65 tokens) on a Shakespeare dataset to enable fast training with limited compute. The model architecture includes multi-head self-attention, MLP layers, residual connections, and layer normalization, totaling 10 million parameters across six transformer blocks with a 256-token context window. The training loop employs next-token prediction with a warm-up cosine decay learning rate schedule and validation loss to detect overfitting. Inference uses temperature sampling (default 0.7) and top-k sampling to improve creativity. Perivolaropoulos explains that audio and multimodal models share the same transformer foundation but differ in tokenization (e.g., mel-spectrograms for audio) and use specialized losses like L2 or KL divergence, while reasoning models result from post-training base models with high-quality chain-of-thought data.

How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS - Ishan Anand

Jul 13, 2025 · 1:41:34

Ishan Anand shows that GPT-2 small implemented in 600 lines of vanilla JavaScript makes LLMs understandable for web developers without ML backgrounds. He explains tokenization via byte-pair encoding, 768-dimensional embeddings representing semantic meaning via co-occurrence, and the Transformer's attention mechanism that lets tokens share context. The multi-layer perceptron learns next-token prediction through backpropagation, while the language head converts embeddings to token probabilities using softmax. Anand demonstrates each step—tokenization, embedding lookup, positional encoding, attention, MLP, and output—in a browser debugger, and notes that GPT-2's architecture underpins ChatGPT, with innovations like scale, supervised fine-tuning, and RLHF. The workshop provides an intuitive mental model of Transformers, turning perceived AI magic into understandable machinery.

Rethinking how we Scaffold AI Agents - Rahul Sengottuvelu, Ramp

Mar 19, 2025 · 16:32

Rahul Sengottuvelu, Head of Applied AI at Ramp and co-founder of Cohere.io, argues that building AI agents should follow the 'bitter lesson' from AI research: systems that scale with compute outperform handcrafted, deterministic code. He illustrates this with Ramp's switching report agent, which ingests arbitrary CSV files from third-party card providers. Three approaches are compared: manually coding parsers for the 50 most common vendors, using LLMs only for column classification, and a fully LLM-driven method where the model writes and runs pandas code via a code interpreter, repeated 50 times in parallel. The last approach, though using 10,000x more compute, costs under a dollar and generalizes better, saving Ramp far more in failed transactions. Sengottuvelu also demonstrates a prototype email client where the backend is an LLM with access to a code interpreter and the user's Gmail token; the LLM renders the UI as Markdown and handles clicks by re-prompting itself, simulating a full web app without traditional backend code. He contends that as models exponentially improve, shifting more execution into 'fuzzy' LLM compute—rather than rigid code—lets builders ride the trend for…

Low Level Technicals of LLMs: Daniel Han

Jul 31, 2024 · 2:52:26

Daniel Han of Unsloth explains how to find and fix bugs in open-source LLMs like Gemma, Phi-3, and Llama, covering tokenizer issues, architecture pitfalls, and finetuning optimizations. He details the eight Gemma bugs Unsloth fixed, including a critical RoPE downcasting error that broke positional encoding, and a 2048 sliding window bug in Phi-3. Han walks through transformer internals: attention masking, layer norms, RoPE embeddings, and SwiGLU activation, showing how to derive gradients for custom kernels. He demonstrates Unsloth's 2x faster finetuning with 70% less memory via Triton kernels, and introduces new features: automatic Ollama model file creation, CSV fine-tuning with merged columns, and chunked cross-entropy for large vocabularies. The session includes live Q&A on learning rate schedules, precision trade-offs, and mechanistic interpretability.