Ishan Anand

How LLMs work for Web Devs: GPT in 600 lines of Vanilla JS - Ishan Anand

Jul 13, 2025 · 1:41:34

Ishan Anand shows that GPT-2 small implemented in 600 lines of vanilla JavaScript makes LLMs understandable for web developers without ML backgrounds. He explains tokenization via byte-pair encoding, 768-dimensional embeddings representing semantic meaning via co-occurrence, and the Transformer's attention mechanism that lets tokens share context. The multi-layer perceptron learns next-token prediction through backpropagation, while the language head converts embeddings to token probabilities using softmax. Anand demonstrates each step—tokenization, embedding lookup, positional encoding, attention, MLP, and output—in a browser debugger, and notes that GPT-2's architecture underpins ChatGPT, with innovations like scale, supervised fine-tuning, and RLHF. The workshop provides an intuitive mental model of Transformers, turning perceived AI magic into understandable machinery.

Decoding the Decoder LLM without de code: Ishan Anand

Aug 9, 2024 · 17:08

Ishan Anand demonstrates how decoder-based LLMs like GPT-2 work internally using a fully functional spreadsheet implementation. He explains tokenization (e.g., 're-injury' splitting into 'rain' and 'injury' due to subword units), 768-dimensional embeddings, multi-headed attention where tokens look backward (e.g., 'he' focuses 0.48 on 'Mike'), and the multi-layer perceptron with matrix multiplies (MMULT). He shows residual connections creating an information superhighway and uses logit lens to reveal layer-by-layer predictions (e.g., 'Wednesday' appears early but only rises to top probability at later layers). Finally, he injects a sparse autoencoder feature vector for 'Jedi' into the residual stream, steering GPT-2 to output 'lightsaber' instead of 'phone', and contrasts this with alternative steering methods like representation engineering and activation steering.