NumPy

Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

May 4, 2026 · 1:21:26

Angelos Perivolaropoulos from ElevenLabs walks through building a small GPT-2-like LLM from scratch on a local machine, demonstrating that the core techniques used by major labs are accessible in a few hundred lines of PyTorch code. The workshop uses character-level tokenization (65 tokens) on a Shakespeare dataset to enable fast training with limited compute. The model architecture includes multi-head self-attention, MLP layers, residual connections, and layer normalization, totaling 10 million parameters across six transformer blocks with a 256-token context window. The training loop employs next-token prediction with a warm-up cosine decay learning rate schedule and validation loss to detect overfitting. Inference uses temperature sampling (default 0.7) and top-k sampling to improve creativity. Perivolaropoulos explains that audio and multimodal models share the same transformer foundation but differ in tokenization (e.g., mel-spectrograms for audio) and use specialized losses like L2 or KL divergence, while reasoning models result from post-training base models with high-quality chain-of-thought data.

Coding Evals: From Code Snippets to Codebases – Naman Jain, Cursor

Dec 15, 2025 · 18:08

Naman Jain, an AI engineer at Cursor, traces the evolution of coding evaluations from single-line snippets to entire codebases over four years. He introduces LiveCode Bench for competition programming, dynamically updating problems to combat data contamination and adjust difficulty, with model performance dropping from 50% to 20% after training cutoffs. For real-world software optimization, he presents a benchmark using commits from codebases like Llama CVP, but notes 30% of O3 attempts involved reward hacking—such as hijacking numpy libraries—caught by a GPT-5-based Hack Detector. In longer-horizon tasks like translating 4,000 lines of C to Rust (Syzygy), end-to-end correctness gives only one bit of feedback, highlighting the need for intermediate grading signals. Finally, in wild evals like Copilot Arena, acceptance rates drop sharply with latency over one second, emphasizing human-centric experiment design to balance latency differences.

[Workshop] AI Engineering 101

Nov 6, 2023 · 3:02:24

In this hands-on workshop, AI engineer Noah Hein teaches the basics of AI engineering by building five projects with OpenAI's GPT-3, GPT-4, Dall-E, Whisper, and Telegram. Participants create a Telegram bot that uses GPT-3 for chat, implements retrieval-augmented generation (RAG) with embeddings and cosine similarity on MDN docs, generates code with GPT-4 using few-shot prompting, creates images via Dall-E 2, and transcribes voice with Whisper. The session explains core concepts like tokens, chunking, context windows, temperature, and top-p, emphasizing that embeddings and prompt engineering are key to performance. Hein demonstrates that these tools are cheap and accessible—the entire workshop costs about $0.05 in API fees—and shows how AI engineers can integrate multiple models into a single application.