NanoGPT

An AI Agent Became the #1 Contributor in OpenAI's Hiring Challenge — Zhengyao Jiang, Weco

Jul 16, 2026 · 16:16

Zhengyao Jiang, CEO of Weco AI, details how their autonomous agent Aiden became the top contributor in OpenAI's Parameter Golf competition, setting 7 leaderboard records—more than twice any human's 3. Over 22 days, Aiden ran 1,300 experiments on a single H100 node, using just 4% of total compute while contributing 15% of records and achieving a 28% leaderboard hit rate (six times the community average). The agent excelled at finding and implementing ideas from human researchers, combining them (e.g., gated attention from a paper with a tokenizer improvement) to produce breakthrough results. Jiang argues that auto-research does not replace human creativity but shifts it upward: humans design evals and code abstractions that bias the agent's search, a new craft he compares to training neural networks. The talk reveals that Aiden's H-index in the community was 10 versus the next human's 7, proving autonomous agents can produce work that others build on.

Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

May 4, 2026 · 1:21:26

Angelos Perivolaropoulos from ElevenLabs walks through building a small GPT-2-like LLM from scratch on a local machine, demonstrating that the core techniques used by major labs are accessible in a few hundred lines of PyTorch code. The workshop uses character-level tokenization (65 tokens) on a Shakespeare dataset to enable fast training with limited compute. The model architecture includes multi-head self-attention, MLP layers, residual connections, and layer normalization, totaling 10 million parameters across six transformer blocks with a 256-token context window. The training loop employs next-token prediction with a warm-up cosine decay learning rate schedule and validation loss to detect overfitting. Inference uses temperature sampling (default 0.7) and top-k sampling to improve creativity. Perivolaropoulos explains that audio and multimodal models share the same transformer foundation but differ in tokenization (e.g., mel-spectrograms for audio) and use specialized losses like L2 or KL divergence, while reasoning models result from post-training base models with high-quality chain-of-thought data.