Command R Plus

Training Albatross An Expert Finance LLM: Leo Pekelis

Feb 13, 2025 · 16:20

Leo Pekelis, chief scientist at Gradient, explains how they transformed an open-source model into Albatross, a finance LLM that tops leaderboards on both general and domain-specific tasks. The key was an automated data pipeline using membership inference to curate finance data from a massive corpus, followed by continual pre-training and alignment via supervised fine-tuning and preference optimization. He also details a one-million-token context extension on a Llama 3-based model that achieves 100% needle-in-the-haystack scores, enabling in-context learning with thousands of examples to reduce hallucinations. The models, v-alpha-tross and the extended-context Llama 3, are open-sourced on Hugging Face.

Open Challenges for AI Engineering: Simon Willison

Jul 17, 2024 · 18:49

Simon Willison argues the GPT-4 barrier has been broken as GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and open models like LLaMA 3 70B now compete, making GPT-4-class models a commodity. He highlights the AI trust crisis with examples of Dropbox and Slack being falsely accused of training on user data, and notes Anthropic trained Claude 3.5 Sonnet without customer data. Willison warns about prompt injection vulnerabilities, citing the Markdown image exfiltration bug affecting six major chatbots, and defines slop as unreviewed AI-generated content, calling for accountability and responsible use patterns.