Mosaic

AI Engineering Without Borders — swyx

Oct 30, 2024 · 10:32

In this talk from the AI Engineer World's Fair, host swyx argues that AI inherently disrespects human-made borders—it is naturally multilingual, multimodal, and indifferent to copyright or ground truth. He challenges the field to define its own laws, distinguishing constants (e.g., humans speak at 80 wpm vs. read at 200 wpm) from contingent facts (e.g., Apple Intelligence's 30 tokens/sec baseline). Reflecting on one year of the 'Rise of the AI Engineer,' he notes that the conference tracks—RAG, code gen, agents, multimodality—are arbitrary constructs we created, not natural categories. He proposes that AI Engineering sits between software engineering and real engineering: it must apply natural sciences for humanity's benefit. The talk concludes with a call to 'disagree more'—with your own conclusions, each other, and the status quo—and to transform the Shoggoth of raw AI into mass transit tools for society.

Building Blocks for LLM Systems & Products: Eugene Yan

Nov 2, 2023 · 17:24

Eugene Yan presents practical patterns for building LLM systems and products, emphasizing evals as the foundation. He argues that automated evals—starting with as few as 40 domain-specific questions—enable faster iteration and safer deployment, while academic benchmarks like MMLU may not fit real tasks. For retrieval-augmented generation, he warns that LLMs perform worse when the answer is in the middle of retrieved documents, and even with perfect retrieval, accuracy tops out at 75%. Guardrails against hallucination can be built via natural language inference at the sentence level or by sampling multiple summaries to check consistency. Finally, he highlights UX design—like Copilot’s accept/reject or Midjourney’s upscale/vary—as a way to collect implicit feedback that builds a data flywheel for evals and fine-tuning.