Palo Alto Networks

The ROI of AI: Why you need Eval Framework - Beyang Liu

Dec 31, 2024 · 25:28

Beyang Liu, CTO of Sourcegraph, presents six evaluation frameworks for measuring the ROI of AI coding tools, arguing that no single metric suffices because AI ROI reduces to the intractable problem of developer productivity. He dismisses the 'roles eliminated' framework as irrelevant for engineering, noting that backlogs are never fully addressed. Instead, he highlights customer examples: A/B testing velocity at Palo Alto Networks showed 20-30% acceleration with Cody; time saved as a function of engagement lower-bounds gains (e.g., each code search saves ~2 minutes); and LITOS tracked value-add zones (building features, writing tests, reviewing code) instead of lines of code. For key initiatives like code migrations, ROI is measured by how many months the migration is pulled forward. Liu also advocates developer surveys bounded by a 5-25% productivity tool budget, and warns against the Mythical Man Month fallacy: preferring one 100x human developer over 100 mediocre AI agents.

Enhancing Quality and Security in CI: Gunjan Patel

Nov 27, 2024 · 18:27

Gunjan Patel, Director of Engineering at Palo Alto Networks, presents Ghost Pilot, an AI-powered CI pipeline that outsources boring software development tasks to enable self-evolving code. Unlike real-time Copilots, Ghost Pilot operates as a 'slow system' in CI, iteratively improving variable names and code comments, then generating unit tests by first listing edge cases and personalizing them via team context files. For security, it simulates three AI roles—Red Team engineer, developer, and engineering manager—who debate identified vulnerabilities, prioritize fixes by risk and effort, and propose changes with citations. Patel shares a reusable CI template with a Bring-Your-Own-LLM option, demonstrated by catching a logical Kubernetes bug missed by static analysis tools.