Galileo

Multi Agent AI and Network Knowledge Graphs for Change — Ola Mabadeje, Cisco

Aug 22, 2025 · 18:49

Ola Mabadeje from Cisco's Outshift presents a multi-agent AI system that combines network knowledge graphs with natural language interfaces to improve change management. The solution ingests data from ITSM tools like ServiceNow, uses five specialized agents built on an open framework (agency.org) to automate impact assessment, test planning, and execution within a digital twin. A fine-tuned query agent reduces token usage and latency when querying the ArangoDB knowledge graph, layered using OpenConfig schema. The demo shows agents collaborating to assess a firewall rule change, generate test cases, and run them on a synthetic network, attaching results back to the ticket. Early evaluations focus on extrinsic metrics tied to customer outcomes, with the knowledge graph and open agent framework as key building blocks for scalable network operations.

2025 is the Year of Evals! Just like 2024, and 2023, and … — John Dickerson, CEO Mozilla AI

Aug 6, 2025 · 19:14

John Dickerson, CEO of Mozilla AI and former co-founder of Arthur AI, argues that 2025 is finally the year for AI evaluation (evals) due to three converging forces: ChatGPT making AI tangible to the C-suite, enterprise budget freezes that funnelled funding into GenAI pet projects, and the rise of agentic systems acting autonomously. He traces how ML monitoring existed pre-2022 but lacked C-suite attention, with Jamie Dimon’s JPMC spending only $100 million on AI from 2017 to 2021. The ChatGPT launch on November 30, 2022, unlocked CEO discretionary budgets for GenAI, leading to 2023 science projects, 2024 production deployments, and 2025 scaling. Now, as agents reason and act, connecting evals to downstream business KPIs—risk mitigation, revenue gains—is a first-class discussion for CEOs, CFOs, CISOs, and CTOs. Dickerson notes that evaluation companies have shifted to multi-agent system monitoring, and while LLM-as-judge is popular, biases require human validation, with companies like Mercor hiring experts at $50–200/hour to lockstep multi-agent outputs.

Taming Rogue AI Agents with Observability-Driven Evaluation — Jim Bennett, Galileo

Jun 27, 2025 · 16:14

Jim Bennett, Principal Developer Advocate at Galileo, argues that AI agents must be tamed using observability-driven evaluation, where LLMs evaluate other LLMs to detect failures like hallucinations and tool misuse. He cites real-world examples: the Chicago Sun-Times publishing a hallucinated summer reading list, and a lawyer citing false AI-generated case law. Bennett demonstrates a fintech chatbot that fails to answer 'what is my account balance' directly, requiring three turns; metrics like 'action completion' and 'action advancement' reveal the agent advances but does not complete. He stresses granular evaluation at every step—LLM calls, tool use, RAG retrieval—and using a better, custom-trained LLM as the evaluator. Human feedback is essential to correct mis-scored metrics and continuously retrain. Bennett urges adding evaluations from day one, even before production, and maintaining them in CI/CD and production with alerting for rogue behavior.