Navigating Challenges and Technical Debt in LLMs Deployment: Ahmed Menshawy
Dec 31, 2024 · 16:15
Ahmed Menshawy, VP of AI Engineering at Mastercard, argues that deploying LLMs requires overcoming massive technical debt, with over 95% of the work outside the ML code. He details how Mastercard boosted fraud detection by 300% using LLMs while adhering to seven responsible AI principles. The episode covers RAG as a solution for hallucination, attribution, and access control, noting that 80% of organizational data is unstructured and 71% of companies struggle to manage it. Menshawy highlights that GPU compute for inference now exceeds training, and that fine-tuning open-source models enables joint training of retriever and generator. He also cautions against AGI hype, quoting Ada Lovelace and a Nature article urging focus on current AI risks.
