Navigating Challenges and Technical Debt in LLMs Deployment: Ahmed Menshawy

Hi everyone. I'll try to touch on three main things, and mainly how AI moved from excellence in structured data to LLMs and the use of unstructured data that most of our organizations have. And also we'll touch on intelligence augmentation and really the hype around AGI and the doomsday.

And finally, we'll briefly talk about the challenges and technical debt, and highlight the findings that we have published recently. So, over the last 10 to 15 years, most of the AI value that we have seen is really coming from structured data.

And we have seen supervised learning and deep learning doing really well at labeling things. But this is not the reality. Like, most of the organizations—it's estimated that most of the organizations' data is unstructured. Specifically, more than 80% of the organizations' data is unstructured.

And it's also estimated that 71% of them really struggle in managing and securing this kind of data. And it would have been ideal to really build automated systems, try to do certain recommendations based on this data. But now it's easy to really use it and have it to contextualize or customize the contextual language models.

So you can easily have this as an external memory to your language model and have it formulate answers based on the domain-specific data that you have within your organization.

And talking about the AGI and the way we see LLMs or generative AI in general at Mastercard, it's really augmenting human productivity. And we have seen a lot of hype around, you know, "generative AI is going to replace our jobs," and doomsday, and it's taken over.

And I recommend you this great article from Nature, which is really talking about—stop talking about tomorrow's AI doomsday when AI poses risks today. So stop doing speculations about what AI will become tomorrow and what kind of risks that it will have tomorrow, and really focus about the current risks that it poses today.

And funny enough, some of the big speakers about the doomsday are actually ones who have AI systems out there to the end users, with a lot of risks that we have seen in the past. And this, of course, will help regulators as well be more focused.

Like, if we highlight the current risks, it will help them more focus to have the laws and policies that can really help them regulate the current AI systems. And at the same time, be early, sort of, or proactive enough to adopt any new laws whenever new algorithmic approaches come up.

Also, when it comes to the algorithmic foundation, like, you know, AI and generative AI specifically has been transforming our lives in so many ways. But the algorithmic foundations itself behind LLMs is not really the ones that will get us to AGI.

And also, I recommend this talk from Lacun, one of the fathers of machine learning, where he talks about the objective-driven learning and the whole idea that, you know, despite the fact that it's transforming our life in so many ways, it's really SODAM at the core of it.

And it's because of the whole idea that it's autoregressive. And whenever it's making a mistake, this mistake really amplifies over time because the other generation of tokens is so dependent on what it's already generated. And I can't help but

bring this quote from Ada Lovelace, otherwise known as the world's first computer programmer. So in her 1843 Analytics Engine paper, she mentions that the analytical engine, or machine learning as we call it today, cannot originate anything by itself.

It can only do what we ask it or what we order it to perform. Because basically, we don't have this algorithmic foundation that can really get us to something that can originate something by itself. And despite being, you know, about 180 years old, this statement still holds, despite the transformations that we have in so many AI algorithms and applications.

And funny enough, like, I've met a lot of people that think OpenAI is the one behind language models. And I do hope that you folks don't share the same misconception. The whole idea of predicting the next token given a specific context is a very intuitive and simple idea that it's not only a few years old, it's a few decades old.

But what was really broken with this is the whole user interface. And a lot of folks have really misunderstood what ChatGPT is all about. So ChatGPT really fixed this whole user interface idea that you were able to naturally, as we speak, be able to prompt the LLM in a natural way and get your response.

And this is what was really broken with the language models before GPT assistants and ChatGPT specifically. Because this kind of data is really rare. And, you know, LLMs or OpenAI specifically have built their base model based on the internet-scale data.

But then in the subsequent phases before they released this GPT assistant, they had to go through outsourcing a lot of folks to really go about generating manual pairs of responses and questions and responses.

And as I said, like, you know, LLMs, despite, you know, being dumb at the core of it, it's really accelerating innovations everywhere. And we have seen great adoption in so many industries. And Mastercard is no different. So we have been de-risking this technology responsibly, of course.

And we have a recent press release in Feb where our president announced how we used LLMs, generative AI specifically, to boost fraud detection in some cases by 300%.

And to go into the last topic of my session is basically about the challenges. So let's first, you know, understand the essentials that anyone needs for building a successful GenAI application. So basically, you need to have access to a variety of foundation models.

And you need to have an environment to customize contextual LLMs. And you need to have an easy-to-use tool to build and deploy applications. So basically, all the, you know, the widely used tools that we have seen before, GenAI wasn't really applicable to the GenAI landscape.

And finally, we need to have a scalable ML infrastructure that can really help in scaling up and down, not just creating replicas, but really creating replicas at a speed that can work for our end users. And I've tried to color-code the different essentials based on the challenges that we would see in building such applications.

So access to a variety of foundation models is not so challenging. Yes, still, you need to do this kind of trade-off between cost and the model size, but it is available. And the environment to customize the language model itself is a bit challenging because, yes, most of the enterprises have their own AI environment, but it is not really something that is built for such models, such large models.

And the easy-to-use tool, I think, is the most challenging part of the whole equation. Because none of the tools that we have seen before, and most of the tools that most of you guys use now, is really as new as LLMs.

None of them has existed before. And finally, the need to have the scalable ML infrastructure is a bit of a challenge as well. And we have seen this nice curve from OpenAI where they show that the GPU compute and RAM for inference is actually getting more or greater than the compute they used for training the model itself.

And before I talk about the challenges in LLM and highlight the papers that we have recently published, I just want to bring up this really nice chart from the NEPS paper, 2015 paper. And it shows that ML code, which is at the core of building any machine learning system, is only a small fraction of what goes into building the end-to-end pipeline.

And specifically, it's less than 5% of what goes into building the end-to-end pipeline. And this is what I call, like, I've met a lot of folks during, you know, before my talk, and they think that, you know, an AI engineer is all about really, you know, connecting APIs and getting this kind of plumbing in place.

But I think it's more than that. It's really everything around this ML code box. It's really building this end-to-end pipeline, which accounts for more than 95% of the work.

Sorry. So before the challenges, so we'll just highlight the two different approaches that are widely used

in the industry. So the first one is really the closed-book approach. So you have a foundation model, you use it as it is, zero-shot or few-shot learning, or you even fine-tune it with your domain-specific data. And, you know, if you ask any of the folks in the enterprises, they will tell you, "We really have a hard time operationalizing such models because we have certain accuracy constraints."

So basically, the hallucination, and they do it very, you know, confidently. Attribution, you know, we can't really understand why the models are saying what they are saying. Stallness, they go out of date, and we have seen the different releases that come out of OpenAI.

Revision, as you know, in GDPR or even in California AI law, folks can opt out of the AI systems, and their information can't be used again for training or influencing the model decisions. So you need to be able to do the model editing.

And this is really hard in the foundation model, even if you fine-tune your model. And finally, customization. So you need to be able to customize these models with your own domain-specific data and have it really more grounded or more factual to generate information only based on your domain-specific data.

And it turned out that the solution to all of these problems is really to couple the foundation model to an external memory, also known as the RAG. So RAG, as you can see, that, you know, the original setup remains as it is, but we have added this additional context, which is coming from your domain-specific data.

And it is grounding, so it's improved the factual recall. There is a very nice paper around RAG augmentation that reduces hallucination in conversation. It kind of rhymes, but, like, it's very nice and shows how this kind of architecture really reduces the hallucination of the LLM systems.

And you can also have it up to date. So you can easily swap in, out vector indices. So you can do the revision, you can do attribution, of course, like all of the problems we have mentioned in the previous slide.

You can also do, as part of this RAG setup. So you have access to the sources coming out of your retriever. So you can easily go back and understand why the model generated certain text or certain decisions. But it's not so easy,right?

So, like, there are so many questions that need to be answered for this system really to be optimized and be able to work in production. And this is not even half of the questions that we have out there.

So mostly, how do we optimize the retriever and generator to work together? So despite, like, the mainstream kind of RAG that most of the people are doingright now is really having the retriever and the generator as two separate brains that none of them knows that each other exists.

But the actual RAG paper that was released by FAIR is actually about training these two in parallel. So you need to have access to the model parameters. And this is now, thanks to the people who are believing in the open source, is possible.

So you can have access to the model parameters, the open source model parameters. So you can fine-tune the generator to generate factual information based on what it gets from the retriever. So it's not just, you know, attaching an external memory and, you know, two sides of the brain that are totally separated.

So this is our paper. So it's very similar to the NEPS one, but it really shows the unique and different challenges that we would see in building an end-to-end LLM application. So you can see that, you know, again, the surrounding boxes around the LLM code or the adoption of foundation model is really, you know, accounts for more than 90% of what goes into building such application.

And it's not really just about, you know, if we pick one box about the domain-specific data collection, it's not just about building or generating the domain-specific data. It's also how do we preserve the access controls within our enterprises into these ecosystems.

So, like, you know, I'm sure most of the organizations that you work with have access controls. You can have access to certain systems, but not the others. So how do we make sure that we don't have a global LLM system that can really have access to all of the data that we have behind the scene?

So we need to maintain the same access controls and have certain specialized models that can work for certain tasks. And also, you know, coming back to this nature article that we need to focus about the current risks that AI poses today and how we build safeguards around it.

And this was really the core, I would say, you know, principle behind Mastercard's move to adopt LLMs. So we have the seven core principles of building responsible AI. And, you know, it's all, everything around privacy, it's around security, reliability.

And, you know, we also have this governing body and clear strategy that really enforces these core principles into the building of such LLM applications. So yes, we can go about really de-risking new technologies, such as LLMs, and use it for some of the services that we have.

But at the same time, we need to have theright safeguards to really make sure that, you know, the access controls are in place and also we are not, you know, generating any biased information. And so, funny enough, one of the reviewers, one of the reviewers who accepted this paper mentioned that, you know, after he read the paper, he was wondering if LLM is theright tool to use for solving some of the applications, given the huge number of challenges and technical debt that he has seen.

But as the saying goes, like, you can't make an omelet without really breaking a few eggs. So you can't really use this kind of transformative technology in your business without really being challenged in so many ways.

And that's all I have for you. And do check out some of the boxes that we have from the AI Engineering team from Mastercard. It's all about putting AI in production and the whole other boxes around ML code or the LLM that we have seen in the figures.

Thank you.