Lessons from building GenAI based applications — Juan Peredo

Hello and welcome. Today we're going to talk about the lessons I've learned from building GenAI applications over the last year and a half.

My name is Juan Peredo, and I'll be your guide in this journey. I'm a founder, architect, consultant, developer, and everything in between, and I have over 15 years' experience in the IT industry.

And one of the first things everyone hears when they're talking about AI is about how much faster developers can code thanks to AI. You can hear this, and you've heard this before, I can guarantee that nowadays you can just take a prompt, put it into an LLM, and the LLM will code your website for you in about a minute or so.

And we all know that that's not true,right? It'll give you some code and it'll get you going, but it won't build everything for you.

But there's a lot of tools that have been created to help you with this, like some of the ones I have listed here, like Codium and Cursor, that interact with your IDE or an IDE all by themselves, that help you build your applications.

And you don't even have to use those tools,right? You can just take your code, paste it into an LLM, and ask a question. And there's some really good ones out there. But it all depends on what type of application you're trying to build,right?

If all you're doing is creating boilerplate for yet another website, then any coding LLM will work just fine. If you're doing something more complex, or if it's something that's newer or just came out, like, I don't know, Svelte 5 for example, then you want a beefier LLM, something like DeepSeek V3 or Claude,right?

And these tools will definitely help you build faster. However, they also

add complexity to your application once you're trying to integrate AI to your application to actually interact with your user. Like here in the left-hand side, we still see that in your typical application that we've been building until a year ago or two years ago, we were building frontends, backends, infrastructure as code, all the stuff that we normally build.

But the moment you add AI to your application, you have to start worrying about all the stuff that's in gray in the diagram on theright. So you have to start wondering about what model you're going to use, are you going to fine-tune it, are you going to

do some prompt engineering, or are you going to try to use RAG to get the correct answer to your questions and to the user questions. How are you going to prevent it from hallucinating? Because we all know that LLMs are so bent on answering your questions that sometimes they'll just make up answers.

And you need to have a way to figure out whether the answer that you're getting isright or not. And so how do you prevent that? That's something you didn't have to worry about a year and a half ago.

Also, you have to figure out how often you're going to have to

replace that model. Because we all know that there's models that are coming out every other week, and they're all more powerful, beefier, but they will not all work with your use case. It's one thing that they're really great at certain benchmarks, but that doesn't mean they're going to work with the rest of the application that you've built.

So you have to constantly be evaluating not only the model that you have already running, but also the models that are coming out that may help you build a better application. And you also have to start figuring out how you're going to host these things now.

Because that's beyond hosting just your application,right? All of a sudden, you don't have to worry just about CPU and memory. All of a sudden, you have to start worrying about GPU so well,right? Where are you going to host this?

Is it going to be in the cloud? Is it going to be on-prem? All this information and all this stuff in gray is new to

and is specific to AI-based applications.

Speaking of hosting, where to host this model is important,right? How are you going to host it? Are you going to host it on-prem, or are you going to host it on your machine while you're doing some exploration? You have to make these decisions,right?

So if you're going to host it locally, you can host it in something like Ollama. Ollama will allow you to download a bunch of models into your machine where you can play with those models for free, interact with them, and see if they're going to work for you.

And there's a host of other solutions as well. Ollama is the one I've been using. It's pretty good. Just you just have to make sure you have a machine that's beefy enough for the model that you're trying to run.

Or you can even use this Ollama and all these other tools to host things

on-prem,right? But if you're going to host it in the cloud, now you have to start thinking not only about what cloud, but also about the cost of your GPUs and is there enough GPUs to go around. And in general, like everything else, you still also have to worry about cost.

And you have the typical players,right? You have your Google Cloud, your AWS, your Azure, etc., etc. But there's also a new set of players that have come up that are meant to simplify the deployment of your application in the cloud.

And that's where tools like Model and SkyPilot

have come up.

And if you're using Python, which a lot of us are using Python for interacting with our models, Model is a great choice. Model allows you to take your Python code, add some

decorators to your code,

and use their CLI to deploy to Model. And Model will take care of building your containers and pushing them to the cloud.

It's really simple. I've used it to host a lot of my models.

And there's also SkyPilot. SkyPilot is great to minimize your cost as they will spread your clusters and your containers into multiple clouds in order to minimize your cost.

And you can configure it on which cloud you want SkyPilot to use for hosting your application.

And at the end of the day,right, if you just want to play with stuff on your machine, you can also use a simple library like the Hugging Face Transformers that you just can run locally on your machine without any other tools.

But at the end of the day, once again, how you host these models is very important.

Now let's think about

the fact that we're not in 2023 anymore,right? Building a chatbot powered by AI is very simple. You can build it in an hour or two, if that much,right? There's a lot of frameworks there that will help you build this for you.

However, even though building the chatbot is easy, making sure that that chatbot gives you the correct answers is really hard. Like for example, this that I have on the screen on theright is actually something that happened to me.

I put in one LLM a few months ago now, what is 3 times 3, and the answer that it gave me was 33. That's obviously notright. So while making the chatbot easy, validating and moderating the chatbot content and making sure that it's giving you the correct answer to your user is really, really hard.

So what can we do to ensure that we are providing theright answer to our users,right? Well, there's a number of techniques. You can do prompt engineering, which is basically you take the question of the user, you put additional details into the prompt that you're passing to the

LLM.

However, you need to be very precise in those additional instructions

so that the LLM can follow them properly. And even then, the LLM may choose to ignore them. And at that point, what do you do,right? You get the wrong answer. You can also put guardrails. And guardrails basically you take a secondary LLM and put it to run in front or before your main LLM.

And this secondary LLM is basically a classifier that will classify the question that the person is answering

as a safe question or as a bad question. And when I say bad question, it could be something that is trying to go around

or asking for criminal activities, or it may be asking about things that are not what you'd expect somebody to answer in general,right? And at that point, the secondary LLM is classifying your questions. And it also could be the answers of your own LLM as either good or bad.

However, adding this secondary LLM adds latency because you're making a secondary call,right? And it also adds additional cost. And like anything else with LLMs, it may or may not beright.

The next thing you could do is you could have Retrieval Augmented Generation, or RAG, which is basically you take a lot of information that you have in your company, you break it into small chunks of information. So you can think of it as, let's say, you take a document and you break it into paragraphs, although it could be anything.

Just think of a chunk as a paragraph for now. You break it into chunks and you store it in a vector database. And then when somebody asks a question, you

do a similarity search on your vector database, pull the information, and provide that information to your LLM and tell the LLM, "Here's information you can use to answer that question." And what sounds really nice and good is really dependent on the type of information you provided in your vector database.

So if you put old information that's no longer relevant, well, guess what? What the LLM is going to reply to you is with information that's no longer relevant. So data is extremely important in this case.

Another thing you can do is you can do fine-tuning. And fine-tuning is basically you do additional training to an LLM. This is more expensive than the other techniques that we're talking about, and it takes a longer period of time.

And it doesn't really guarantee that you're going to get good results. In fact, depending on the type of information you're putting into your fine-tuning, you could actually degrade the LLM quality responses. Because it may actually forget some of the information it already had, or it may get confused between the new information you're passing and the information that it already had.

So at the end of the day, there's no technique that will guarantee that the answer of your chatbot is going to be good. But

using many of these techniques will help you get there or get you a long way.

Another thing that you should do is you should evaluate your models and your application at each step of the development life cycle. For example, when you are starting to explore which LLMs to use in your application or your idea of an application, you could use something like Ollama.

And we talked a little bit about Ollama before. It will let you run hundreds of LLMs locally on your machine.

Once you are interested in one or another, you can start using it in your application. Or if you want to play with thousands of open-source models, you can go to Hugging Face. And there you can build some Hugging Face spaces.

You can see the spaces that other people have built. And this is basically a lot of code that interacts with LLMs and that you can freely use and review to see whether you're interested to build something similar or use the LLMs that they're using.

Now, if you want to interact with other models, including some of the closed models, you can go to OpenRouter. OpenRouter will basically take your prompt and it will run it against a number of models or a model of your choosing.

And one of the things that makes OpenRouter really cool is that it will

take your prompt and let's say you want to run that prompt against LLaMA 3, it will look at the providers that are actually offering LLaMA 3 as an API and it will run it against the one that's cheaper, which is really cool.

And it's a great way to save money.

I use it every day and I put like $50 about six months ago and I still have about 40 bucks left. And I use it, as I say, I use it every day. So it's a great way to interact with models and see which one I want to use.

Obviously, if you're going to do OpenRouter in production, there are a number of safeguards that you may want to look into. You don't want just OpenRouter just running into a random LLM provider without you knowing which one it is.

But for testing, it's a great thing. And finally, the last tool I have here listed is LangSmith, which is created by the creators of LangChain, which is a very popular open-source

framework to interact with LLMs. But LangSmith, one of the great things about it is that it lets you constantly evaluate your models. So

it can help you record a number of runs that you have with your LLMs. You can compare over time how your LLMs have been performing. It's a really, really cool platform just to keep track of what your application and your models are doing.

So yeah, evaluation, you really should be doing this at each step of the way.

Another thing that is very important is externalizing your prompts. And I know that for all of us, it's really easy to just take a prompt, put it in the code, hardcode it in there, and move on. But if you do that, you're shortchanging yourself.

Externalizing your prompt in tools like repositories like LangChain Hub, which I have here a screenshot on the left, will allow you to

share those prompts with experts. So if you are building an application that has to do with education, you can reach out to an education expert that has never coded in their lives and you can have them log in to the hub, play with the prompts, run the LLMs, see the output that they're getting from the LLMs.

And then when they're ready and they're happy with that they're getting the correct prompts, all they have to do is just commit those prompts and that will flow directly into your application.

It's really cool. Another thing that this will help you with is future proofing. We all know that models are coming out every other week. There's a new model and that's better than the previous one.

And unfortunately, even let's say if you're using a LLaMA model and the next LLaMA model comes out, your prompt may not work properly with the new version. You may not get the same results. So by allowing you to easily access the prompts and modify them and test them against something new, then you are basically future proofing and making your life easier in the future.

And that in turn leads to faster development.

So bottom line, prompts should never be encoded in your code list.

I'll talk a little bit about agents. Agents are, and I expect them to be in this year, one of the best additions to the world of LLMs. Because agents are basically allowing the LLMs to break out of the chatbot and actually interact with the real world.

So you can have an agent, for example, that allows you to, for example, you have a book, you want to

create the audio for it and you want to translate it into multiple languages. That's something that you can do and you can have an output out of those things. So agents will allow us to interact with the real world.

All these LLMs that have been confined to just chatting, all of a sudden will have a real impact in the world.

And we're still not there. We're still working on it, but it's a really exciting

set of applications that will be coming out soon.

And here I have a couple of drawings of what an agent looks like. And the one on the left

that has four nodes, so it has start nodes, assistant tools, and it's really, really powerful.

Because basically what you're doing is when somebody gives you a question, you pass it to the assistant and the assistant basically has an LLM embedded and that LLM will analyze the question and then will start calling tools. And tools is a fancy name for functions.

It will start calling functions until it gets to the answer that it needs to answer the question of the end user. So for example, it could be that the end user sends the text of a book to the assistant and he tells it to translate it and create the audio.

And then the assistant will call the tool to create the audio, then it will call the tool to the translation and then it will end.

And the assistant is making those decisions all by itself, which is a really, really powerful concept.

And while it's really cool that those agents can make those decisions without you actually having to hardcode all those paths,

they do have its disadvantages,right? And one of the big ones is that LLMs have a lot of latency. Calling one of those LLMs takes several seconds.

So all of a sudden, if you have an agent that is integrating with four or five LLMs and each one of them takes, let's say, three seconds to run, then all of a sudden you have

a process that takes 12, 15 seconds. And we know that our users nowadays are used to running things on the web and those things taking only milliseconds. And if they have to wait for a second or two, people are leaving our sites and going to our competition, which is not the best result.

So we should take advantage of these platforms that have been built like LangChain, LLaMA, Index, Langflow. There's many of them,right? That allow us to do things like concurrent calls and branching

in these agents. So for example, I have here on the left a diagram of an agent that's actually calling a couple of things in parallel. So it could be, once again, this could be

the user passing a book to Node.A calling the audio creation and the translation in parallel and then sending the answer back to the user. Now, if each one of these things took two seconds, instead of these things taking eight seconds, all of a sudden it takes six.

So that's much better.

So there's a lot of things and a lot of things that need to come to mind when you're building these agents so that you can make them as efficient as possible so that we can still get to the end to fulfill the expectations of our users

while taking advantage of the LLMs' powers.

And while I do believe that agents are the future for LLMs, we really have to be aware of the costs of running these agents.

Because the costs can add up really quickly. Let's take, for example, the one I have here already. Once again, it's a simple agent.

And we're going to assume that this is a call center. This call center takes 3,000 calls a day and each one of those calls requires 15 function calls or tool calls. And we're going to assume that we're using OpenAI O1.

It's a great model,right? So we're going to take advantage of it because it's one of the biggest and greatest. And the prices for OpenAI O1 is $15 per million tokens or $60 per million tokens out. And if we assume that we're inputting 1,500 input tokens and 3,000 output tokens, meaning we're sending a bunch of information into the LLM, the LLM is

outputting a bunch of information out. Let's figure out how much that will cost us,right? Because it sounds like very little, $15 per one million token,right? It sounds like it shouldn't be much. But once you do the math, and the math is at the bottom if you want to take a look at it, you won't discuss it here, but it comes out to $3.24 per call, which turns into $9,270 per day and which in turn is almost $300,000 per month.

So if you were expecting to get a very small price tag, well, you're going to have a little bit of a surprise.

At the end of the day,right, especially if you're using

provider endpoints for your APIs to interact with the LLMs, you have to keep in mind that the cost of running this thing is linear,right? So each additional user will incur you an additional cost. Now, obviously, you have to make sure that you're pricing your products accordingly.

But if you were expecting to get very low prices and very low costs, you may want to do the math.

But even if you are renting the GPUs,right, so you don't have to make the API calls, GPUs are at a premium nowadays. So

if we compare the price of a GPU versus the price of a CPU on the cloud, we're talking cents versus dollars,right? So having an A100 GPU for an hour will cost you potentially $4 or $5 an hour versus a traditional EC2 instance that could cost you a fraction of a cent or a couple of cents an hour.

So keep that in mind.

And the choice of the model and the provider that you're going to use in your application is really, really important. And we can see in the price chart on the left that there's a significant value difference between the

left, so the first LLM in the list and the last one,right? It's over 20 times difference in terms of price,right?

So you have to validate, do you really need the power of a very pricey LLM or can you do the same thing with a cheaper LLM or two that may not be quite as powerful, but they can do the task,right?

It might be from a price perspective, it might be much, much, much more efficient,right? The other thing that's important, like we said, is auto speed,right? How fast these models can respond to questions.

And that's what we see in the chart on theright, the output speed. And we can see that Cerebras and Grok can run LLaMA 3.1 70B at over 1,000 tokens per second. In fact, Cerebras is over 2,000 tokens per second, while all the way on theright,

the providers are running 31 and 29 tokens per second,right? All that translates to latency for your users. And usually the size of the model that you're picking also has a big impact,right? So bigger models will take longer to run, smaller models will go faster.

Which one you're using and which one you need to use will have a big impact on your application. So at the end of the day,right, small is beautiful and better for the environment and better for your wallet.

So let's go back to our example,right? Now, what happens if we replace OpenAI O1 with LLaMA 3.3 70B just from a price perspective,right? You may need to use OpenAI for your use case, but what if you didn't,right? You can just use LLaMA 3.3.

Well, we go from over $3 per call to $0.52 per call. And we go from almost $300,000 per month to almost $50,000 per month.

So once again, pick theright model for the use case. At the end of the day, your wallet and the environment will thank you.

And the last thing I want to touch on is observability for your agents. I can't tell you how hard it is to try to figure out what an agent is doing once you get multiple nodes that are interacting with each other.

This is a lot harder, in my experience, than trying to figure out what a regular application is doing because a regular application is deterministic. It will do A, B, C, D, while

a probabilistic model

adds lots of unknowns to your application. So all of a sudden, things that were running 100 times with no issue will start failing out on 101st item,right? And I'll give you an example,right? I was running an LLM and all it had to do was get information about a user from the database,right?

So let's say, for example, get information for user John,right? And this would run for the longest time. And all of a sudden it started failing. And we realized that eventually, because we had the traces that the user was entering lowercase John instead of just capital J and the rest of the word,right?

And all it took was going back to the prompt, which we had externalized, luckily, and changing the prompt to add the sentence, "Ignore case," and everything started working again. But because this was an agentic application and it was calling multiple models, it would have taken us a very long time to figure out what was going on.

I mean, we could

have put the debugger and print statements and all that, and it would still have been very, very hard. So how observability for agents, there's a lot of tools out there like Langsmith that will make this very simple.

And if you don't want to use one of the tools that's available there, you can build one or you can build all this, but build it because you are going to need it. And in here, we have our agent traces on the left

of the screen. We have an agent that's first calling LLaMA guard, then it's calling LLaMA 3.1, then it's calling Mistral. And that

gives us an order of everything that had run in that particular call to the agent. It has metadata, it has the inputs and the outputs. All that is crucial,right? And when it throws an error, it keeps track of the errors,right?

All the information that was in and out of that LLM so you can see exactly what happened. And you can see the error, the trace like we see on the trace on the picture on theright. Yeah. So like I said, building observability for your agents is really important.

And with that, any questions?