The LLM Triangle: Engineering Principles for Robust AI Applications - Almog Baku:

Hey, thank you for having me, and thank you for being here today. Here we're going to talk about the LLM Triangle Principles, engineering principles for building robust AI applications. My name is Almog, and I'm going to talk with you about—surprise, surprise—LLMs.

So, this is a prerecorded session, but I think it's a safe bet to say that everyone told you today, or is going to tell you today, that LLMs are amazing. They're going to change the world. They—they're basically going to shift the way how we think about technology and software.

And, to be honest, most of the use cases today are just like sparklers, like small projects that a little bit enhance the workflow. And regarding to what they said, I totally agree with them. LLMs are amazing. They're blowing my mind.

And today, this is just the beginning. This is just the tip of the iceberg. Most of the use cases today might be LLM wrappers, or GPT wrappers, but the future is going to blow your mind. But in order to get there, we need to shift our thinking.

Uh, I think that a good analogy for that is thinking of LLMs like a new resource, like GPU or CPU, and Google just showed us today that quantum computer is around the corner. Actually, it wasn't today, it was a few weeks ago, but never mind.

But everyone know that when quantum computer will be here, nobody will write Python to program quantum computing applications. So why do we do that with LLMs? I think this is very confusing because we do build LLM applications with Python or, I don't know, רחמנא ליצלן, JavaScript or PHP.

Just kidding. Amazing languages. Not really. Yes, they do.

We do write them with Python and JavaScript and Go and whatever, but we also write them in native language. And I guess that everyone here that tried to build something, or built something, know that building POCs is very easy.

But production applications? That's performing in a certain quality? That's damn hard. Because every time we take the POC and show that in our demo to a real customer, every time we're just facing the only use case that makes everything break and the customer is just, "Don't get it," like, "Why the hell are you wasting my time?"

So, over the past few months, um, past few years, I've been thinking about it, and I came to the realization that LLM-native applications are 10% sophisticated models and 90% of hard experimentation and data-driven engineering work. And trust me about that, I'm an engineer, I just made this up.

So, uh, let's wrap it up. So my name is Almog. I'm a serial entrepreneur, started a few companies. My previous company was in the AI infrastructure world. I also the founder of GenAI Israel community. We are the largest community in Israel with 5,000 engineers and researchers and data scientists working on GenAI applications.

And over the past 2 years, I've been helping to build dozens of applications for my customers.

And over the next few minutes, I'm going to help you, uh, to maybe, uh, see the way I see about applications and learn from my experience. So I'm trying to distill my experience into, I don't know, the next 15, 20-ish minutes, and to help you to reproduce what I did.

Alright. So the LLM Triangle Principles build upon a few principles. We have the model, obviously, the foundation model, the engineering techniques, and the data. And all of that are guided by, of course, the fourth principle of the triangle, which is the standard operation procedure.

If you haven't heard about standard operating procedure, that's totally fine. That probably meant that you haven't worked in a huge corporation or a factory. So, to be fair, I borrowed this terminology from the manufacturing industry, and it basically means that standard operating procedure is a set of step-by-step instructions compiled by large organizations to help you, uh, to help the worker to carry out a routine work while maintaining a certain quality.

So, let's think about a huge fulfillment center,right? They can't allow themselves to hire some PhD in physics in order to wrap some package so it won't, I don't know, get lost or get unwrapped,right? Because if a package gets unwrapped, that costs them a lot of money, like customer support, the price of goods, their name, etc.

So instead of that, they just hire a lot of interns and tell them exactly what you do. What to do. First, you need to check all the items out there, and you need to mark them in our system, and then you put the label, and then you make sure that the package is allright, or whatever.

That's exactly how we should look of the LLM,right? We should look of the LLM like an intern. He's really smart. He's really, he's really nice guy,right? He says, "Yeah, you'reright about whatever we say." And on the other hand, he's inexperienced on our task, at our task.

So we should prompt it and tell it exactly what to do. If you will, like a step-by-step recipe. This way, we can ensure it's excelling our task. So how do we do that? In order to think about that, let's take an example.

So our example going to be how to build a landing page builder. So first, we want to assume that we are the expert, or maybe to interview a real expert, and to reduce the cognitive load of the task by simplifying it into some smaller steps.

So we'll start to interview the expert, or assume to be the expert. How would you approach this task? How would you create a new landing page? What's the key elements you consider? And then we will reflect on it and we'll phrase it for him.

Usually, the expert will say, "Oh no, no, that's not what I'm doing. I'm doing this and this and this." And this will allow us to have more information about the nuances that the expert know, but not always tell us,right?

So eventually, we'll have like a list of steps, and our goal is to find the hidden implicit cognition jumps that the expert know, but we don't. Because it's probably obvious that after you gather data, you write the page, but in between, the expert writing a compelling story that the landing page should deliver to us.

And that might be obvious to him, but not for us. And this expert, she might, she might know all of the nuances because obviously everybody know them, but we don't. And we need to identify them. And sometimes we'll do that and they won't even notice that they do that.

So we need to be very specific and ask very good questions, or at least assume to be them. Alright, after that, we'll compile a list, and I found that sometimes it's very helpful to describe this process as a graph.

Because some of the processes are like, like loop, or like we have a recursion there. So it's very helpful to draw it. Alright. So now we know what to build. Now that's, that's the time to actually build it.

So the engineering techniques, that's exactly what they do. It's a set of tools we can use in order to implement our SOP. So there are basically two types of engineering techniques. We have engineering techniques solely in the prompt layer, like chain of thought, role assumption, few shots, etc.

And we have some techniques that are solely in the software,right? Like chains, tools, fine-tuning, and obviously we have everything that in between. Like react, structured output, and we need to try and think about what do we want to achieve and how do we achieve it.

I want to focus today on two critical techniques. The first one is the LLM-native architecture. Alright. So simply put, it's just an architecture. Alright. Some people call it flow engineering, or chains, or workflow. It's all the same. It's simply an architecture.

What we do here is we take the SOP and we split, based on the SOP, the process into multiple steps. Multiple agents, if you will. Every agent will do something different, and sometimes we'll have some piece of code that will do that.

So for example, let's say we are building a Wikipedia writer. So the SOP said that we start to understand what the real subject we want to write about, and then we do some literature, we researching it, and then we write a table of content, and then we expand on each section,right?

So in order to build that, we'll split it into some process. So first, we normalize and distill the user request. We categorize the subject. We do some Google search and find some data. And if you haven't noticed, LLMs are really good with Markdown, and they are so, so with HTML.

So before we're going to deliver this information to the LLM, we're going to, uh, we're going to process it with some piece of code that's formatted to a Markdown, and then to another agent that write a table of contents, and etc.

So you can notice that some parts are done by the LLM.

We process it using regular code. So LLM is an amazing piece of technology. Seriously, it's mind-blowing. That being said, it doesn't mean that we need to do everything with LLMs,right? Always. LLMs, like, like a deterministic code, will always be deterministic.

It will always provide us with consistent quality. So you might notice that I said agents, agents, agents. What are agents? So based on the LLM Triangle Principles, agent is simply an LLM. A LLM agent is a standalone component of the LLM architecture that involve calling LLM.

To simply put it, it's LLM model plus prompt equal agent. And by prompt, I mean prompt template. And not all LLM agents are equal. Some of them are inferral, some of them are reusable, some of them are recursive.

And basically, an agent is just take me from A to B by doing something. And we have a special kind of agent, which is the autonomous agent. Autonomous agent are given with the liberty, with theright, to decide how to implement this task.

So it's basically agent with some tools. So the agent can decide how to plan and how to solve the problem. So take me from A to B by doing something, I don't care how, just do it. And we can implement the same problem with both ways,right?

So for example, let's say we just want to build some assistant. Alright. So we can say, "Hey, you are a smart assistant with access to the weather and RAG tool." The knowledge tool,right? And the agent will just do whatever it wants in order to complete the task.

Or we can achieve exactly the same with an architecture,right? First, we'll process the message. What does the user want from us? Emotional support, to know the weather, to plan some trip, to get some support. And we process that with some agent, and then we get like, like simply to, we feed the agent with enum and tell the agent, "Hey, just select one of them."

And then based on each selection, we'll do different processes. One of them will fetch information from RAG, the other one maybe will fetch some weather information, and then the result of that will deliver to a second agent which composing the response, and only that the user going to see.

And that's pretty complex. Why would we want that? Like the first option is much simpler, it's much cleaner. Like life is full of trade-offs,right? There's no such magic. So autonomous agents are

elegant software. Seriously, amazing. Very creative, very fast to build. Like you simply write some functions and provide the LLM with the way to call them. That being said, they're very hard to debug, and their quality might be unsustainable.

Like one time you might have a very creative, amazing solution, and the other time, exactly at the point when you're demoing to the customer, it's going to fail. In production, we need to have some sort of certainty. And with a crafted flow, we have a sustainable quality.

The process is predictable. We know what's going after each step. Higher possibility to debug it. And from the other side, it's very expensive because we need to invest a lot of time and a lot of resources to handcraft and develop it.

My two cents is that reality, like always, depends. The way I see it is that I highly recommend to scope the autonomous agents. So like, they can be amazing as long as they have clear borders. Like this is the way, this is the area that you can be creative.

So worst case, it won't deliver you good results, but it's very hedged.

Alright. So we've discussed the engineering techniques. Now, let's talk about the model. Not all LLMs create it equal. We need to match the task to the model. There are various factors to select the model. The complexity of the task.

For example, simpler tasks like summarization or entity extractions are easier for LLMs to fulfill rather than having a complex reasoning about some, some thought they need to compile.

Infrastructure and performance. A lot of models are out there, open sourced, and require us to build some sort of infrastructure in order to run them versus some proprietary software which is more expensive and we can use like the large models from Anthropic or OpenAI.

Also, we have the consideration of latency,right? The cost effectiveness. What's the price of the model versus how much it worth for us?

What's our budget constraints? The data available for us. If we have a lot of label data, maybe we can do some fine-tuning, or maybe we can feed it with some few shots. But if we rely heavily on the model knowledge, then we need probably a different model.

I found that this chart very helps me. So, alright, let's go through it very quick. Large models are expensive and they probably good at most use cases. Alright, we can throw most problems on O1 or O3 or O9 and it will be amazing.

It doesn't mean that it will be fast and cheap. Smaller models are pretty fine with simpler tasks or, and when we have a lot of data to feed them.

Smaller models with a very complex task probably will underperform, and large models with complex tasks and limited data set will probably perform fine.

So what should we do? My two cents is, you're going to build an LLM-native application. It's not a short rodeo. It's going to take you time. Start with the big model, collect data, understand what's going on, and then optimize it incrementally.

Incrementally, sorry. That means that you start with a big model, you understand what's going on, maybe you leverage some few shots you're creating, maybe some dynamic few shots, we'll talk about it later. And maybe after a few months you'll fine-tune or change the model.

We don't need to, to, to jump into the, the heavy lifting of fine-tuning from day zero,right? Alright. We can't talk about LLMs without talking about data. It's all about the data. Seriously. You can forget everything and just focus on the data because this is the most, most important part of my talk.

So it's important to realize that the model doesn't really understand what context or data is, or user prompt, or even system prompt. It only gets a, it only gets a single text, a prompt. So when we use

OpenAI or Claude APIs, we provide it with a system prompt and a user prompt and an assistant prompt. Behind the scene, there is a piece of software that takes all of these pieces, glue them together with some template, and provide it to the model.

The model sees some input text and returns you with some output text. That's the way it works. And

we, as builders, we need to make sense out of it. So we split it. And a very important piece of concept is the prompt template. Basically, we have some variables. We can place some variables in our template and to feed it to the model.

So we have like a prompt template, but each time it's slightly different. Again, the model doesn't have any memory. It doesn't understand what the context and the variables are. It's stateless. It's just a trick for us. But it helps us to maintain some piece of sustainable code,right?

And the key is to make the data relevant,right? Maybe you've heard about the needle in the haystack problem. So like the needle in the haystack problem says that if we'll feed a model with tons of data and we'll place some needle inside of the haystack, maybe the model won't find it.

And recent publications by the LLM vendors said, "Oh, now we solved the problem. Now we can find this piece of hidden needle in our text." To be honest, I don't really believe them because in reality we don't really have a needle in the haystack.

We have a lot of needles in a huge haystack. So if we'll feed the model with a lot of data and we'll ask the model to understand something, that would be very hard. From the other side, if we won't provide the model with enough information, obviously it won't be able to provide us with theright answer.

So we need to find theright balance of giving the model the context, but not too much a broader context, and how to focus the data we provide it.

There are a few practices and techniques how to teach the model what to do. For example, few shots. So we can ask the model, "Hey, please classify the product by its names." And we say PlayStation, and the model will return us gaming console.

That'sright. But we meant something slightly different. By providing the model with some examples, we can basically utilize one of the greatest

qualities of these foundation models. Their way to be in-context learners, to learn something from the context we provide without fine-tuning it. So classify the product by its name, the same system prompt. But when I say TV, you say consumer good.

TV, consumer good. And we just play with it,right? When I say printer, you say business equipment. Printer, business equipment. When I say PlayStation, you say, and then the model suddenly used a very different language. It said consumer electronic.

Alright, so both of the answers are correct, but the second answer is more in the scent. It's more in the vibe we are looking for, which is consumer electronic,right? So that's a very good way to teach the model some concept that is very, very hard to, to describe in words,right?

So we teach it by example. I think that

a key phrase to remember is show, don't tell. Sometimes it's easier to teach by showing and not by explaining. That's one of the key moments to use few shots. Some companies took this concept even further. They said, "Alright, we have tons of examples, and based off the use case, we're going to find you the best example for, for the model."

And based off the, based on the use case, we're only selecting a handful set of examples. Not all the examples, only five or seven. We can use some similarity mechanisms like vectoric search or something like that, much like RAG, or we can do some logic selection of the, of the, of the, of the example.

Like by keyword or by category or whatever.

So this is the LLM Triangle. It's composed by three

key principles: the model, the engineering techniques, and the contextual data, all guided by the SOP. Again, data is the king. Without data, we don't have a kingdom. Look at your data. Try to think how you specialize it using the engineering techniques, the SOP, to feed it to the model and to yield higher quality.

Thank you. If you have any questions, please reach out to me over LinkedIn. You can read the full white paper using this link. And thank you. Have a good one.