AI Engineering 201: The Rest of the Owl

So the, like, a lot of effort has gone into thinking about the engineering of inference, and not so much effort ha— and not so much success has been had at the engine- at engineering the rest of the, like, whole product around inference that actually, you know, delivers value.

Much like the two, uh, beautiful mathematical solids here that does provide the, the bones or the interior, but not the whole thing. So let's talk about a couple of, like, architectures and patterns for uses of language models, and then talk about the, like, first attempts at, like, trying to make these things better over time with monitoring, observability, and evaluation.

So architectures and patterns. So the, the foment and excitement around this stuff has been around for about a year, and so patterns are starting to emerge very slowly of, like, typical ways you might apply these things. So let's talk about them and what, uh, what problems have arisen.

So my favorite way of thinking about this in general is that the thing that we're buildingright now are language user interfaces, sort of the n like, LUIs by analogy to GUIs or graphical user interfaces. First, they're hitting existing features.

Soon, uh, they'll be for, like, completely new whole products.

In ancient times, in the 1970s, the interface for computers was primarily, like, textual in a terminal. This is still the way we interact with machines when we really want to control them, like when we're running a server or when we are frustrated with VS Code.

And this u this was the user interface for computers for a while, and they were not very popular until the invention of the graphical user interface, which instead of preventi- presenting the users with just, like, "You have to learn this special language to speak to me," like, here's this, like, sensory experience where you can bring your, like, your intuition from space and your visual system to understand how to use the machine.

And this was what took computers sort of, like, out of the hobbyist and business and military realm and into, like, people's homes. And the with the rise of language models, it's clear that we're we have an opportunity to once again change the interface between humans and machines by telling them what we want in natural language, and then they do it for us.

And no less Augusta Personage than Sam Altman likes the idea of language interface, language user interface. So this similar character to graphical user interfaces, it, like, makes it a more approachable interface. And this is something that people have wanted to do for a long time, as long back as, like, the Eliza chatbots or the Eliza chatbot from the, uh, 1960s, the Sherlu, uh, basically, like, this is only graphical, not an actual robot, but you could, like, tell a f a, a computer robot, like, give it language instructions, like, "Pick up a big red block."

Ask Jeeves was originally presented as a language interface to the internet where you just type what you want instead of a URL. Alexa and other a-assistants have attempted to do a similar thing. And the big win here is with language models, we might believe that we can actually do a really, really good job at providing this kind of language interface in a very generic way with foundation models, not just, like, a tiny environment like the Eliza psychotherapy environment or the Sherlu blocks world.

Soright now, that's we're getting language user interfaces for existing systems that kind of admit them easily. So Sequoia put out a piece fairly recently talking about this, that the, like, f this, like, act two of generative AI is using foundation models as a piece of a more comprehensive solution rather than an entire solution that offers, like, a language interface where it wasn't possible before.

So, like, this query assistant from Honeycomb takes what would normally be this, like, less approachable query language constructor and just says, like, "Can you show me slow requests? What are my errors? Latency distribution by status code?" Like, that's a much friendlier interface.

And, you know, even SQL when it was originally presented was like, it's a it's a language that's so natural even a businessman can write queries, you know? I-it's a dream, but, like, you know, th-this "Can you show me slow requests?"

Like, that's pretty close, you know? So that's the f the, like, maybe understandable that that's the first direction things have gone. Longer term, this, like, a machines that have graphical interfaces look very different from ones that have terminal interfaces.

And so, like, mainframes became less popular, and, like, mobile is, like, quite different from, like, desktop compute. So we should expect like, if you're thinking about what do I want to build in 5 years or 10 years, this is kind of the direction to be thinking.

So, for example, Google has worked on integrating language models with robots, like this example from the Say Can project or paper where it's like, "What I want to when I need something is to just ask for it and not to, like, pull out an app and then go through three dropdown menus and be like, 'I want a water bottle.'

I just want to say, 'I want a water bottle,' and then there's a water bottle." And that's what a language interface to something like a robotics platform can provide.

Still not there yet as the 4x speed in the top left might suggest, but getting there. Okay. So that's, like, the highest level pattern, I think. So let's talk about a couple of lower level patterns. RAG chatbots, retrieval augmented generation chatbots.

I've emerged kind of like the to-do list app, the sort of, like, starter project of language user interfaces. This pattern is probably here to stay in that it's just about information retrieval for language models. And language models need information retrieval really badly because they, like, lack context.

They've slurped up everything on the internet, but they don't know anything about you. They are sort of trying to simulate a generically helpful individual who is, like, generically knowledgeable about the world, and that's, like, not particularly helpful until they have context.

So the solution that's emerged is to collect that context for them, like, store it, then index it, and by default, people reached for the most similar thing to what the language model was doing, which is, like, turn it into vectors and use that.

Use, like, a fast index over vectors. And, like, that once you've retrieved a particular piece of information, you just stuff it into the prompt. So I am not innocent. I have made my own RAG chatbot and inflicted it on the world.

This was based on the Full Stack Deep Learning content, and in our Discord, people can ask questions and get answers that are not just, like, generic Google results search answers about language models, but things drawn from past lectures, things drawn from papers that I like, things drawn from our, like, website, and so can get our, you know, our opinions on these things.

So this, like, this has led to a lot of excitement about vector storage because it's, like, this v this step here where you have a fast retrieval of vectors by similarity is the, like, new sexy piece. But that was, like, really only the thing that people reached for because OpenAI also offers embeddings.

So it's like, you've already imported the library, so it's only a call away. And then also, like, transformers are kind of like these, like, weird vector retrieval things, like, in their inside. So if you are the type of person who's been into language models for a while and you're like, "How would I retrieve information?"

Probably with a dot product and then, like, a softmax, and then I pick the largest number. So, like, yeah. So, like, the ease of setting this up and the, like, naturalness of setting this up has led to, like, an explosion of these, like, chat with document examples.

And the, like, the thing that has more staying power is that you need to make these things useful, you need context, and so you need, like, information retrieval and search for the, like, the context that might be helpful for the model before it gets going.

And so there are many options to use here. Some of them are specialized vector databases like Pinecone or Chroma. Some of them are general, like, text search databases, like, that do keyword search, like Elasticsearch style

things. And

being able to, like, combine those two things together is very powerful. So, for example, Vespa has, like, offered that combination for a very long time. It is also, in the end, like, what you're doing is creating a fast way to look up information from a very large store.

So this is, like, bread and butter for databases in general. And so Redis and Postgres, for example, like, not only do they provide the sa like, information retrieval that you could do, like, to enrich your enrich prompts without thinking about vectors, they also have built-in vector search.

Postgres only fairly recently, Redis for, like, a year. It's not particularly fun to use Redis vector search, but it does it c it can run and has decent performance. Yeah. And in the end, it's about, like, an a holistic strategy that uses probably because the queries are fairly heterogeneous, the things that are coming in are, like, people just typing text, you're probably going to need some more MLA stuff that's more like keyword search or, or vector search, and they're hybrid together.

But that, like, meta like, extracting metadata with a language model so that you can then use that to do, like, direct filtering is, is, like, very powerful pattern. So there's some great posts on this, the data quarry, great series of posts about vector databases from coming from, like, somebody who's clearly really into databases and not so much the, like, ML side.

And I found that very useful. Yeah.

Yeah. So the, like, the final takeaway there is just that the problems end up being in the main the problems of information retrieval with only some light added things from, like, recommendation systems maybe of a more MLA type of search.

Yeah. Any questions on, on vector databases or information retrieval for language model applications?

Right. Obviously, you combine that so you get the context to answer a question and use that send that to the language model and then use that context to help answer the question. That's the one.

Yeah. Yeah. So you, you get information from the outside world. You, like, come up with a strategy for searching the information that you have saved that goes into the language model's prompt. Yeah. Yeah. That, that pattern, very, very stable, very general.

It's general enough that how that pattern gets actually implemented is very broad. And so it includes a lot of things that are exi like, bread and butter database stuff and not just the fancy new vector database stuff.

How does this so RAG's approach to injecting context, how does that

how is that similar or different to your history when you're interacting with, say, GPT-4 or whatever? How does it retain that compared to context injection?

Yeah. So the question was, how does retrieval augmented generation differ from history within a chat? So usually when really, so the, the when you call the GPT-4 API, you can make whatever make up whatever you want as the past.

You could insert little messages from the user. You could insert messages from the assistant and incept it into believing that it has said something which it has not said. Great way to jailbreak. Don't do it, obviously, because it violates the terms of service, but a great way to jailbreak it.

And so you, you aren't actually, like, actually beholden to that, like, system system assistant human fiction that, that happens inside of, like, a, a discrete chat. When people do this, I think a lot of people put the retrieved information in the system prompt, especially if they're just going to retrieve once.

I've definitely I've also seen people, like, every time the user interacts, they do a retrieval step, and so the system message changes every time. That's an example of kind of, like, incepting or not actually following the implied temporal order.

So you yeah. You definitely can do that. The system message is nice because the model really pays close attention to it, has been, like, fine-tuned to pay close attention to it. Yeah. I think it'd be weird to pretend that that's something the person said and to, like, put it in an earlier user message, put above the user's message in the conversation.

I don't think I've ever seen that, but you could. Yeah.

But yeah. I would say, like, most of the time, yeah, this information retrieval step is something where the creator of the application, the programmer is inserting themselves and saying, "I know some additional information that the language model should, should have."

And so, like,

yeah. It's very different from, like, a user just sort of, like, providing information about themselves or whatever.

Yeah. Yeah. The question.

So we heard a couple times today when it comes to knowledge retrieval, RAG is a great pattern and fine-tuning is really more about the format, the style of the output.

Is that unconditionally true or do you see use cases for fine-tuning enhancing the model's knowledge? So, for example, what happens if we fine-tune it with 100 gigabytes of research papers? You know, there's not many things to it that is measurable beyond the style of its output.

Yeah. So the statement was that the common wisdom is that fine-tuning is for style and retrieval is for information. And I think that that's that is a solid common piece of common wisdom because most of the fine-tuning that people if you're fine-tuning OpenAI's model, you're going through their fine-tuning API and you have a limit on the number of rows you can send.

10,000?

Yeah. 10,000. I was going to say yeah. So you have a limited number of, of rows you can send and, like, there's a limited amount of information in there to, like, create gradients to update the weights. So there's a limited amount of change that you can achieve.

And if you look at the LoRA paper, they look at, like, you know, the like, you're only ch you're, you're making a very low-rank change to each layer of the of the language model. And that suggests, like, there's only so much that you can change about the about the model.

And most of what you see when you do LoRA fine-tunes is, like, what used to be a low-priority computation for the model becomes, like, a higher-priority one. So, like, every model ha every capable language model has within it a little Homer Simpson simulator, a little, like, Rick Sanchez simulator, whatever.

And that's it's just, like, not usually that important for the final log probs. It's, like, helpful for the, like, fifth bit of the log probs, but the models are at the point where they're maxim minimizing cross-entropy by, like, really hitting those, like, very rare those very rare things.

And so what the fine-tune has done is reordered those, like, computations. So, like, actually, you should be the Homer Simpson circuit is the most critical circuitright now because you are a Homer Simpson chatbot, and it's, like, reordering them and, and reemphasizing them.

So that intuition applies specifically to low-rank fine-tuning, which is and fine-tuning, which is based on small amounts of data. So if you grabbed 100 gigabytes of textbooks, you would no longer be doing fine-tuning, and so you would no longer expect it to only change style.

And so that's something I would expect people will be doing with, like, you know, LLaMA fine-tunes. There are, like, LLaMA fine-tunes for coding, and that's way more than just style. It definitely has learned more knowledge about, about programming languages and knowledge about libraries released after 2021 and, yeah, all that kind of stuff.

So I think that that generic wisdom is conditionally true for low-rank fine-tunes where it is, like, pretty rock solid.

And that kind of hybrid search concept that you see in the RAG database for the RAG functionality, have you inserted graph, like, input data in there to capture the value from those databases?

Yeah. So the question was, what about knowledge graphs and graph databases? I will say that, like, when I have talked to I, I, like, personally don't really specialize in, in databases, but when I've talked to people who are super into them, they're like, "I would never use a graph database because you can represent a graph in Postgres."

And, like, I've seen some, like, reasonably sized deployments on that pattern. And also, you can kind of see the, like, graph databases kind of, like, peaking and, and not spreading further. And there are it's there is a very hard problem to shard a graph database because there's no obvious way to cut an arbitrary graph.

And if new links get added to the graph and now you need, like, the optimal shard is different, it's like, that's a ver that's like a database it's equivalent to a database migration, but it's something that should be happening, like, behind the scenes when it's sharding.

So that's, that's, like, the closest thing to an objective statement about why or, like, a reason why graph databases haven't worked well. However, for many language model applications, the purpose of the database is not to serve, like, a billion users, but rather to, like, serve as an external memory for a language model.

And maybe you don't care whether it scales or rather, like, maybe the maximum scale that we're talking about is, like, tens of thousands, hundreds of thousands of requests per second on megabytes, gigabytes of data. And that's just, like, you know, that's the point at which that kind of, like, can it be sharded across 1024 machines?

Like, doesn't matter. So I so there is some cool work on knowledge graphs and, and incorporating with LLMs, and I see the natural fit there in the same way that there's a natural fit with vector indices and vector databases.

But the

yeah. It hasn't no, no, like, killer app has appeared from my perspective.

So I asked this question on the, the first talk, but basically, like, if you have extra context and hard metadata about some of the boolean value or some other things, how would you kind of incorporate I think the previous answer that was that, like, do hard filters on the on the hard metadata and then do the soft kind of, like, stuff based on the, the, the, like, textual descriptions and stuff.

But I wanted to keep you in additional context.

Yeah. So the question was about how to incorporate hard metadata, like, you know, booleans or, like, subcategories with vector-based search. Yeah. So depending on the, like, vector database, the depending on the, like, index, you will either have, like, prefiltering or post-filtering.

Post-filtering is, like, pretty easy. You just, like, apply a metadata filter after you've done your vector search. Anybody can kind of do that. The problem is that you're now what you really want to say is, "I want to find all the stuff that's similar to crabs in San Francisco while searching restaurants," not find all the restaurants that have anything to do with crabs and then see if any are in San Francisco.

So the prefiltering step is hard because it impacts the construction of the index, impacts the construction of, like, the like, how you make it actually fast to search over all of the data. You kind of, like, need to construct specific indices for these different, like, flags you might, like, put on, like, is in San Francisco, not in San Francisco, or geographic location.

And so I depend like, different vector databases or, or different databases have, like, pushed further in different directions on what kinds of filters they support for prefiltering. And yeah. I like, Vespa and Weaviate have a reputation for doing a really good job at those things, but yeah.

I don't know what the full landscape looks like.

Great. Okay. I want to make sure to get through everything. So I'll stick around and we can talk throughout the conference. Okay. So structured outputs are, like, one of the patterns that I think people are sleeping on relative to information retrieval.

Structured outputs are great for improving the robustness of models, and they came from tool use. So the problem is that language models just generate text. And, like, if anything, we have, like, too much text already. Like, I don't know if you've ever been on a social media website, but the problem is not the quantity of text.

And that's, like, kind of boring. Like, who wants to just make strings? Like, there's other things that we want to do. The solution is to connect their text outputs to other systems' inputs. And now, like, it's not just a language model.

It's, like, a cognitive engine for providing a language interface to something else. That's pretty rad, but there's a problem, which is language models generate unstructured text because they have been trained on the utterances of humans on the internet, notorious for their unstructuredness.

So the solution is to add structure to their outputs. And there are many ways to do this. You can do it by prompting and begging. So, like, you can write some write some, like, loops around it to be like like, or actually, React wasn't even a whole there's some looping.

Yeah. So you, you can write a prompt in such a way that you have examples that encourage it to, like, to call out to external APIs, and then you filter, and when it generates the tokens, that would, would call to an external API instead of letting it hallucinate the rest of what would come out of that API, which is what, like, GPT-3 would have done.

You, like, grab it and you then go to that external API and you,

like, yeah, pull the information from there. The and you can in those prompts I guess really the thing I want to point out is that in those prompts, you can sort of, like, beg for structure. Riley Goodside had a great example where it was like, "If you do not output structured JSON, an orphan will die."

And that actually is extremely effective. Yeah. So the so there's so, like, there's prompting tricks to get, like, things that are closer to structured, structured outputs and to make use of those structured outputs. There's fine-tuning. So there's a the Gorilla LM is, like, fine-tuned on this problem.

And that goes back to Toolformer, which was, like, very GPT-J, so one of the first open

generative pre-trained transformers. They you just train the model to output structured stuff. So you can't do that with OpenAI's model. I doubt that fine-tuning it would make it that much better at, like, outputting the structure that you want.

You can do it with

with open models, and there are people releasing their own forks, LLaMA forks, with this fine-tuning on them. You can you can retry, which is, like, when the model outputs something that doesn't fit the schema, you can do what you do when your direct reports provide you something that does not fit what you wanted, which is that you can discipline them and ask them to try again.

So guardrails is a great

library for this. It's, like, XML-based, so probably would work pretty well with Claude given what we heard about, about Claude from

Karina. And then a fun one that that kind of requires control over the log probs is grammar-based sampling, which was merged into LLaMA CPP, where you say, like, "Hey, when you're about to generate a token, like, if it would violate some grammar, if it would violate some template or format, just set the probability of generating that to zero."

So just add, like, minus infinity to all the all the log probs.

And the so you can do that. You can, like, do it fast if you have these, like, nice, you know, Chomsky-ing Chomsky-ite things, like context-free grammars. And this works well for, like, you know, JSON for generating, you know, generating code, generating all the kinds of, like, structured outputs that our systems actually expect.

We have written systems that expect inputs to follow grammars so that traditional computing system can parse them. And so adding that to the outputs of these systems is a very powerful thing to do. So this is something that this is a, like, really nice example of how having tight control over the log probs can, like, increase the utility of a model to the point where, like, a capabilities gap is less important.

Have you seen TypeChat? Does that fit this pattern?

TypeChat? I don't think I have.

Yeah. So there's.

One question.

Yeah.

So yeah. So could you take the output from, like, GPT and then pass it to Gorilla to then get the structure, like, stacking models like that? Would that work?

Yeah. So the question was whether you could do better you could solve this problem by chaining models. I think, yeah, the problem of going from the output of a language model to a structured output is an easier problem than the initial one, which is why people think that, like, retrying might work.

Like, like, the guardrails the guardrails example, the, like, retrying is often, like, kicked off to a to a smaller language model. Like, your main line thing is GPT-4 and your error handling is GPT-3.5. And so, like, I I do believe that there's, like, kind of a temptation if you know that it's always and only going to be doing, like, structured output, then you have a reason to have a specialized model for it.

But yeah, chaining chaining is definitely a good solution. And that's, you know, one reason why LangChain was popular. Yeah.

You mentioned that OpenAI retired the log probs from the 3.5 instructions.

Yeah.

Do they still have log bias to, to bias the token by to do the grammar-based sampling?

Yeah. So those are technically distinct things. Yeah. So I do believe they still give you the ability to bias tokens via the API.

So would you.

Yeah. So it's not the it's not a perfect example of the utility of log probs because yeah, I think you can still do this in the OpenAI API.

Yeah. Do you need anything other than biasing and grammar-based sampling? No. I guess yeah. No. The real okay. I remember now. The real thing here is that for this grammar-based sampling, it's single-token-based,right? Like, if you're doing it from the OpenAI API, one, the token you you'd have to make a request, you get the thing back, you have a single and you have a single token, and you'd have to you have to, like, apply a bias every single time.

So now you're like, every token has a network call rather than one call, like, 100 tokens. So that's one reason why this doesn't work well on the OpenAI API. Number two, like, kind of longer term is that really you don't want to just think at a single token level.

You're just, like, at each token, you're, like, marginally just saying, like, adjust the probabilities here. You'd really want to do something more like Monte Carlo tree search where you're, like, generating stuff, many things that follow the grammar, and then accepting the best one at the end.

And that's something that's probably going to come first to open models and not to proprietary model services. So that's that's the better reason to connect grammar-based sampling and open models.

Okay. So the problem with fine-tuning and an annoying thing about prompting is that if there's not a kind of shared like, the Gorilla model is, like, fine-tuned on a bunch of APIs from, like, Torch Hub, TensorFlow Hub, and Hugging Face.

So the Gorilla model is really good at using other machine learning models, but not, like, generic possible tools, at least this example. They maybe they have tuned more than one. But this is a general problem that if you train a model to use a specific tool, then, like, the it's not going to be able to use, like, any tool.

But if you train a model to use a very broad class of tools by using something that's, like, kind of closer to this grammar where there's, like, a format for tools, then you are now people write an interface between the

that standard and the and the thing that they actually want to use. So this has shown up in OpenAI like, in OpenAI's API as the use of JSON schema for describing function calls. So this allows them to train a model on fairly generic stuff that all fits this like, it all fits the JSON schema spec.

And so the model has learned a bunch of stuff about the JSON schema spec and how to generate that correctly. You can imagine using grammar-based sampling to enforce that. And this allows it to connect to many, many tools because now all you need to do is write a tiny connector between, like, the JSON format and the actual thing you want to use.

And that's, like, pretty easy. It's, like, a big part of web development, from my understanding, is that you just, like, pass JSON blobs back and forth until somebody gives you money. And so yeah. So this is a very good kind of schema.

But one thing that people miss is that the tool doesn't have to actually be real. Like, the key thing that happens here is the language model goes from outputting unstructured text to outputting JSON that fits the schema. And it just so happens that the primary use case for that that OpenAI envisaged was putting it through a, like, function call, putting it through some downstream computer system.

But, like, really that some downstream system. But really, it, like, doesn't have to be a real function. You can tell it about a fake function that's, like, please pass a string, like, describing whether the input was spam or not spam so that I can, like, render an HTML element,right?

And so the model is now trying to, like, call a function that's, like, that in order to provide the arguments to that function, it has to decide whether an input is spam or not spam. And that's maybe the thing you really care about.

And so you, like, invent a little fictional function for it to call that you don't call, and then you just use it for something else. So this is a pattern in there's a library for this called Instructor from Jason Jason Liu, who's going to be speaking later at the conference.

Yeah.

And is it important to make it act, like, you're about to pass the function, or can you just say, "I want this to be spam"?

You have to fit the JSON schema, which the schema that they like, the there's, like, a meta-schema kind of thing. They're like, it has to it has to be a function call. And the model has been trained on things that are, like, you know, get name, get current weather.

So yeah. I mean, you can hack in because, you know, functional programming has taught us that everything is just a function. Like, a constant is just a function that always returns the same thing. And so you can you can, like, hack it in there.

And Instructor has some fun, like, kind of functional programming stuff built into it, like maybes and stuff. So, you know. And also, somebody did, like, DAG construction where it's, like, you give it a schema for a DAG constructor, and then it, like, writes a DAG of function calls instead of just a single function call.

So you can really go wild, which is very fun.

And yeah.

Most of the time when you generate something,

if you want to extract something out of the output, you want to display it to the end user, and then the question of latency comes in. That's why you stream it.

Mm-hmm.

If we use this, how do we solve the streaming problem?

Yeah. So that is a great question. The answer is that this basically breaks your ability to stream. I think it's not so this is maybe a little bit more oriented to, like, back-of-house stuff where you're using language models to, like, handle data rather than using language models to directly interact with a user.

I think if you set up a pipeline correctly, then you can stream the outputs from one call into the inputs of the next one. And if you have the relevant information you need from the function call one, then you can just immediately kick off the next thing.

And you can just you can write, you know, like, more like a Unix pipe style, and then you start to get back to being streaming. But you don't have like, the Unix pipes work because of new lines as a separator that lets you break work out, and there's not an obvious way to do that with this.

So yeah. The short answer, I guess, is that it's really hard to get back that kind of streaming thing when using these. Yeah. I'm going to let's see. How much more do I have? I'm going to push forward because I want to make sure to get to the last section, but I will be around to answer people's questions.

Okay. So this conference is not called NLP Engineer Summit, and we've been talking about, like, you know, structured out extracting structured outputs from language, information retrieval. Like, that's also natural language processing and language user interfaces. Like, that's not artificial intelligence.

Like, where's the AI? The like, the thing that really feels like artificial intelligence with language models is something like agents that are that have memory that they keep over time. So, for example, the generative agents that was let's see.

It's mostly Stanford, people, if I remember correctly. But the, like, generative agents paper combined, like, a stream of memories generated as these agents interacted in, like, a video game environment with, like, some, like, reasoning flows to create these, like, little tiny characters that had personalities that developed over time in interaction with each other.

And, like and that is, like, much closer to what people imagine when they hear AI than even a chatbot.

And there's been a lot of advancement in using these things in simulated environments. So that was, like, a full all-language models simulated environment with generative agents. There's also a ton of really cool stuff going on in the Minecraft world, which is, like, people have this Voyager agent writes JavaScript code JavaScript code to call the, like, this, like, Minecraft API that allows it to, like, drive a little, you know, a little character in the Minecraft world.

And it starts with basically nothing, and then it writes itself a bunch of little subroutines to, like, Minewood Log or, like, Stab Zombie or whatever. And it, like, accumulates them over time, like, learns how to do new stuff, like, comes up with its own curriculum for how to like, how to get better, and was able to, like, do extremely well at this notoriously hard RL task, Mine Diamond, which was, like, a grand challenge for the RL world only a couple of years ago.

So they're like, they can accumulate information over time, they can accumulate skills over time, they can use tools. This is all very cool. They are they have a couple of problems, the biggest one being the, like, problem of reliability.

Structured outputs can't help with that. And there's, like, only limited work, I would say, on agents that has come out since at least, like, published, you know, research work since the like, since function calling got really good in the OpenAI API.

Also, there's kind of, like, a cacophony of different techniques out there with, like, Voyager. React is kind of an agent generative agents. There's a really, like, awesome paper from Tom Griffith's group at Princeton, Cognitive Architectures for Language Agents, that brings back a bunch of ideas from good old-fashioned AI in the '80s on, like, production systems and cognitive architectures, a bunch of stuff that was, like, really cool ideas, but it could never, like, get past the demo stage on, like, how to create the things that we know about or that we believe about human and animal cognition, like procedural memories, semantic memories, episodic memories, how to implement that in software.

And the problem like, those systems could do cool stuff, but the problem was always that they lacked this, like, general world knowledge and common sense. With language models, they don't have memory, and they don't like, they don't have this, like, structured aspect to their cognition, but they do have that, like, world knowledge and that common sense.

So this is, like, mushing those two things together and using a language model to do basically these kinds of, like, observing the world or doing cognition and doing decision procedures, like, marries the best of both worlds. And it is actually, like, a pretty effective way of breaking down the existing agent architectures.

Like, and there are different choices about how to do long-term memory, how to do external grounding, how they, like, interact with the external world. The concept of internal actions comes from cognitive architectures, which is, like, choosing to spend time reasoning or choosing to update your, like, long-term memory or yeah, or your decision procedure.

Yeah. And then also explicitly calling out a decision-making procedure. So there's a and that paper is also just, like, has an entire research agenda in it on, like, ways that you could just start filling out the cross-product, just filling out a big array of, like, try this idea from language models with, like, this idea from cognitive architectures.

And there's just, like, a billion really cool ideas in there. So if you are interested in agents but have, like, struggled to, like, like, wrap your brain around all the different ways you could you could do stuff and around, like, how to make them a little bit more tame, I think the Koala paper has some good pointers.

Oh, yeah. And then lastly, for this LLM patterns thing, I was talking generally about, like, different ways people are building stuff with LLMs. Eugenia Anne's blog has some of the best writing on this, both on patterns and anti-patterns.

Okay. I want to give some time for monitoring, evaluation, and observability. So I'm just going to I know there's probably lots of interesting things that people have to say on the agents stuff, but we'll we have the rest of the conference to talk about that.

So the goal here is to talk about AI engineering. So that last part was about AI. What about the engineering? In engineering, we want to have a process for building, like, a process for creating these things and a process for improving them.

And progress on this front has been pretty halting. And so the, like, the dominant ideologyright now is that you should ship to learn rather than learning to ship. And so this is one of the big ideas in the Full Stack Deep Learning course that I've taught in.

It's something that Andrej Karpathy has really hammered on, the, like, idea of a data engine or data flywheel, where in order to do well, you need to go out there and collect data from the world, find issues in your data, and use that to improve your model in, like, you know, an unending cycle.

Charity Majors from Honeycomb, who's big in the monitoring and observability world, like, has said that this is something that she has come to like about ML. In software, you start with tests, and then you graduate to production when the tests pass, or at least, like, that's what you tell people on the internet and, like, your manager.

But with ML, you can't even lie, and you know that you have to, like, start with production and use that to find out the, like, issues with to, like, generate your tests. So, you know, it's oops, all regression tests version.

And so what that means is that monitoring is very critical from the very beginning. We monitor for user behavior, we monitor for performance and cost, and we monitor for bugs. So some of these are just, like, regular old monitoring stuff, and this is just, like, bread and butter things that can be, yeah, like, similar to the way we do with existing software.

Monitoring users always reveals, like, like, both misuse and product insights. So one thing that I found from running this Discord bot is, like, one of the things that you get the most are, like, meta questions. Like, are you getting feedback from these emojis?

Who's a good bot? That's maybe not a meta question. Does your dataset include your own source code? What do you do? Like, these are very common, like, things that people input, and it wasn't it wasn't in my head that that was important.

So now there's, like, special stuff in the prompt for handling that class of questions. So by monitoring how users use your system, you can get really great product insights. Yeah.

Did you get these from just, like, a few-minute manual check of your crew?

Oh, yeah. I had logged them to Gantry, and then I looked at the ones that had up and down thumbs. And I also read all of them because it was, like, only a couple hundred rows.

Oh, man. My battery is going to run out. Allright. We got to move fast. So monitoring monitoring performance can help us manage the constraints that I, like, talked about when we were thinking about all the different places our models might run.

So as always, you want to monitor things like latency quantiles, like, like, how long do requests take? Oh, wow. That's nice. Thank you.

Huge.

And so, like, latency quantiles, like, that's how long, like, take all the requests. What is the probability that a request took at least this long? People often think, like, if I get 90% of them, like, below something, that's great.

And the problem with thinking that way is that users don't just make one request. They make many requests in sequence. So by the time you've made, like, 30 requests, if there's a 10% chance of hitting, like, a really slow one, then, you know, you have hit a slow request.

So that so you really need to care about those, like, 99th percentile latencies. Those are also often your most useful and engaged users. So watch those watch those extreme quantiles. And obviously, like, throughput is a distinct thing to also monitor for the quality, you know, quality of the system.

You want to marry that with things like the profiles and traces that I talked about before, like spot check ones randomly subsampled so you can check what like, so you can actually debug that throughput issues. That's fairly general stuff.

If you're using inference as a service provider, you're going to want to monitor API rates and errors, monitor costs. If you're self-serving inference, you have a lot more stuff to monitor. And that's, like, compute utilization.

Yeah, I guess I already talked about this. Yeah, yeah. Well, so it's an even harder like, maintaining throughput when you're doing the inference yourself is, like, much more your problem and much more AI ML-specific stuff. Yeah. Okay. Monitoring for bugs is another can of worms.

We'll talk about that in a second. This is, like, just generally, this is a very fast-growing field. So there are generic monitoring observability solutions for all kinds of, like, you know, complex apps and web apps: Datadog, Sentry, New Relic, Honeycomb.

Like, these are like, you can adapt those, and that might be the thing that wins. There is you can, of course, just roll your own with the, like, you know, open telemetry compliant, you know, tooling, and you could use the existing MLOps tooling.

So there's a lot of stuff that has been built for monitoring observability of general ML applications, so including weights and biases. Where I used to work, Fiddler, Horizon, Gantry are the, like, three larger startups in that space with more of a focus on monitoring systems in production and less on the, like, MLOps, like, kind of, like, serving and, like, managing managing training, like, weights and biases.

There's also, because generation times are now six months or less, a new generation of ops tooling for LLM ops, including Langsmith from LangChain and LangFuse, which was in Y Combinator's recent batch. It's, like, very unclear which of these is going to be the best solution.

So I think it's, like, you know, dealer's choice, try them all out. I think I like tools with as much ability to, like, make crazy queries of unstructured data as possible. So that's something that I really like about weights and biases production monitoring offering.

Gantry has some similar stuff. I've tried less of it with the other tools. I think if you're doing if you're doing it with Datadog, Sentry, et cetera, you're probably going to need to roll some of that stuff yourself.

But maybe that's fine. Jupyter Notebooks are fun. I was going to check out the, like, LangFuse monitoring interface, but in interest of time, going to go past that. They have an awesome demo where you can interact with their docs chatbot, and it shows up in their monitoring interface.

So, like, they have a live demo of their monitoring tool where you can actually, like, use it to monitor an app that you can also use. So that's just it's really it was really fun to, like, actually try out the tool that way.

I recommend you try it out. But just monitoring, like, just getting a hold of information is not enough. This is something that's known from, like, the distributed systems monitoring world. What you really want is observability. What both Charity and Andrej were talking about is about how you improve a system based off of what you observe.

It's, like, not enough to just, like, throw something out there and observe and, like, just see the mistakes. You want to, like, fix the mistakes. And so there's this Honeycomb and Charity are big on the idea of observability as the as an idea from, like, control theory, from, like, old-school, like, control theory, systems theory.

Observability is whether you can actually figure out what is going on inside of a system just from observing it from the outside. So it's like, can you actually debug this software just from looking at your logs and not having to go into a live debugger inside of the system?

And that's, like, when live debugging does not work and when systems have outpaced our ability to predict what's going to break, this is the only solution. And for AI systems, that is where we can't predict what's going to break, and you can't, like, drop into a debugger 13 layers deep in GPT-3 and or GPT-4 and, like, debug its inference.

You have no choice but to monitor stuff sufficiently that you can fix the issues.

The blocker here is that actually determining whether the model isright or wrong is itself hard, which makes figuring out how to fix it also hard because you don't necessarily know whether it's messing up, and you don't know whether you fixed it.

So we're in a tough phase for this problemright now. There will be lots of discussion of evaluation at this conference, which is very exciting. Lots of people complaining about how difficult evaluation is, Anthropic, and Arvind Narayanan from Ansayesh Kapoor, who write the AI Snake Oil Substack, really high-quality stuff.

And OpenAI, like, open-source their eval framework because, in part, they, like, don't can't really evaluate their system themselves. It's like, that's how hard this problem is. It's also what we saw with the false promise of imitating proprietary LLMs.

Like, a large community of people were, like, kind of convinced that models were doing better than they actually were. So the solution, like, is to, like, one of the key solutions is to spend time looking at your data.

Stella Biederman from Allure has talked about this. Jason Wei has talked about how critical this is. Jason is at OpenAI now and talked about spending, like, a ton of time just, like, getting very good at evals, like, building tooling, internal tooling for evals, spending time with, like, understanding the evaluations.

And somebody on Hacker News said it's a major differentiator. So, you know, that's that's definitely the orange website never lies. So evaluation is particularly hard, and all these complaints about evaluation are when you're dealing with, like, open-ended generations from a language model, like, no structure to them, no real structure to the user inputs, and, like, limited data sources.

But there's this nice flow chart from the Full Stack LLM Bootcamp that my fellow instructor, Josh Tobin, made that sort of helps you avoid getting into that pit of evaluations. So if you can find a correct answer, then you can stick with existing ML metrics, and you, like, don't have to worry about the problems of evalu of, like, the difficulty of evaluating open-ended generations.

If you have a reference answer, you can check for, like, reference matching, which is, like, a looser thing than, like, a literal correct answer, which is, like, A, B, C, or D in multiple choice is a correct answer.

A reference answer is, like, a short, like, generation, like, a short answer on the test. If you have a previous answer from your system, you can at least see if your system is getting better by comparing the two and that, like, kind of which is better comparison can be done by a human, can be done by a language model.

And if you have human feedback, you can actually check, like, between the input and the output, was the feedback incorporated by the language model? Like, a human said, "I didn't like that." Did the language model get better? And it's only if you don't have any of those things that you, like, are out in the unstructured world.

The people at Elicit who have worked on doing extraction of information from scientific papers have a very principled approach of iterated decomposition where you start with a task that runs end to end, and then you when you notice a failure, you look at the failures and you see how you could have broken the task out into multiple pieces in such a way that the failure would arise in a simpler subtask and then optimize that subtask.

So you run into the problem that's been mentioned before about latency if you're, like, chaining calls. And it's not always easy to, like, decompose the, for example, to decompose the process of responding to a user in a chatbot.

That's kind of challenging. But but when you can do this, this is another great way to, like, get yourself out of the hole of needing to evaluate open-ended generations. But if you're stuck evaluating natural text, there's a couple of, like, basic approaches.

You can just keep a few trusty test cases at hand. And, you know, if it does well on those couple of test cases, looks good to me, let's ship it. Unclear what to do when it fails, just, like, hit the language model with a wrench.

But this is what kind of grows out into that data engine. You start with something like this, and then you start adding stuff from your production observations into it, and then you, like, put it in a GitHub action.

And, like, now that's, like, that's that's basically testing,right? That's is certified software. You can, like, you can try and get user feedback, and you want to do it as naturally as possible. Like, if you're, like, you would you really want to reveal preferences from user behavior.

So the image generation world is very ahead of the language modeling world, I think, on this. If you look at Midjourney, for example, that is what Honeycomb did with their query builder. They attached it to, get this, downstream business objectives.

Wow. What a way to build a software system. That's theright way to do it. And so, like, connecting a chain of metrics from the actual system that you're improving to the actual downstream, like, organizational goals through things like reveal preferences of users or, like, yeah, general user behavior much better than, like, demanding users fill out a form.

You could also pay people to do that work of giving you feedback on your system with an annotation team. This is what the large this is what OpenAI does to improve their models. But as alluded to by Karina, it's actually much more effective to use language models in that place because language models are maybe not as smart as all humans, but they tend to outperform crowd workers on a large number of very textual tasks.

And so you might find that the task of, like, annotating and improving your data, if you're at the point where you're starting to think about crowd workers, you'll find lower cost for equal performance with, like, GPT-3.5 Turbo is, like, a median crowd worker.

GPT-4 is, like, a 90th percentile crowd worker. And maybe a hybrid approach with some crowd workers, a smaller number of crowd workers managing some language models is also been discussed.

Allright. So the there's, like, not that much to say in the end about that, like, that aspect of the engineering of systems. We don't know what the user interfaces and the user experiences are going to look like. We don't know we don't know a lot about how to engineer these things to be correct.

So I guess the exciting thing about that is that the people who are here in this room on the stream at the summit are here to fill in all of the steps here that lead to from the circle to the fully drawn owl by, like, by figuring it out, by, like, trying things and sharing what works, like people will do at this conference.

So that's why I'm excited to be here, and I hope you are as well. Allright. Thank you, everyone.