No more bad outputs with structured generation: Remi Louf

So yeah, my name is Rémi. I'm the co-author and co-maintainer of the open-source library Outlines, which some of you might know, and I'm also the CEO and co-founder of .txt or .text, whichever you prefer. We're more traditional machine-learning people, and the motivation for work is the very simple observation that large language models are fundamentally flawed.

I'll give you a very simple example: you're trying to extract flight information from a bunch of emails. Of course you want them to be, you know, a JSON object, you know, with origin, destination, etc. So you go to OpenAI, you prompt the model to death, you threaten it, you use function calling, and what you get sometimes as an answer is JSON decode error.

I gave you very simple examples, but this has, like, very fundamental implications because computing rests on interfaces. We're able to build modular infrastructures and very complex infrastructure because we can trust the API over the pieces of code. And here what I'm telling you, and what you've probably witnessed, you can't actually trust large language models to return consistent outputs.

And, you know, in short is that the technology for agents is currently not there.

So the good news is that structured generation, which is the ability of guiding the model to return to the specific structure, actually solves. All see is that it allows you to be GPT-4 as sort of a byproduct. The goals for today are first to introduce the open-source library Outlines for those of you who don't know about it, then very briefly explain how it works, I won't get into the technical details, and then try to convince you that you should use it today for, you know, most of the workflows that you have to deal with, and sort of a very short glimpse into the near future.

So Outlines is a Python library, emphasis on library. You can actually include Outlines in your workflow, and it's not like frameworks where you have to make your workflow, you know, fit inside a framework. I think as a result it's been adopted by VLLM and TGI in the serving frameworks, and if you use function calling in either of these libraries you're actually using Outlines under the hood.

Outlines under the hood. So I'm co-author, but Outlines would be nothing without its contributors. Today it's 87, I think it might be 88, I think I merged a PR this morning, I don't remember. And so Outlines would be nothing without all these people, and I thank them, thank them a lot.

People thought we were crazy about a year ago when we were talking about structured generation, but since then we're pretty happy because it looks like people are sort of caught up with the topic and realize that you can actually, you know, you can actually do structured output.

So just now, just a run-through, quick run-through Outline. So usually generating text happens in three stages. The first stage is that you need to choose the model and instantiate it. So Outlines is purely focused on open-source models. We have integration with six different model providers, Transformers, Llama CPP, and also recently we added MLXLM.

We have an integration with OpenAI, but that's mostly for us to compare the results that we get with open models with the results that are given by OpenAI. The second step is to, I mean, generate text. What you do is that you instantiate a generator using generate.text.

Here we just want to, you know, return a single sentence. So we're telling the generator, "Stop whenever you encounter a period." And question is, "Describe," then you call the generator with your prompt, and here is, "Describe the benefits of structured generation in one sentence."

And you'll have to wait for 10 more minutes, hopefully less. Okay, now we get into structured generation. So with Outlines, without Outlines, if you ask what is the IP address of the public Google DNS servers and you just generate text, you just let the LLM do its thing, then generally it will yap for a long time, you know, 100 tokens, 500 tokens, and the answer will be somewhere in there.

And the way you extract the answer is using regular expressions generally. Here, what you can do with Outlines is actually taking that regular expression that you use, you would use to extract the answer, and use it to guide the model, to tell the model this is the structure that the output should follow.

And as you see, you kind of remove the yapping, you print the, you just call generate.regex, call the generator, and what you get is just the result, and it's actually the correct answer. That was with Mistral 7B v01.

Regular expressions are not the only way to define structure. Something that people need a lot in practice is that JSON. And Outlines allow you to generate, to generate text that, you know, is a JSON object with a given structure.

The way you specify the structure is using JSON schema, or you can pass pydantic models as well. Now you might notice on the flight information, so here we're, you know, it's the example that I used at the beginning, you're extracting flight information from an email.

I could have used string as a type for origin and destination, but I did not. I used actually a custom type that we implemented in Outlines, and the reason is that origin and destination have way more structure than just text.

It's actually, you know, it's an airport code that has three letters that's capitalized, and you can actually specify more and more structure, all the structure that you have in your problem, basically. You can use this with vision models.

That's something that we merged recently. So here we took, I think it's a picture from Wikipedia of a dish. We tell the model what is the JSON that we expect as an output, and then we instantiate the generator and then pass the image and the prompt to the generator, and we get valid JSON.

If you want to install Outlines and you think you could benefit from structured generation, then it's very simple, just pip install Outlines. Now I'm going to try to very quickly explain how it works. So models themselves, what Mistral and Cohere on this wall are doing is they're actually training model weights.

What a model does is that you input a prompt, you send a prompt, it's like token IDs, and what you get as an output is not text, it's logits, it's a probability distribution over the next token. Now what happens after that, when you want to generate text, the first step is that you have a logit processor that biases the logits.

You probably use this every day, actually, without noticing it. When you use temperature or you use top k top p sampling, you're actually biasing the logits. And once you have your biased logits, you use a sampling algorithm, then you get a token, and once you have your token, you add it to the prompt and then feed it back to the LLM.

And where we fit is here. We actually wire the model, whenever the model generates logits, we look at every token and we say, "If I add this token to the current generation, is it going to violate the structure?"

If the answer is yes, we, like, we mask it so that it doesn't get generated. Now that story is very simple. What is really hard is doing that efficiently, and that's what we figured out at .text, and that's what makes us different from the other libraries like guidance or LMQL that do structured generation.

And now I'm going to convince you that there is absolutely no reason to not use, sorry for the double negation here, to not use structured generation. The first reason is that most text is structured. I talked to you about JSON earlier, we talked about regular expressions, but here I just took the GSMAK dataset.

If you look at it, if you're not me and don't turn everywhere, notice it immediately. If you look at theright, you can actually see that it's highly structured. It's always queue, period, text until a question mark, then etcetera, so on and so forth, arithmetic operation, which is defined by a context-free grammar, and you could actually express this in Outlines and just get the answer at the end, which is, you know, very sexy.

So there's a lot of structured text out there, not just, thank you.

I'll be, I'll be quick. Of course, the second benefit is that you get valid structure. I mean, that's an obvious one, that's what we're doing it. I like this meme at the bottom, this is what people are currently doing.

It's just crazy stuff to get valid JSON as an output, and it's not even guaranteed, and here with Outlines you just sample what you want. It's as simple as this. And as an experiment, it's actually an experiment that PrettyBase did.

They took Mistral 7B v01, they used a version of CohenNL that they modified so that it gives structured output, JSON. What they found is Mistral 7B v01 only gets valid JSON 17% of the time. When you add structured generation on top of it, you get 99.9%, and that's without optimizing the prompt.

So you can actually get, you know, you can actually get better than this.

The nice thing is that it also adds negligible overhead, so you actually have, you know, you don't have to fear for that affecting inference time, which is the highly, you know, highly non-trivial thing. Here we compared the overhead introduced by guidance when they do structured generation, you know, as a function of the number of generated tokens, and at the bottom is Outlines.

Outlines says approximately zero until the end. That's a trade-off, there's a compilation time, but during inference it doesn't slow down inference. Now we're at a point where we could integrate this in Grok, and you wouldn't see the difference between structured and unstructured.

So no overhead, but even more than no overhead, it is faster to generate text with structured generation. The first is that when you take JSON, you don't need to generate the tokens that correspond to the bracket and to the field names.

I know that in advance, I don't need to ask the model to return these tokens. So here on this very simple example, only 5 out of 10 tokens need to be generated, so only one half. But there's an even more subtle way in which it accelerates inference, and this is the example that we took at the beginning.

So here I asked ChatGPT, like a good model, like ChatGPT, the same question. What is the public, like the, of Google's public DNS servers? And ChatGPT took 50 tokens, you know, it yapped it, yapped it, yapped, and gave it after 50 tokens.

It's not as bad, it could get a lot worse with lesser models. But when you use structured generation, you just generate 8 tokens. So that's a subtle win which it accelerates inference by a lot.

Then it improves efficiency, and that's probably the most actually mind-blowing result that we've had. So here what you're looking at is the accuracy on GSMAK with, again, Mistral 7B v01, structured and unstructured. And here we look at the accuracy as a function of the number of shots, so the number of examples that you give to the model before asking the question.

And what we found is that, yeah, for unstructured, normal, one shot is worse than 8 shots, that's completely expected. But what we found with structured is that you actually, and that's really surprised us, is that you actually get in the same ballpark in terms of accuracy with one shot as you do with 8 shots, which is surprising for a machine learning.

It's not like you would think that examples are there to teach the model about the task, but it looks like it's actually there to teach the model about the structure of the problem.

There's more investigation to do on this line, but that was very mind-blowing. And the last one, which probably, you know, after faster, a lot of people care about here, is that it does improve the performance of open-source models.

Here what you're looking at is the Berkeley function calling leaderboard, a simple function benchmark, and we'll look at the accuracy. So the first thing we did is that we took Microsoft 3D Medium model, which is a small model, but we looked at its accuracy without structured generation.

It's 86%, which is pretty good for an open model. Phi Three is actually a pretty good model. When you add structured generation, you get 96.5%, and as a comparison, GPT-4, the best version of GPT-4 on this task gets 93.5% on this benchmark.

And now there are two things to note is that 96.5% gets dangerously useful. And the second thing is that we have open models that are available today that can beat, you know, larger models without fine-tuning. So it's a pretty huge win for open models, and that's why I'm really bullish on open models.

I think, you know, as a community, we can actually extract a lot more out of these models.

And this is just a glimpse. The work that I just showed you is what we did at .text about a year ago. Since then, we've generalized from regular expression to what you call context-free grammars. Context-free grammars are used to define code, they're used to define protein structure, I mean, and to define as well what I showed you earlier on the GSMAK example.

So we can do the same thing, structured generation with no overhead with context-free grammar. We also started working on semantics, like adding some semantic constraints to the generation. And one very popular example of this is SQL. Text to SQL, most models.

That SQL syntax, usually what they get wrong is they hallucinate table or column names. And hey, internally we're able to get perfect text to SQL. So I can't guarantee you that the query will be correct and give you the answer that you expect, but I can guarantee you that it will run.

So that's a pretty huge advance in text to SQL. And what else? Oh yeah, and we're also starting to bubble up computations into the structured generation, into the model architecture, because when you think about it, we're biasing logits.

When you're biasing logits, the model is actually doing computation for nothing. And so you can gain even more inefficiency by preventing the model from doing these computations in the first place. And that's all work that we'll actually publish in a blog post, I think, in the next couple of weeks.

So all that to say that if you're doing, if you're not doing a chatbot, there's a really good chance that you will be using structured generation, you know, it's just a matter of time until you adopt it, I think.

Our users are pretty, pretty, pretty happy. So yeah, thank you for your attention. And

all the crazy claims that I made, you can go to the QR code, there's a link to all the blog posts.