DSPy: The End of Prompt Engineering - Kevin Madura, AlixPartners

Thanks, everybody, for—uh, for joining. I'm here to talk to you today about DSPy, um, and feel free to jump in with questions or anything throughout the talk. It's, you know, I don't span—I don't plan on spending the full hour and a half or so.

I know it's the last session of the day, so, um, keep it casual. Feel free to jump in. I'll start with a little bit of background. Don't want to go through too many slides. I'm technically a consultant, so I have to do some slides.

But we will dive into the code for the latter half, and there's a GitHub repo that you can download to follow along and play around with it on your own. Um, so, how many people here have heard of DSPy?

Almost everyone, that's awesome. How many people have actually used it, kind of day-to-day, in production or anything like that? Three. Okay, good. So hopefully we can convert some more of you today. Um, so, high level, DSPy—this is straight from the website—it's a declarative framework for how you can build modular software.

And most important for someone like myself, I'm not necessarily, uh, an engineer that is writing code all day, every day. As I mentioned before, I'm more of a technical consultant, so I run across a variety of different problems.

Could be, um, an investigation for a law firm, it could be helping a company understand how to improve their processes, how to deploy AI internally. Maybe we need to look through 10,000 contracts to identify a particular clause, um, or paragraph.

And so, DSPy has been a really nice way for me personally and my team to iterate really, really quickly. I'm building these applications. Most importantly, building programs. It's not, um, it's not kind of iterating with prompts and tweaking things back and forth.

It is building a proper Python program, and DSPy is a really good way for you to do that. So I mentioned before, there's a repo online. If you want to download it now and kind of just get everything set up, I'll put this on the screen later on.

Um, but if you want to go here and just kind of download some of the code, it, uh, it's been put together over the past couple days, so it's not going to be perfect production-level code. It's much more of utilities and little things here and there to just come and kind of demonstrate the usefulness, demonstrate the point of what we're talking about today.

In that, and we'll walk through all of these, these different use cases. So, um, sentiment classifier, going through a PDF, some multimodal work, uh, a very, very simple web research agent detecting boundaries of a PDF document. You'll see how to summarize basically arbitrary length text and then go into an optimizer, uh, with JetBoat.

But before we do that, just again, kind of level set, the biggest thing for me personally, DSPy is a really nice way to decompose your logic into a program that treats LLMs as a first-class citizen. So at the end of the day, you're fundamentally just calling a function that under the hood just happens to be an LLM.

And DSPy gives you a really nice, intuitive, easy way to do that, with some guarantees about the input and output types. So of course there are structured outputs, of course there are other ways to do this, Pydantic and others, um, but DSPy has a set of primitives that when you put it all together, allows you to build a cohesive, modular piece of software that you then happen to be able to optimize.

And we'll get into that, uh, in a minute. So, just a few reasons of why I'm such an advocate. It sits at—it sits at this really nice level of abstraction. So it's, I would say, it doesn't get in your way as much as a LangChain, and that's not a knock on LangChain.

It's just a different kind of paradigm in the way that DSPy is, is structured. Um, and it allows you to focus on things that actually matter. So you're not writing dot choices, zero messages content, you're not—you're not doing string parsing, you're not doing a bunch of stuff under the hood.

You're just declaring your intent of how you want the program to operate, what you want your outputs and outputs to be. Because of this, it allows you to create computer programs, as I mentioned before, not just tweaking strings and sending them back and forth.

You are building a program first. It just happens to also use LLMs. And really the most kind of important part of this is that, and Omar, the Khatib, the, uh, the founder of this, or the original developer of it, had this really good podcast with A16Z.

I think it came out just like two or three days ago. But it—he put it a really nice way. He said it's built with a systems mindset, and it's really about how you're encoding or expressing your intent of what you want to do.

Most importantly, in a way that's transferable. So the design of your system, I would imagine, or your program, isn't going to move necessarily as quickly as maybe the model capabilities are under the hood. When we see new releases almost every single day, different capabilities, better models.

And so DSPy allows you to structure it in a way that retains the control flow, uh, retains the intent of your system, your program, um, while allowing you to bounce from model to model to the extent that you want to or need to.

Convenience comes for free. There's no parsing, JSON, things like that. Again, it sits at a nice level of abstraction where you can still understand what's going on under the hood. If you want to, you can go in and tweak things.

But it allows you to kind of focus on just what you want to do, while retaining the level of precision that you—that I think most of us would like to have in kind of building your programs. Um, as mentioned, it's robust to kind of model and paradigm shifts.

So you can, again, keep the logic of your program, um, but keep those LLMs infused in, uh, basically inline. Now, that being said, you know, there are absolutely other great libraries out there, Pydantic AI, LangChain, and there's many, many others that allow you to do similar things.

Agno is another one. Um, this is just one perspective. And, um, it may not be perfect for your use case. For me, it took me a little bit to kind of grok how DSPy works, and you'll see why that is in a minute.

Um, so I would just recommend kind of have an open mind, play with it. Um, run the code, tweak the code, do whatever you need to do, um, and just see how it might work, might work for you.

And really this talk is more about ways that I found it useful. It's not a dissertation on the ins and outs of every nook and cranny of DSPy. It's more of, you know, I've run into these problems myself, now I naturally run to DSPy to solve them, uh, and this is kind of why.

And the hope is that you can extrapolate some of this to your own use cases. So we'll go through everything, uh, fairly quickly here. But the core concepts of DSPy really come down to arguably five or these six that you see on the screen here.

So we'll go into each of these in more detail, but high level, signatures specify what you want the LL—what, basically what you want your function call to do. This is when you specify your inputs, your outputs. Inputs and outputs can both be typed, um, and you defer the rest of the, basically the how, the implementation of it to the LLM.

And we'll see how we—how that all kind of comes together, uh, in a minute. Modules themselves are ways to logically structure your program. They're based off of signatures. So a module can have one or more signatures embedded within it, in addition to, uh, additional logic.

And it's based off of, um, PyTorch and some of the—in terms of like the methodology for how it's structured. And you'll see how that, uh, comes to be in a minute. Tools, we're all familiar with tools, MCP and others.

And really tools, fundamentally, as DSPy looks at them, are just Python functions. So it's just a way for you to very easily expose Python functions to the LLM within the DSPy kind of ecosystem, if you will. Um, adapters live in between your signature and the LLM call itself.

I mean, as we all know, prompts are ultimately just strings of text that are sent to the LLM. Signatures are a way for you to express your intent at a—at a higher level. And so adapters are the things that sit in between those two.

So it's how you translate your inputs and outputs into a format that basically explodes out from your initial signature into a format that is ultimately the prompt that is sent to the LLM. And so, you know, there's some debate or some research on if certain models perform better with XML, as an example, or BAML, or JSON, or others.

And so adapters give you a nice, easy abstraction to basically mix and match those at will, as you want. Optimizers, um, are the most interesting and for whatever reason the most controversial part of DSPy. That's kind of the first thing that people think of, or at least when they hear of DSPy, they think optimizers.

We'll see a quote in a minute. It's not optimizers first, it is just a nice added benefit and a nice capability that DSPy offers in addition to the ability to structure your program with the signatures and modules and everything else.

Um, and metrics are used in tandem with optimizers that basically defines how you measure success in your—in, uh, your DSPy program. So the optimizers use the metrics to determine if it's finding theright path, if you will. So signatures, I mentioned before, it's how you express your intent, your declarative intent.

It can be super simple strings. This is the weirdest part for me initially, but it is one of the most powerful parts, uh, of it now. Or it can be more complicated, class-based objects. If you've used Pydantic, that's basically what it runs on under the hood.

So this is an example of one of the class-based signatures. Again, it's basically just a Pydantic object. What's super interesting about this is that

the—the names of the fields themselves act almost as like mini prompts, as part of the prompt itself. And you'll see how this comes to life in a minute. But what's ultimately passed to the model from something like this is, it will say, okay, your inputs are going to be a parameter called text, and it's based off of the name of that particular parameter in this class.

And so these things are actually passed through. And so it's very important, uh, to be able to name your parameters in a way that is intuitive for the model to be able to pick it up. Um, and you can add some additional context or what have you in the description field here.

So most of this, if not all of this, yes, it is proper, you know, typed Python code, but it's also—it also serves almost as a prompt ultimately that feeds into the model. Um, and that's basically translated through the use of adapters.

Um, and so just to highlight here, like these, it's the ones that are a little bit darker and bold, you know, those are the things that are effectively part of the prompt, uh, that's been sent in. And you'll see kind of how DSPy works with all of this and formats it in a way that, again, allow you to just worry about what you want, worry about constructing your signature instead of figuring out how best to word something in the prompt.

Go ahead. Uh, Kevin, sorry, I just want to jump in with one thing. Because like a lot of people get hung up with like, but I have a really good prompt. Sure. And then I don't want this thing to get prompted.

And for that, you literally just put your really good prompt in the docs and it'll send it. Yeah. And then you literally start from there. That's exactlyright. A lot of people's like aversion to DSPy, I think, comes from thinking that there's a really good prompt in.

Sure. And so it's like, okay, this is a good prompt. So the question for folks online is, what if I already have a great prompt? I've done all this work, I'm an amazing prompt engineer, I don't want my job to go away or whatever.

Um, yes. So you can absolutely start with a custom prompt or something that you have demonstrated works really well. And you're exactlyright. That can be done in the docstring itself. There's some other methods in order, uh, for you to inject basically system instructions or add additional things at certain parts of the ultimate prompt.

And/or, of course, you can just inject it in the final string anyway. I mean, it's just, you know, a string that is constructed by DSPy. So, um, absolutely, this doesn't necessarily prevent you. It does not prevent you from adding in some super prompt that you already have, absolutely.

Um, and to your point, it is—it can serve as a nice starting point from which to build the rest of the system.

Here's a shorthand version of the same exact thing, which to me, the first time I saw this, this was like baffling to me. Um, but that's exactly how it works, is that you're basically, again, kind of deferring the implementation or the logic or what have you to DSPy and the model to basically figure out what you want to do.

So in this case, if I want a super, super simple text, uh, sentiment classifier, this is basically all you need. You're just saying, okay, I'm going to give you text as an input, I want the sentiment as an integer as the output.

Now you probably want to specify some additional instructions to say, okay, your sentiment, you know, a lower number means negative, you know, a higher number is more positive sentiment, etc. But it just gives you a nice kind of easy way to kind of scaffold these things out in a way that you don't have to worry about like, you know, creating this whole prompt from hand.

It's like, okay, I just want to see how this works. And then if it works, then I can add the additional instructions, then I can create a module out of it, or, you know, whatever it might be. It's these shorthand, or it is this shorthand that makes experimentation and iteration incredibly quick.

So modules, it's that base abstraction layer for DSPy programs. There are a bunch of modules that are built in. And these are a collection of kind of prompting techniques, if you will. And you can always create your own module.

So to the question before, if you have something that you know works really well, sure, yeah, put it in a module. That's now kind of the base assumption, the base module that others can build off of. And all of DSPy is meant to be composable, optimizable.

And when you deconstruct your business logic or whatever you're trying to achieve by using these different primitives, it all—it's intended to kind of fit together and flow together. Um, and we'll get to optimizers in a minute, but at least for me and my team's experience, just being able to logically separate the different components of a program, but basically inlining, uh, LLM calls has been incredibly powerful for us.

And it's just an added benefit that at the end of the day, because we're just kind of in the DSPy paradigm, we happen to also be able to optimize it at the end of the day. Uh, so it comes with a bunch of standard ones built in.

I don't use some of these bottom ones as much, although they're super interesting. Um, the base one at the top there is just dspy.predict. That's literally just, you know, an LLM call. That's just, uh, a vanilla call. Chain of thought, uh, probably isn't as relevant anymore these days because models have kind of ironed those out.

But, um, it is a good example of the types of, um, kind of prompting techniques that can be built into some of these modules. Um, and basically all this does is add, um, some of the, uh, strings from literature to say, okay, let's think step by step or whatever that might be.

Same thing for React and Codeact. React is basically the way that you expose the tool to the model. So it's wrapping and doing some things under the hood with, um, basically taking your signatures and, uh, it's injecting the Python functions that you've given it as tools.

And basically React is how you do tool calling in DSPy. Program with thought is, uh, is pretty cool. It kind of forces the model to think in code and then will return the result. Um, and you can give it a—it comes with a Python interpreter built in, but you can give it some custom one, some type of custom harness if you wanted to.

Um, I haven't played with that one too, too much, but it is super interesting if you have like a highly technical problem or workflow or something like that where you want the model to inject reasoning in code at certain parts of your pipeline.

That's kind of a really easy way to do it. And then some of these other ones are basically just different methodologies for comparing outputs or running things in parallel. So here's what one looks like. Again, it's fairly simple.

It's, you know, it is a Python class at the end of the day. Um, and so you do some initialization up top. In this case, you're seeing the, uh, uh, the shorthand signature up there. So I'm—this module, uh, just to give you some context is an excerpt from, um, one of the Python, um, files that's in the repo.

It's basically taking in a bunch of time entries and making sure that they adhere to certain standards, making sure that things are capitalized properly, or that there are periods at the end of the sentences, or whatever it might be.

That's from a real client use case where they had hundreds of thousands of time entries and they needed to make sure that they all adhere to the same format. This was one way to kind of do that very elegantly, at least in my opinion, was taking up top, you can define the signature, it's adding some additional instructions that were defined elsewhere, and then saying for this module, the change tense, um, call is going to be just a vanilla predict call.

And then when you actually call the module, you enter into the forward function, which you can basically intersperse the LLM call, which would be the first one, and then do some kind of hard-coded business logic beneath it.

Uh, tools, as I mentioned before, these are just vanilla kind of Python functions. It's the DSPy's tool interface. So under the hood, DSPy uses LiteLLM. And so there needs to be some kind of coupling between the two, but fundamentally, um, any type of tool that you would use elsewhere, you can also use in DSPy.

And this is probably obvious to most of you, but here's just an example. You have two functions, get weather, search web. You include that with a signature. So in this case, I'm saying the signature is, I'm going to give you a question, please give me an answer.

I'm not even specifying the types, it's just going to infer what that means. Uh, I'm giving it the get weather and the search web tools, and I'm saying, okay, do your thing, but only go five rounds, just so it doesn't spin off into something crazy.

And then a call here is literally just calling the React agent that I created above with the question, what's the weather like in Tokyo? We'll see an example of this in the code session. But basically what this would do is give the model the prompt, the tools, and let it do its thing.

So adapters, before I cover this a little bit, they're basically prompt formatters, if you will. So the description from the docs probably says it best. It takes your signature, the inputs, other attributes, and it converts them into some type of message format that you have specified or that the adapter has specified.

And so as an example, the JSON adapter, taking, say, a Pydantic object that we defined before, this is the actual prompt that's sent into the LLM. And so you can see the input fields. So this would have been defined as, okay, clinical note type string, patient info as a patient details object, which would have been defined elsewhere.

And then this is the definition of the patient info. It's basically a JSON dump of that Pydantic object. Go ahead. So this idea that it's like a base adapter or default. Yeah. That's good for most cases, and this is if you want to tweak that to do something more specific.

That'sright. Yeah, the question was if there's a base adapter, and would this be an example of where you want to do something specific? Answer is yes. So, um, there's a guy, Prashanth, who is, um, I have his Twitter at the end of this presentation, but he's been great.

He did some testing comparing the JSON adapter with the BAML adapter. Um, and you can see just intuitively, even for us humans, the way that this is formatted is a little bit more intuitive. It's probably more token efficient too.

Just considering, like, if you look at the messy JSON that's here versus the, I guess, slightly better formatted BAML that's here, um, can actually improve performance by, you know, 5 to 10 percent depending on your use case. So it's a good example of how you can format things differently.

The rest of the program wouldn't have changed at all. You just specify the BAML adapter, and it totally changes how the information is presented under the hood to the LLM.

Multimodality, I mean, this obviously is more at the model level, but DSPy supports multiple modalities by default. So images, audio, some others. Um, and the same type of thing, you kind of just feed it in as part of your signature, and then you can get some very nice clean output.

This allows you to work with them very, very, very easily, very quickly. And for those, uh, eagle-eyed participants, you can see the first, uh, line up there is attachments. It's probably a lesser-known library. Another guy on Twitter is awesome, uh, Maxine, I think it is.

Uh, he created this library that just is basically a catch-all for working with different types of files and converting them into a format that's super easy to use with LLMs. Um, he's a big DSPy fan as well, so he made basically an adapter that's specific to this.

But that's all it takes to pull in images, PDFs, whatever it might be. You'll see some examples of that. Uh, it just makes, at least has made my life super, super easy.

Here's another example of the same sort of thing. So this is a PDF of a Form 4 form, some, you know, public SEC form from NVIDIA. Um, up top, I'm just giving it the link. I'm saying, okay, attachments, do your thing, pull it down, create images, whatever you're going to do.

I don't need to worry about it. I don't care about it. This is super simple RAG, but basically, okay, I want to do RAG over this document. I'm going to give you a question. I'm going to give you the document, and I want the answer.

Um, and you can see how simple that is. Literally just feeding in the document, how many shares were sold. Interestingly here, I'm not sure if it's super easy to see, but you actually have two transactions here. So it's going to have to do some math, likely under the hood.

And you can see here the thinking and the ultimate answer. Go ahead. Is it, on the RAG step, is it creating a vector store of some kind or creating embeddings and searching over those? Is there a bunch going on in the background there or what's happening?

This is poor man's RAG. I should have clarified. This is literally just pulling in the document images. And I think attachments will do some basic OCR under the hood. Um, but it doesn't do anything other than that. That's it.

All we're feeding in here, the actual document object that's being fed in, yeah, is literally just the text that's been OCR'd. The images, the model does the rest.

Allright, so optimizers. Uh, let's see how we're doing. Okay. Um, optimizers are a super powerful, super interesting concept. There's been some research, um, that argues, I think, that it's just as performant, if not in certain situations, more performant than fine-tuning would be for certain models, for certain situations.

There's all this research about in-context learning and such. And so whether you want to go fine-tune and do all of that, nothing stops you. But I would recommend at least trying this first to see how far you can get without having to set up a bunch of infrastructure and, you know, go through all of that.

See how the optimizers work. Um, but fundamentally what it allows you to do is DSPy gives you the primitives that you need and the organization you need to be able to measure and then quantitatively improve that performance. And I mentioned transferability before.

This transferability is enabled arguably through the use of optimizers. Because if you can get, okay, I want to have the classification task, works really well with 4.1, but maybe it's a little bit costly because I have to run it a million times a day.

Can I try it with 4.1 nano? Okay, maybe it's at 70%, whatever it might be. But I run the optimizer on 4.1 nano, and I can get the performance back up to maybe 87%. Maybe that's okay for my use case.

But I've now just dropped my cost profile by multiple orders of magnitude. And it's the optimizer that allows you to do that type of model and kind of use case transferability, if you will. But really all it does at the end of the day under the hood is iteratively optimize or tweak that prompt, that string under the hood.

And because you've constructed your program using the different modules, DSPy kind of handles all of that for you under the hood. So if you compose a program with multiple modules and you're optimizing against all of that, it by itself, DSPy, will optimize the various components in order to improve the input and output performance.

And we'll take it from the man himself, Omar. You know, DSPy is not an optimizer. I've said this multiple times. It's just a set of programming abstractions or a way to program. You just happen to be able to optimize it.

Um, so again, the value that I've gotten and my team has gotten is mostly because of the programming abstractions. It's just this incredible added benefit that you are also able to, should you choose to optimize it afterwards. And I was listening to this, to our question and, uh, Carpathy the other day.

And this kind of, I was like prepping for this talk and this like hit home perfectly. I was thinking about the optimizers. And someone smarter than me can, you know, please correct me, but I think this makes sense because he was basically talking about using LLM as a judge can be a bad thing because the model being judged can find adversarial examples and degrade the performance or basically, um, create a situation where the judge is not, uh, not scoring something properly.

Um, because he's saying that the model will find these little cracks. It'll find these little spurious things in the nooks and crannies of the giant model and find a way to cheat it. Basically saying that LLM as a judge can only go so far until the other model, uh, finds those adversarial examples.

If you kind of invert that and flip that on its head, it's this property that the optimizers for DSPy are taking advantage of to find the nooks and crannies in the model, whether it's a bigger model or smaller model, to improve the performance against your data set.

So that's what the optimizer is doing. It's finding these nooks and crannies in the model to optimize and improve that performance. So a typical flow, I'm not going to spend too much time on this, but fairly logical constructor program, which is decomposing your logic into the modules.

You use your metrics to define basically the contours of how the program works, and you optimize all that through, um, to get your final result.

So another talk that this guy, Chris Potts, just had maybe two days ago, um, where he made the point, this is what I was mentioning before, where JEPA, which is, uh, you probably saw some of the talks the other day, um, where the optimizers are on par or exceed the performance of something like GRPO and other kind of fine-tuning methods.

So pretty impressive. I think it's an active area of research. People a lot smarter than me, like Omar and Chris and others are leading the way on this. But, uh, point being, I think prompt optimization is a pretty exciting place to be, and if nothing else, is worth exploring.

And then finally, metrics. Again, these are kind of the building blocks that allow you to define what success looks like for the optimizer. So this is what it's using. And you can have many of these, and we'll see examples of this where, again, at a high level, your program works on inputs, it works on outputs.

The optimizer is going to use the metrics to understand, okay, my last tweak in the prompt, did it improve performance? Did it degrade performance? And the way you define your metrics, uh, provides that direct feedback for the optimizers to work on.

Uh, so here's another example, a super simple one from that time entry example I mentioned before. Um, so they can be, the metrics can either be like fairly rigorous in terms of like, does this equal one or, you know, some type of a quality check, or a little bit more subjective where you're using LLM as a judge to say, whatever, was this generated, um, string, does it adhere to these, you know, various criteria, whatever it might be.

But that itself can be a metric. And so all of this is to say, it's a very long-winded way of saying, in my opinion, this is probably most, if not all, of what you need to construct arbitrarily complex workflows, data processing pipelines, business logic, whatever that might be, different ways to work with LLMs.

If nothing else, DSPy gives you the primitives that you need in order to build these modular composable systems. So if you're interested, some people online, um, there's many, many more. There's a Discord community as well. Um, but usually these people are on top of the latest and greatest.

And so I would recommend giving them a follow. You don't need to follow me. I don't really do much. But, uh, the others on there are really pretty good. Okay. So the fun part, we'll actually get into some code.

So if you haven't had a chance, now's your last chance to get the repo. Um, but I'll just kind of go through a few different examples here of what we talked about, maybe.

Yeah. Okay. Okay. So I'll set up Phoenix, which is from Arise, uh, which is basically an observability platform. Uh, I just did this today, so I don't know if it's going to work or not, but we'll see. We'll give it a shot.

Uh, but basically what this allows you to do is have a bunch of observability and tracing for all the calls that are happening under the hood. We'll see if this works. We'll give it like another five seconds.

Um, but it should, I think, automatically do all this stuff for me. Yeah. So let's see. Yeah. Allright. So something's up. Okay. Cool. So I'm just going to run through the notebook, which is a collection of different use cases, basically putting into practice a lot of what we just saw.

Feel free to jump in, any questions, anything like that. We'll start with this notebook. There's a couple of other, uh, more proper Python programs that we'll walk through afterwards. Uh, but really the intent is a rapid-fire review of different ways that DSPy has been useful to me and others.

So load in the .env file. Usually I'll have some type of config object like this where I can very easily use these later on. So if I'm like, call it like model mixing. So if I have like a super hairy problem or like some workload I know will need the power of a reasoning model like GPT-5 or something else like that, I'll define multiple LLMs.

So like one will be 4.1, one will be 5. Maybe I'll do a 4.1 nano, um, you know, Gemini 2.5 Flash, stuff like that. And then I can kind of intermingle or intersperse them depending on what I think or what I'm reasonably sure the workload will be.

And you'll see how that comes into play in terms of classification and others. Um, I'll pull in a few others here. I'm using OpenRouter for this. So if you have an OpenRouter API key, I would recommend plugging that in.

So now I have three different LLMs I can work with. I have Claude, I have Gemini, I have 4.1 Mini. And then I'll ask basically for each of them, who's best between Google and Anthropic, OpenAI? All of them are hedging a little bit.

They say, subjective, subjective, undefined. Allright, great. It's not very helpful. But because DSPy works on Pydantic, I can define the answer as a literal. So I'm basically forcing it to only give me those three options. And then I can go through each of those, and you can see each of them, of course, chooses their own organization.

Um, the reason that those came back so fast is that DSPy has caching automated under the hood. So as long as nothing has changed in terms of your signature definitions or basically if nothing has changed, this is super useful for testing, it will just load it from the cache.

Um, so I ran this before, that's why those came back so quickly. Uh, but that's another kind of super useful, um, piece here. Let's see.

Okay. Make sure we're up and running. So if I change this to hello with a space, you can see we're making a live call. Okay, great. We're still up. So super simple sentiment classifier. Obviously this can be built into something arbitrarily complex.

Make this a little bit bigger. Um, but I'm basically, I'm giving it the text, the sentiment that you saw before, and I'm adding that additional specification to say, okay, lower is more negative, higher is more positive. I'm going to define that as my signature.

I'm going to pass this into just a super simple predict object. And then I'm going to say, okay, well, this hotel stinks. Okay, that's probably pretty negative. Now if I flip that to, I'm feeling pretty happy. Oops.

Good thing I'm not in a hotelright now. Uh, you can see I'm feeling pretty happy. Comes out to eight. And this might not seem that impressive and, you know, it's not really, but, uh, the important part here is that it just demonstrates the use of the shorthand, um, signature.

So I have the string, I have the integer, I pass in the custom instructions, which would be in the docstring if I use the class-based method. The other interesting part or useful part about DSPy comes with a bunch of usage information built in.

So, um, because it's cached, it's going to be an empty object. But when I change it, you can see that I'm using Azureright now. But for each call, you get this nice breakdown. I think it's from LightLLM, but it allows you to very easily track your usage, token usage, et cetera, for observability and optimization and everything like that.

Just nice little tidbits, uh, that are part of it here and there. Make this smaller.

Uh, we saw the example before in the slides, but I'm going to pull in that form four, uh, off of online. I'm going to create this doc objects using attachments. You can see some of the stuff it did under the hood.

So it pulled out, um, PDF Plumber, it created Markdown from it, pulled out the images, et cetera. Again, I don't have to worry about all that. Attachments make that super easy. I'm going to show you what we're working with here.

In this case, we have the form four. And then I'm going to do that poor man's rag that I mentioned before. Okay, great. How many shares were sold in total? It's going to go through that whole chain of thought and bring back the response.

That's all well and good. But the power in my mind of DSPy is that you can have these arbitrarily complex data structures. That's fairly obvious because it uses Pydantic and everything else, but you can get a little creative with it.

So in this case, I'm going to say, okay, a different type of document analyzer signature. I'm just going to give it a document, and then I'm just going to defer to the model on defining the structure of what it thinks is most important from the document.

So in this case, I'm defining a dictionary object. And so it will hopefully return to me a series of key value pairs that describe important information in the document in a structured way. And so you can see here, again, this is probably cached, uh, but I passed in, I did it all in one line in this case, but I'm saying I want to do chain of thought using the document analyzer signature.

And I'm going to pass in the input field, which is just the document here. I'm going to pass in the document that I got before. And you can see here it pulled out a bunch of great information in this super structured way.

And I didn't have to really think about it. I just kind of deferred all this to the model, to DSPy for how to do this. Now, of course, you can do the inverse in saying, okay, I have a very specific business use case.

I have something specific in terms of the formatting or the content that I want to get out of the document. I define that as just kind of your typical Pydantic classes. So in this case, I want to pull out the, if there's multiple transactions, the schema itself, important information like the filing date.

I'm going to define the document analyzer schema signature. Uh, again, super simple input field, which is just the document itself, which is parsed by attachments, gives me the text and the images. And then I'm passing in the document schema parameter, which has the document schema type, which is defined above.

And this is the, this is effectively what you would pass into structured outputs, um, but just doing it the DSPy way, where it's going to give you, um, basically the output in that specific format. So you can see it pulled out things super nicely.

Filing date, form date, uh, form type, transactions themselves, and then the ultimate answer. And it's nice because it exposes it in a way that you can use dot notation so you can just very quickly access the resulting objects.

So looking at adapters, um, I'll use another little tidbit from DSPy, which is the inspect history. So for those who want to know what's going on under the hood, inspect history will give you the raw dump of what's actually going on.

So you can see here the system message that was, uh, constructed under the hood was all of this. So you can see input fields are document, output fields are reasoning and the schema. It's going to pass these in.

And then you can see here the actual document content that was extracted and put into the text, into the prompt, uh, with some metadata. This is all generated by attachments. And then you get the response, which follows this specific format.

So you can see the different fields that are here. And it's this kind of relatively arbitrary response, um, basically format for the names, which is then parsed by DSPy and passed back to you as the user. Um, so I can do, okay, response.document schema and get the actual result.

To show you what the BAML adapter looks like, we can basically do two different calls. So this is an example from, uh, my buddy Prashanth online again. So what we do here is define Pydantic model, super simple one, patient address, and then patient details.

Patient details has the patient address object within it. And then we're going to say, we're going to create a super simple DSPy signature to say, taken a clinical note, which is a string. The patient info is the output type.

And then note, so I'm going to run this two different ways. The first time with the smart LLM that I mentioned before and just use the built-in adapter. So I don't specify anything there. And then the second one will be using the BAML adapter, which is defined there.

Um, so there's a few things going on here. One is the ability to use Python's context, which is the lines starting with, which allow you to basically break out of what the global LLM, um, has been defined as and use a specific one just for that call.

So you can see in this case, I'm using the same LLM, but if I want to change this to like LLM Anthropic or something,

I think that that should work. Um, but basically what that's doing is just offloading that call to the other, whatever LLM that you're defining for that particular call and something happened. And I'm on a VPN, so let's kill that.

Sorry, Alex Partners. Okay.

Okay, great. So we had two separate calls. One was to the smart LLM, which is I think 4.1. The other one was to Anthropic. Everything else is the exact same. The notes are the exact same, et cetera. We got the same exact output.

That's great. But what I wanted to show here is the adapters themselves. So in this case, I'm doing inspect history equals two. So I'm going to get both of the last two calls and we're going to see how the prompts are going to be different.

And so you can see here the first one, this is the built-in JSON schema. This crazy long JSON string. Yeah, LLMs are good enough to handle that, but, um, you know, probably not for super complicated ones. Um, uh, and then you see here for the second one, it used the BAML notation, which as we saw in the slides, a little bit easier to comprehend.

Um, and on super complicated use cases can actually have a measurable improvement.

Multimodal example, same sort of thing as before. I'll pull in the image itself. Let's just see what we're working with. Okay, great. We're looking at these various street signs. And I'm just going to ask it a super simple question.

It's this time of day. Can I park here now? When should I leave? And you can see I'm just passing in again the super simple, um, shorthand for defining a signature, which then I get about the Boolean in this case and a string of when I can leave.

Um, so modules themselves, it's again fairly simple. You just kind of wrap all this in a class. Question?

So does it return reasoning by default always?

Oh, good question. Yeah. So when you do.

Can you repeat the question?

Yes. So for those online, the question was, does it always return reasoning by default? When you call DSPy.chain of thought as part of the module where it's built in, it's adding the reasoning, uh, automatically into your response. You're not defining that.

It's a great question. It's not defined in the signature, as you can see up here, uh, but it will add that in and expose that to you, um, to the extent that you want to retain it for any, you know, any reason.

Uh, but that's, so if I change this to predict, you wouldn't get that same response,right? You just, you literally just get that part. Um, so that's actually a good segue to the modules. Um, so module is basically just wrapping all of that into some type of replicable, uh, logic.

Um, and so we're just, we're giving it the signature here. We're saying self-dup predict. We're, in this case, it's just a demonstration of how it's being used as a class. So I'll just add this module identifier and some sort of counter, but this can be any type of arbitrary business logic or control flow or any database action or whatever it might be.

When this image analyzer class is called, this function would run. Um, and then when you actually invoke it, this is when it's actually going to run the core logic. And so you can see I'm just passing in the, so I'm instantiating it, the analyzer of AIE123, and then I'll call it.

Great. It called that. And you can see the counter incrementing each time I actually make the call. So super simple example. Um, we don't have a ton of time, but I'll show you some of the other modules and how that kind of works out.

In terms of tool calling, fairly straightforward. I'm going to define two different functions, perplexity search and get URL content, creating a bioagent module. So this is going to define Gemini 2.5 as this particular module's, um, LLM. It's going to create an answer generator object, which is a React call.

So I'm going to basically do tool calling whenever this is called. And then the forward function is literally just calling that answer generator with the parameters that are provided to it. And then I'm creating an async version of that function as well.

So I can do that here. I'm going to say, okay, identify instances where a particular person has been at their company for more than 10 years. It needs to do tool calling to do this, to get the most up-to-date information.

And so what this is doing, I'm basically looping through, um, and it's going to call that bioagent, which is using the tool calls in the background, and it will make a determination as to whether their background is applicable per my criteria.

In this case, Satya is true. Brian should be false. Um, but what's interesting here while that's going, in a, uh, similar to the reasoning, uh, param or the reasoning object that you get back for chain of thought, you can get a trajectory back for things like React.

So you can see what tools it's calling, the arguments that are passed in, um, and the observations for each of those calls, which is nice for debugging and, and, and other, obviously other uses. Um, I want to get to the other content.

So I'm going to speed through the rest of this. This is basically an async version of the same thing. So you would run both of them in parallel. Same idea. Um, I'm going to skip the JEPA example here just for a second.

Um, I can show you what the output looks like, but basically what this is doing is creating a dataset. It is showing you what's in the dataset. It's creating a variety of signatures. In this case, it's going to create a system that categorizes and classifies different, basically help messages, um, that is part of the dataset.

So my sync is broken or my light is out or whatever it is. They want to classify whether it's positive, neutral, or negative, and the, uh, theurgency of the actual message. It's going to categorize it. And then it's going to pack all this stuff, all those different modules into a single support analyzer module.

And then from there, what it's going to do is define a bunch of metrics, which is based off of the dataset itself. So it's going to say, okay, how do we score theurgency? This is a very simple one where it's, okay, it either matches or it doesn't.

Um, and there's other ones where it can be a little bit more subjective and then you can run it. This is going to take too long. It probably takes 20 minutes or so. Um, but, uh, what it will do is basically evaluate the performance of the base model and then apply those metrics, uh, and iteratively come up with new prompts to, uh, to create that.

Now, I want to pause here just for a second because there's different types of metrics and in particular for JEPA, it uses feedback from the teacher model in this case. So it can work with the same level of model, but in particular when you're trying to use, say, a smaller model, um, it can actually provide textual feedback.

So it says, not only did you get this classification wrong, but it's going to give you some additional, um, information or feedback, as you can see here for why it got it wrong and what the answer should have been, which allows it, you can read the paper, but it basically allows it to, um, iteratively find that kind of pareto frontier of how it should, uh, tweak the prompt to optimize it based off that feedback.

It basically just tightens that iteration loop. Um, you can see there's a bunch here, um, and then you can run it and see how it works. Um, but kind of just to give you a concrete example of how it all comes together.

So we took a bunch of those examples from before. We're basically going to do a bit of, um, categorization. So I have things like contracts, I have images, I have different things that one DSPy program can comprehend and do some type of processing with.

So this is something that we see fairly regularly in terms of we might run into a client situation where they have just a big dump of files. They don't really know what's in it. They want to find something of, uh, they want to maybe find SEC filings and process them a certain way.

They want to find contracts and process those a certain way. Maybe there's some images in there and they want to process those a certain way. Uh, and so this is an example of how you would do that where if I start at the bottom here, this is a regular Python file, um, and it uses DSPy to do all those things I just mentioned.

So we're pulling in the configurations. We're setting the regular LLM, the small, and one we use for an image. As an example, Gemini models might be better at image recognition than others. So I might want to defer or use a particular model for a particular workload.

So if I detect an image, I will route the request to Gemini. If I detect something else, I'll route it to a 4.1 or whatever it might be. So I'm going to process a single file. And what it does is use our handy attachments, um, library to put it into a format that we can use.

And then I'm going to classify it. And it's not super obvious here, but I'm getting a file type from this classify file, uh, function call. And then I'm doing some different type of logic depending on what type of file it is.

So if it's an SEC filing, I do certain things. If it's a certain type of SEC filing, I do something else. Uh, if it's a contract, maybe I'll summarize it. If it's something that looks like city infrastructure, in this case, the image that we saw before, I might do some more visual interpretation of it.

Um, so if I dive into classify file super quick, it's running the document classifier. And all that is, is basically doing a predict on the image from the file and, um, making sure

it returns a type. Where is this? Returns a type, which would be document type. And so you can see here at the end of the day, it's a fairly simple signature. And so what we've done is basically take the PDF file in this case, take all the images from it, and take the first image or first few images, in this case, a list of images as the input field.

And I'm saying, okay, just give me the type. What is this? And I'm giving it an option of these document types. So obviously this is a fairly simple use case, but it's basically saying, given these three images, the first three pages of a document, is it an SEC filing?

Is it a patent filing? Is it a contract? City infrastructure? Pretty different things. So the model really shouldn't have an issue with any of those. And then we have a catch-all bucket for other. And then, as I mentioned before, um, depending on the file type that you get back, you can process them differently.

So I'm using the small model to do the same type of form for extraction that we saw before. Um,

and then asserting basically in this case that it is what we think it is. Uh, a contract in this case, we're saying, uh, let's see, I have like 10 more minutes. So we can go, we'll, we'll stop after this, uh, after this file.

But for the particular contract, we'll go, we'll create this summarizer object. So we'll go through as many pages as there are. We'll do some, uh, basically recursive summarization of that using a separate DSPy function. And then we'll detect some type of boundaries of that document too.

So we'll say, I want the summaries and I want the boundaries of the document. Um, and then we'll print those things out. So let's just see if I can run this. It's going to classify it. It should as a contract.

So it's, you're just relying on the model itself to realize that it's a city infrastructure document.

Yeah. The question was, I'm, I'm just relying on the model to determine if it's a city infrastructure. Yes. I mean, this is more just like a workshop, quick and dirty example. It's only because there's one picture of the street signs.

Um, and if we look in the data folder, I have a contract, some image that's irrelevant, the form for SEC filing, and then the parking too. Um, they're pretty different. The model should have no problem out of those categories that I gave it to categorize it properly.

In some type of production use case, you would want much more stringent or maybe even multiple passes of classification, maybe using different models to do that. Um, but yeah, given those options, at least the many times I've run it, I had no problem.

So in this case, I gave it, um, one of these contract documents and it ran some additional summarization logic under the hood. So if I go to that super quick, um, you can find all this in the code, but basically what it does is use three separate signatures to basically decompose the contents of the, the, um, the contract and then summarize them up.

So it's basically just iteratively working through each of the chunks of the document to create a summary that you see here at the bottom. And then just for good measure, we're also detecting basically the, the boundaries of the document to say, okay, here's out of the 13 pages, you have the main document and then some of the exhibits or the schedules that are a part of it.

So let me just bring it up super quick

just to show you what we're working with. This is just some random thing I found online. And you can see, so it said the main document was from page zero to six. And the way, and so 0, 1, 2, 3, 4, 5, 6 seems reasonable.

Now we have the start of schedule one. Schedule one, it says it's the next two pages. That looks pretty good. Schedule two is just the one page, nine to nine. That looks good. And then schedule three through to the end of the document.

That looks pretty good too. And so the way we did that under the hood was basically take the PDF, convert it to a list of images, and then for each of the images, pass those to classifier, um, and then use that to, well, let's just look at the code, but basically take the list of those classifications, give that to another DSPy signature to say, given these classifications of the document, give me the structure and basically give me a key pair of, you know, name of the section and two integers, a tuple of integers that detect or that, uh, determine the, um, you know, the boundaries essentially.

Um, so that's what that part does. Um, if we go back, so city infrastructure, I'll do this one super quick just 'cause it's pretty interesting on how it uses tool calls. And while this is running, I should use theright one.

Hold on.

Yeah.

Good question.

Yeah.

The second part, like I want you to generate the list of like my documents from zero to six. Do you have like original document as an input or no?

No. Uh, so let, let's just go to that, uh, super quick. So

that should be boundary detector.

So there's a blog post on this that I published probably in, in August or so that goes into a little bit more detail. The code is actually pretty crappy in that one. It's, it's going to be better here.

Um, but basically what it does is

this is probably the main logic. So for each of the images in the PDF, we're going to call classify page. We're going to gather the results. So it's doing all that asynchronously, pulling it all back, saying, okay, all these, you know, all the different page classifications that there are.

And then I pass the output of that into a new signature that says, given tuple of page, I don't even define it here. Given tuple of page and classification, give me this, I don't know, relatively complicated output of a dictionary of a string, tuple, integer, integer.

And I give it this set of instructions to say, just detect the boundaries. Like this is all very like non-production code, obviously, but the point is that you can do these types of things super, super quickly. Like I'm not specifying much, not giving it much context.

And it worked like pretty well. Like it, it's worked pretty well on most of my testing. Now, obviously there is a ton of low-hanging fruit in terms of ways to improve that, optimize it, et cetera. Um, but all this is doing is taking that signature, these instructions, and then I call react.

And then all I give it is, uh, the ability to basically self-reflect and call, um, get page images. So it says, okay, I'm going to look at this boundary. Well, let me get the, the page images for these three pages to, and make sure basically that the boundary is correct.

And then it uses that to construct the final answer. And so it's really, this is a perfect example of like the tight iteration loop that you can have both in, um, building it, but then the, you can kind of take advantage of the model's introspective ability, if you will, to use function calls against the data itself, the data it generated itself, et cetera, to kind of keep that loop going.

Question.

So under the hood, the, the beauty of DSPy then is that it enforces kind of structured output on a, on a model?

I mean, yes. I think that's probably reductive of, of like its full potential, but generally that's, that's correct. I mean, yes, you can use structured outputs, but you have to do a bunch of crap basically to coordinate like feeding all of it, feeding it into the rest of the program.

Maybe you want to call a model differently or use XML here or use a different type of model or whatever it might be, um, to, to do that. So absolutely, yeah. I'm not saying this is the only way, obviously, to kind of create these applications or that you shouldn't use pedantic or shouldn't use structured outputs.

You absolutely should. Um, it's just a way that once you kind of wrap your head around the, the primitives that DSPy gives you, you can start to very quickly build these types of arguably, uh, I mean, these are like prototypesright now, but like if you want to take this to the next level to production scale, you have all the pieces in front of you to be able to do that.

Um, any other questions? I probably got about five minutes left. Go ahead.

Can you talk about your experience using prompt optimization DSPy and just JEP as a primary?

Yeah. Yeah. So JEP, and actually I'll pull up, uh, I, I, I ran oneright before this. Um, this uses a, a different algorithm called MIPRO, but basically, um, the optimizers, as long as you have well-structured data, so for the machine learning folks in the room, which is probably everybody, obviously the quality of your data, of your data is very important.

Um, you don't need thousands and thousands of examples necessarily, but as long as you have enough, maybe 10 to a hundred of inputs and outputs, and if you're constructing your metrics in a way that is relatively intuitive and, and that, you know, accurately describes what you're trying to achieve, the improvement can be pretty significant.

Um, and so that time entry corrector thing that I mentioned before, uh, you can see the output of here. It's kind of iterating through. It's measuring the output metrics for each of these. And then you can see all the way at the bottom once it goes through all of its optimization stuff, you can see the actual performance on, um, the basic versus the optimized model.

In this case, it went from 86 to 89. And then interestingly, this is still in development, this one in particular, but you can break it down by metrics. So you can see where the model's optimizing better, performing better across certain metrics.

And this can be really telling as to whether you need to tweak your metric, maybe you need to decompose your metric, maybe there's other areas within your data set or the, the basically the structure of your program that you can improve.

Um, but it's a really nice way to understand what's going on under the, under the hood. And if, if you don't care about some of these and the optimizer isn't doing as well on them, maybe you can, maybe you can throw them out too.

So it's, it's a very kind of flexible system, flexible way of kind of doing all that.

Yeah. What's the output of the optimization? Like what do you get out of it and then how to use that?

Yeah.

Whatever it is.

Yeah. Yeah. So the output of the optimizers is basically just another, um, it's almost like a compiled object, if you will.

Okay.

So DSPy allows you to save and load programs as well. So the output of the optimizer is basically just a module that you can then serialize and save off somewhere, or you can call it later, uh, as you would any other module.

And it's just manipulating the phrasing of the prompts or like what does it actually like, you know, what's its solutions basically?

Yeah. Yeah. Under the hood, it's literally just iterating on the actual prompt itself. Maybe it's adding additional instructions. It's saying, well, I keep failing on this particular thing, like not capitalizing the names correctly. I need to add in my upfront criteria in the prompt an instruction to the model to say, you must capitalize names properly.

And Chris, uh, who I mentioned before has a really good way of putting this, and I'm going to butcher it now, but like the optimizer is basically finding latent requirements that you might not have specified initially upfront, but based off of the data, it's kind of like a poor man's deep learning, I guess, but like it's learning from the data.

It's learning what it's doing well, what, what it's doing not so well, and it's dynamically constructing a prompt that improves the performance based off of your metrics.

And is that like LLM guided? Like is it like.

It's all, yeah.

Thinking about like capitalization?

Yeah. Yeah. Question being, is it all LLM guided? Yes. It's particularly for JEPA, it's using LLMs to improve LLMs performance. So it's using the LLM to dynamically construct new prompts, which are then fed into the system, measured, and then it kind of iterates.

So it's using AI to build AI, if you will.

Thank you.

Yeah.

Well, question. Why is the solution object not just the optimized prompt?

Why is the solution object not what?

Not just the optimized prompt? Why are you saying it's like.

Oh, it absolutely is. You can get it under the hood. I mean, you can, the question was, why don't you just get the optimized prompt? You can. Absolutely. Um.

What else is there besides the optimized prompt?

The, the, so what else is there other than the prompt? The DSPy object itself. So the module, the way things, um, well, we can probably look at one if we have time. Um.

If I could see a dump of what gets, you know, what is the optimized state? That would be interesting.

Yeah. Yeah. Sure. Let me see if I can find one quick. Um, but fundamentally at the end of the day, yes, you get an optimized prompt, a string that you can dump somewhere if you, if you want to.

Um, actually, um.

There's a lot of pieces to the signature,right? So it's like how you describe your fields.

Yes.

In the docstring.

This is a perfect segue, and I'll, I'll concluderight after this. I was playing around something. I was, I was playing around this thing called DSPy Hub that I kind of created to create a repository of optimized programs. So basically like if you're an expert in whatever, you optimize an LLM against this data set or have a great classifier for city infrastructure images or whatever, kind of like a Hugging Face, you can download something that has been pre-optimized.

And then what I have here, this is the actual loaded program. This would be the output of the optimized process or it, it is. And then I can call it as I would any, anything else. And so you can see here, this is the output.

And I used the optimized program that I downloaded from, from this hub. And if we inspect maybe the loaded program, you can see under the hood, it's a predict object with a string signature of time and reasoning. Here's the optimized prompt ultimately.

This is the output of the optimization process, this long string here. Um, and then the various, uh, specifications and definitions of the inputs and outputs.

Have you found specific uses of those? Like to his question, like what is it, what can you do with that?

It's up to your, it's up to your use case. So if I, if I have a, so a document classifier might be a good example. If in my business I come across whatever documents of a certain type, I might optimize a classifier against those, and then I can use that somewhere else on a different project or something like that.

So out of a hundred thousand documents, I want to find only the pages that have an invoice on it as an example. Now, sure, a hundred percent, you can use a typical ML classifier to do that. That's great.

This is just an example, but you can also theoretically train or optimize a model to do that type of classification or some type of generation of text or what have you, which then you have the optimized state of, which then lives in your data processing pipeline, you know, and you can use it for other types of purposes or give it to other teams or whatever it might be.

So it's just up to your particular use case. Um, something like this, like Hub, who maybe it's not useful because each individual's use case is so hyper-specific, I don't really know, but, um, yeah, you can do with it kind of whatever you want.

Probably last question. Yeah.

Is generally, you know, like using DSPY something where people kind of do replays just to optimize their prompt or is there a way to sort of do it a little bit more real time given delayed? Like what I mean by delayed is, okay, ChatGPT gives you your answer and you can thumbs up or thumbs down.

Mm-hmm.

You know, that thumbs up comes, you know, like 10 minutes later, 30 minutes later, a day later,right?

So is the question more about like continuous learning-ish? Like how would you do that here?

Do like more continuous learning? You can be the judge. So you can. Well, but how are you feeding back the late metrics to optimize it? Why, why does it need to be delayed? Because, you know, usually maybe the feedback is from the user,right?

Like it could be delayed.

Oh, I mean, you're saying not from yourself.

Yeah. Well, then.

Yeah. That'sright. If you it would basically be added to the data set and then you would use the latest optimize and just keep, keep optimizing off of that.

Wrong too. Data set.

That'sright.

You will collect the outputs of your optimization and feed it back and the, the loop repeats. Yeah. But at that point you're kind of doing offline optimization,right?

Yeah. Yes.

But I'm, I'm asking can you do this online where with the late metric feedback?

If you're a good, if you're a good enough engineer, you probably do it, but I'm not, I'm not recommending replacing ML models with like optimized DSPy programs for particular use cases. Maybe like classification's a terrible example. I recognize that.

But for other, other areas, other in theory, yes, you know, you could do something like that. Yes.

But for, for particular LLM tasks, I'm sure we all have interesting ones. If you have something that is relatively well-defined where you have known inputs and outputs, it might be a candidate for something worth optimizing. If nothing else, to transfer it to a smaller model to preserve the level of performance at a lower cost.

That's really one of the biggest benefits I see.

Allright. Last, last question.

I've heard that, uh, DSPy is can be kind of expensive 'cause you're doing all of these LLM calls.

Mm-hmm.

Um, so I was curious your experience with that and maybe relatedly like if you have any experience with like large contexts in your optimization data set, if you reckon with ways of shrinking those.

Yeah. So the question was, do can DSPy be expensive? And then for large contexts, kind of how have you seen that? How have you managed that? The expensive part is totally up to you. If you call a function a million times asynchronously, you're going to generate a lot of cost.

I don't think DSPy necessarily, maybe it makes it easier to call things, but it, it's not inherently expensive. It might, to your point, add more content to the prompt. Like sure, the signature is a string, but the actual text that's sent to the model is much longer than that.

That's totally true. I wouldn't say that it's a large cost driver. I mean, it, again, it's ultimately, it's more, more of a programming paradigm. So you can write your compressed adapter if you want that like, you know, reduces the amount that's sent to the, to, uh, to the model.

Um, in terms of large context, I, it's kind of the same answer, I think, in terms of if you're worried about that, maybe you have some additional logic either in the program itself or in an adapter or part of the module that keeps track of that.

Maybe you do some like context compression or something like that. There's some really good talks about that past few days, obviously. I have a feeling that that will kind of go away at some point. What either context windows get bigger or context management is abstracted away somehow.

I don't really have an answer. Just that's more of an intuition. Um, but DSPy, again, kind of gives you the tools, the primitives for you to do that should you choose, um, and kind of track that state, track that management over time.

So I think that's it. We're going to get kicked out soon. So thanks so much for your time. Really appreciate it.