# Judge the Judge: Building LLM Evaluators That Actually Work with GEPA — Mahmoud Mabrouk, Agenta AI AI Engineer · 2026-04-10 Mahmoud Mabrouk, co-founder of Agenta AI, demonstrates how to build calibrated LLM-as-a-judge evaluators using the GEPA prompt optimization algorithm, arguing that miscalibrated evals are worse than none. He walks through a practical workflow for a customer support agent using the TaoBench airline dataset, covering metric design, data annotation, and GEPA-based optimization. The seed judge achieved 61% accuracy; after optimization, accuracy rose to 74% with reduced bias, though the judge still struggled to fully learn the complex policy. Mabrouk shares key lessons: start with a seed prompt biased toward compliance, use larger models for refinement, overfit to training data first, and beware of high token costs. ## Questions this episode answers ### Why are uncalibrated LLM-as-a-judge evaluations counterproductive? Mahmoud Mabrouk explains that miscalibrated LLM judges give false confidence while providing useless signal, because they don't correlate with human annotations. This creates a bottleneck in the development loop: you can iterate fast, but if the evaluation is wrong, you won't improve the actual application. He emphasizes that calibration with human ground truth is essential to make iteration speed meaningful. [1:28](https://aie.addtry.com/a79391a4-ea87-445d-9ca1-2eefd72ee843?t=88000) ### How does GEPA optimize LLM-as-a-judge prompts? GEPA is a genetic algorithm that starts with a seed prompt and iteratively samples new candidates through mutation (reflecting on failures) and merging (combining two prompts). It evaluates candidates on mini-batches, then selects a Pareto frontier of best performers per task to maintain diversity. This process repeats until a compute budget is exhausted, aiming for one prompt that solves all training cases. Mabrouk used the 'optimize anything' API for this. [14:50](https://aie.addtry.com/a79391a4-ea87-445d-9ca1-2eefd72ee843?t=890000) ### What are the key practical challenges when using GEPA to build calibrated LLM judges? Mabrouk found that naive application fails; success requires careful seed prompt design (starting with assumption of compliance), customizing the refinement template to include ground truth annotations, using larger models like GPT-4 Mini (smaller models failed), and iterating to overfit on training data before scaling. Costs can be high—small experiments cost $200–$300. Debugging by stopping early and examining candidates was crucial. [34:00](https://aie.addtry.com/a79391a4-ea87-445d-9ca1-2eefd72ee843?t=2040000) ## Key moments - **[0:00] Intro** - [0:01] Miscalibrated evals give false confidence: a hallucination judge with a trivial prompt fails in production - [0:52] Mahmoud Mabrouk: 'If it could, then your app would have worked from day 1.' - **[1:11] Motivation** - [1:11] Calibrated LLM judges use human annotation and prompt optimization with GEPA - [1:55] An LLM judge correlated with human annotations speeds up the AI development loop - [2:35] Calibrated LLM judges enable rapid detection of production improvements and data shifts - [3:16] The data flywheel—optimizing harnesses, observing traces, and adding evals—is the holy grail of AI engineering - **[4:06] Speaker** - **[4:53] Workshop Plan** - **[5:48] Dataset** - [5:48] TaoBench from Sierra provides 599 annotated traces of an airline customer support agent - [7:39] The airline agent dataset is 62% compliant and 38% non-compliant across 599 traces - **[8:14] Metrics** - [8:32] Metrics for LLM judges must come from the business use case, defined by subject matter experts - [9:07] Hamel’s error analysis workflow: annotate traces, cluster error types, and build separate judges per error - [10:14] Breaking evals into specific binary judges per error type yields better calibration than a single judge - [10:57] Binary judgments with reasoning produce more reliable LLM judges than numeric 1-to-5 scales - [11:34] Human reasoning in annotations is essential for the optimization algorithm to learn policy adherence - **[11:49] Annotation** - [13:12] Data curation and annotation quality are the hardest part of building a calibrated LLM judge - **[14:32] GEPA Optimization** - [14:32] GEPA is a genetic-style algorithm that optimizes prompts by sampling, evaluating, and filtering via a Pareto frontier - [15:52] GEPA’s prompt mutation strategy uses LLM reflection on failures to generate improved judge prompts - [16:54] GEPA’s merge strategy combines two candidate prompts to consolidate their guidelines into one - [17:45] GEPA’s Pareto frontier selection keeps candidates that are best per task, not just highest average score - [19:44] The GEPA library’s Optimize Anything API extends prompt optimization to any algorithm configuration - **[22:13] Live Demo** - [24:04] The seed LLM judge biased toward compliance scores 61% accuracy, predicting compliant 98% of the time - [26:29] Starting a seed judge with a bias toward compliance prevents the LLM from injecting its own random biases - [28:44] GEPA optimization lifts judge accuracy from 69% to 74% and boosts non-compliance recall from nearly zero - [29:01] Pareto frontier accuracy hits 100%, yet merging all task-specific candidates into a single judge prompt remains difficult - [30:13] Smaller models such as GPT-4.0 completely fail as LLM judges or refiners for complex policy-adherence tasks - [31:39] A seed prompt without pre-loaded policy outperforms one that includes the agent policy, avoiding local minima - **[32:17] Results** - [33:27] Optimizing an LLM judge with GEPA costs $200–$300 in API tokens for a small dataset of long trajectories - [34:06] Overfitting on training data and visualizing traces is essential before scaling GEPA optimization experiments - **[34:22] Lessons Learned** - **[40:34] Outro** ## Speakers - **Mahmoud Mabrouk** (guest) ## Topics LLM-as-Judge ## Mentioned Sierra (company), Agenta (product), DeepSeek (product), Despy (product), GEPA (product), GPT-4 (product), Gemini (product), Grok (product), Hamel Dev (product), Mini (product), Nano (product), TaoBench (product) ## Transcript ### Intro **Mahmoud Mabrouk** [0:01] Hello everyone, and welcome to my talk slash workshop, "Judge the Judge," and today we're going to talk about LLM as a judge. I'm quite sure you know this scenario: you have an agent in production, and someone from the team says, "We need to monitor the reliability." So you go to one of the libraries and maybe use the hallucination LLM as a judge. You put it in production within your observability platform, and things look fine. But customers are actually saying that the agent is not working, and you look at the traces: it's not working. You look now under the hood about this hallucination LLM as a judge, and you'll find a prompt not very far from this one. You'll be given an LLM output, write whether it's in a hallucination, make no mistakes. Now, obviously, how the hell would the agent know whether it's a hallucination? If it could, then your app would have worked from day 1. So today we're going to talk about how can we build calibrated LLMs that judge as a judge that works. Calibrated mean calibrated with human annotation, and the way we're going to calibrate them is by using optimization, or prompt optimization. Specifically, we're going to use GEPA, a quite good algorithm for optimizing prompts. Now, why do we want this? Why do we want calibrated LLM as a judges, or good LLM as a judges? First thing is for our fla- our offline evaluations. ### Motivation **Mahmoud Mabrouk** [1:28] As you know, like, usually to create a good agent or a good prompt, the way you do is you try to experiment with a prompt, then run your evals, see if it improves things or not. If it does, good. If it does not, you go back and you improve it a little bit, prove the harness, the prompt, and do it again and again. And the speed in which you move to production or add features is actually the speed into which you can complete this loop. And the bottleneck in this loop is actually the evaluation. How fast can you evaluate? Obviously, the slowest possible evaluation is having human annotator really look at your whole test set and annotate it manually. Um, the quality is quite good, but then each iteration will take a lot. Um, you can have faster ones by using an LLM as a judge, but then if that LLM as a judge does not, um, correlate with a human annotation, then you'll end up with useless signal, and although this loop will move fast, it won't go anywhere. So having calibrated LLM as a judge with a similar quality, let's say, as human annotator, will make your development much faster. The second thing is basically the online eval, like our example from the beginning. If you have an online eval and you want to see, basically, in production if things are improving or not improving. Same thing, if you have LLM as a judge that are calibrated with your business goals, then you can quickly see whether changes that you've made are improving, not improvement, whether, um, there is some change in the distribution of the data, how people are interacting, uh, with with your agent or model, and basically react rapidly. And finally, and I would call this is the holy grail of AI engineering, is really to build this data fla- flywheel, where you optimize your harness, observe some traces, and then add new evals based on these traces, the edge cases, and do it again and again. And again, here, if if you have a way to kind of add new evaluations quickly, obviously automatic evaluations, um, from the traces, from kind of the annotations and data, you can go through this loop faster and faster to the moment, or to the point that you can think of it as an automatic loop,right? Because you can optimize the harnesses with optimization techniques like GEPA, which we're going to use here, but the same thing you can do it for the evals, and basically over time your application will improve just with, uh, new observations. So today we're going to build and optimize these LLM as judges that are calibrated with human annotations. Uh, but before going there, a small intro about myself. My name is Mahmoud, I'm the co-founder and CEO of Agenta. Uh, Agenta is an open-source LLMOps platform, uh, basically providing you all the tools from observability, prompt management, evaluation, covering the whole life cycle of building reliable agents. ### Speaker **Mahmoud Mabrouk** [4:34] Um, my experience is in machine learning. I have more than 15 years' experience in that. In a previous life I was in academia, uh, worked on machine learning applied to computational biology, protein structure prediction, andright now we're working a lot on these sampling and auto-optimization workflows. So, uh, if you're interested in that, please reach out. We'd love to have a conversation and show you also what we're building. So what's the plan for this talk? Basically we're going to work on a practical use case. ### Workshop Plan **Mahmoud Mabrouk** [5:04] It's a customer support agent that we want to evaluate, and we are going to build an LLM as a judge that is calibrated with the annotations, basically human annotations, for that customer support agent. The plan would be really to go over the whole process of building this, starting with how we design the metrics, how to think about the data curation, the labeling. But the main focus would be really about, um, the part about optimizing the LLM as a judge using GEPA, and obviously then validating the results. All the code and the data used in this, uh, will be found in GitHub, and you can find them in the links in this video and the last slide. So let's start with the data set. We're going to use TaoBench. TaoBench is a benchmark and a large data set built by Sierra, uh, customer support, um, Scaleup, I think. And they have, like, multiple benchmark for real-world scenarios for customer support agents. ### Dataset **Mahmoud Mabrouk** [6:08] Uh, one of them is the airline agent, which we are going to use, um, in this example. And basically what they have is an airline customer support agent that has access to multiple tools to manage reservations, access flight information, access user information, and has a quite complex policy to behold to, like, when to change a reservation, uh, when to provide information, and so on and so forth. Just like a customer support agent, a human one. And the data that we have is, is the agent itself, but most impor- most importantly, um, 599 conversation traces that are generated with annotations. Uh, now the format, the original format of the annotations is like in the format of assertion, but, uh, I pre-processed or I post-processed the data so that we have for each trace an annotation like a human annotation, uh, where it says, for example, like in this case, you have the conversation that you see here, and then you have an annotation that the agent is not compliant because it improved approved the cancellation without verifying that the reservation met the airline cancellation rule. So basically the evaluation failed, and the reason is because the agent, uh, um, canceled the reservation without verifying something. So basically, uh, it did not behold to the policy. And the data is, is more or less, uh, kind of, um, not very skewed. It's 62% compliant, 38% compliant, and it's generated with multiple models and trials. Overall the data is, is kind of, the problem is quite complex because the policy is quite complex. The data has caveats. Honestly, it's not very clean, um, due to the reasons how it's also generated. But for our use case, I think it's very, um, interesting use case to test, uh, GEPA and to kind of demo how it would work, uh, in a real, uh, test case. The workflow that we have is four steps. First thing is designing the metrics. That is deciding what will the LLM as a judge measure, uh, what are the different axes that it would look to. The second thing is annotating the data, uh, and then we will optimize the judge and validate the results. ### Metrics **Mahmoud Mabrouk** [8:32] Now, the most important thing to take here is that the metrics need to come from, uh, the use case itself. It does not make sense to have general metrics like hallucination, uh, when you're evaluating your AI agent. It really depends on kind of the business use case. And the best person, the best people to, to, um, to determine these metrics are the subject matter experts. For example, in the case of a customer support agent, you need to have, um, a subject matter expert look at the conversation, provide the feedback. And I think the, the workflow to do that, the best one that described it is Hamel, uh, from Hamel Dev. Um, I'm going to share his, uh, blog and, uh, and the YouTube video, and he really described this idea of error analysis very well, but, but I'm going to go over it very quickly, and also the annotation workflow very quickly. So the idea is that you, you provide your subject matter expert with all these traces of, uh, the trajectories of the conversation, and they would annotate them first by commenting what did work or what did not work, but then, uh, kind of slowly trying to cluster, create clusters of the error types, like when it's failing, why is it failing. And, uh, here I'm showing in Agenta how it's done, and basically here I have, like, these I, I discovered kind of these four error types while going through these, uh, traces. So there is sometimes issues with policy adherence, uh, sometimes issue with the response style, uh, information delivery. Basically the agent is not informing the customer that they made a change or something like that. And finally, some tools has not been colli, uh, called correctly. And the idea is that we're going to take these four error types, and then we're going to build four LLM as judges for these. So it does not make sense to have one LLM as a judge, which is success, um, uh, basically try to evaluate all of these. It will make it too complex, and it's very hard to learn. And you will see a little bit later that even with a simple, like, when we're going to simplify it, it's still hard to learn a calibrated LLM as a judge or optimize a calibrated LLM as a judge. So it makes a lot of sense to make things, uh, very specific in these metrics, uh, that we want to evaluate. And the second thing is to move away from one to five scores or like percentages, and instead have, like, really binary a solution, like whether it adhered to policy or not, um, with obviously some reasoning. Um, and the reason it's, again, it's already quite hard to calibrate an LLM as a judge with a true false, like binary classification, uh, adding another layer and saying, okay, it should be a number between one and five. It's, it's hard. It's even hard for a human annotator, like, to have two human annot- annotator agree on the same score. So the moment that we have defined these different metrics, uh, start the point of annotation. Um, here again I'm using, uh, Agenta. Basically, um, uh, you would take your traces, create an annotation queue, and, and kind of specify for your annotator, like, uh, the name of the, um, the feedback or the evaluator, policy adherence here, and then providing, like, they should provide each one with whether it's adhered to the policy, whether it does not, and, uh, provide the reasoning. ### Annotation **Mahmoud Mabrouk** [12:12] And the reasoning here is very important, uh, because without the reasoning we will see, uh, the optimization algorithm will need to discover itself, like why it failed. And it's going to be very hard, like in, uh, unless it's a very specific, uh, kind of feedback, for example, tool, um, failure where it can infer things. It's very hard to see, for example, from a conversation why it did not adhere to policy without someone providing information in the beginning about it. So having that reasoning is very important later for the LLM as a judge to learn. And again, this reasoning, uh, as I've shown previously with the annotation, uh, it, it kind of describe why, for example, the agent here is not compliant, uh, because, uh, it approved the cancellation without before verifying, um, that the reservation met the cancellation rules. So now that we have, so now that we have the annotations, we can go to the optimization. But before going there, uh, I want to add a small note. Although we went very quickly through the first step and the second step, these are actually the hardest part of the problem. Like in reality, uh, as every data scientist know, getting your data, um, is the hardest thing. And you need to make sure to look at the data, look at the annotation, make sure that the distribution is, is good, uh, that, uh, the annotation and the information within the data is enough for the algorithm to learn a representation of the LLM as a judge that is meaningful. Uh, in our case here, the data is not that good. Uh, the number of traces is small, the problem is quite complex, uh, they're not very well distributed because of the reason they have been generated. Uh, also the annotation is actually, uh, kind of AI generated based on the assertion and the original data. You can see a little bit more how it's done in, um, uh, in the repository. Um, so it makes things, uh, quite complex for this problem. Um, but it's still quite a good demonstration of how it would, uh, work. ### GEPA Optimization **Mahmoud Mabrouk** [14:32] So now that we have the annotated data, we can start with the optimization. And for the optimization we are going to use the GEPA algorithm. So I'm going to explain in the beginning how the GEPA algorithm works, and then we can jump to the Jupyter Notebook and start optimizing based on the annotated data. It's very important to understand how the algorithm works because you will see that practically you need to play around a little bit with the parameters and to get it to work. And it's very hard, obviously, to play around with the parameters if you don't understand what do they do. Uh, the algorithm is, um, similar to a genetic, uh, algorithm. So basically the idea is that you start with a seed prompt, and then at each iteration you try to sample new prompts, see which one works, and then, uh, basically select the new ones and, and kind of improve over time. Uh, that's kind of the general shape of, uh, the algorithm. And we're going to go, uh, and look at each step. So it's three steps. Basically you sample new candidates each time, evaluate them, see which one wo-work well. And then do some filtering using this kind of Pareto frontier I'm going to talk about, and then do it again and again. So let's see how it works. So the way, uh, it works is first you start with a seed candidate. Here in our case we're going to use kind of a very simple LLM as a judge, like e-evaluate whether this customer service agent violated policy, and start with assuming as the agent is compliant. And now in each iteration we are going to sample new candidates from the filtered candidates from the last iteration. Now here in the first one we have only one seed candidate, but in, in the next iteration we will have a larger bag of candidates. So GEPA has two strategies to sample new candidates. Uh, one is prompt mutation, and the other one is merging multiple candidates. For prompt mutation, which is what we're going to use in the beginning since we have only one candidate, the idea is that you would run the first LLM as a judge here, um, with the trajectory, and, uh, if it fails, uh, it's like this LLM as a judge will reflect and propose a new prompt. Basically there will be some kind of reflection, which means basically we're using the intelligence of the LLM to try to improve the prompt because it looks at the input, looks at the outputs, looks at the results, and try to infer how to make it better. The other strategy is the merge strategy, which we're going to use in the next iteration. And basically here it takes two prompts and then kind of put them together. And if you think about it with an LLM as a judge, usually you, you have like these guidelines, and basically probably what it's going to try to do is to take guidelines from prompt A, prompt B, and put them together. So now after we generated a lot of samples, uh, we need to select which one are good. And basically the way it works is that it would evaluate these new prompts against mini batches of the eval, so not everything. Um, and if they improve the performance, uh, compared to the starting point, uh, then we select them. And, uh, they will be added to our bag of prompts. And then starts the next iteration, which is the other, uh, innovation of this algorithm, which is the idea of the Pareto frontier. Basically the way we select which prompts or which candidates we're going to use as a seed for the new iteration is not that we select the ones that have the average best score, like that would be the trivial, um, uh, solution,right? Look at my, my prompt, see which one work most by looking at the average over everything, and then select these. Uh, instead what they do is that they, um, try to add diversity by trying to look at what are the best candidate per task,right? You have like a set of kind of tasks in your evaluation, like in our case set of trajectories. And you look for each of these trajectory which is the best candidate, and that is kind of the Pareto frontier. Um, and then you try to select from these. And basically what you do is you try to select a set at the end of the day that covers your whole test case. So basically for, for each test case there is at least one candidate that solves it. And obviously you see there that the idea is that you get like a good Pareto frontier, and then you start merging things, and at the end of the day you have this prompt that solves everything in your training. Now that you have like this filtered set of candidate, again we sample new candidates from these, uh, these ones using, uh, kind of the mutation and the merge strategy, and we keep doing these, um, until basically the compute, uh, budget, uh, finishes. Now there is a lot of libraries that implement this algorithm. I think the most known is Despy. It popularized the idea of optimizing prompts or harnesses. Uh, but now there is, uh, I think a new library by the authors of GEPA, an open source one called GEPA. And they have implemented, uh, in the last month, uh, a new interface called Optimize Anything, a new, uh, API, which is what we're going to use, and which can be used not only to optimize prompts but really to optimize, um, any, almost any algorithm using this same idea. It's quite powerful. Uh, let me show you how it works. So basically, uh, the API here is called Optimize Anything. You see this function. And what it takes is a seed candidate. The candidate is the configuration that you want to optimize. In our case that would be the LLM as a judge prompt,right? Uh, we can make it even a dict if we want. So it's kind of, for example, the LLM as a judge, uh, prompt plus temperature, let's say so. Or it could be a chain of prompts and so on and so forth. So it's not limited. Then we have, uh, the evaluator, which is basically the thing that will be used by GEPA to optimize. And the expectation from the evaluator is that, um, it would obviously run the system. In our case it would run the LLM as a judge, uh, parameterized by the candidate. Uh, and then it would, uh, log, uh, kind of diagnostic not only the output, uh, but also the error, the reasoning. And, and the idea you can add as much as you want, and you see, you will see this is how we're going to do it with, with our optimization for the LLM as a judge. Uh, but the idea is that, uh, if you remember here we used some kind of, uh, reflection and, and, uh, um, reasoning to improve our prompt. And that reasoning is something that we ourself will build, uh, through this evaluate. And then other than that there is ways in the configuration, for example, uh, to configure how many calls to do per iteration. Um, the objective basically providing context for, uh, the refinement prompt on how to, uh, improve, uh, and so on and so forth. But, but the, um, the core, uh, flow is actually quite simple. So now let's jump to the Jupyter Notebook and really look into how to do it, uh, step by step. So you'll find this Jupyter Notebook in the GitHub repository that you'll find in the links and in the last slide. ### Live Demo **Mahmoud Mabrouk** [22:33] So we start by, uh, installing the library. So it's .env, lite llm, and, uh, uh, GEPA. I'm not installing here GEPA because I did the optimization. It takes, uh, kind of quite a time, I think, uh, a couple of hours, uh, before. So I'm going to jump this step. Um, but obviously if you want to run it within the repository, within the Jupyter Notebook, you should also install it. So we install this and we kind of do our imports. We have kind of a couple of, um, of functions that, that I extracted outside. We're going to look at them later. And we start by loading the data. Uh, as I mentioned, so we're using the data from TaoBench. Uh, I just kind of, uh, preprocessed it, uh, processed it in the beginning to change the type of the assertion so that they look like the annotation I showed in the presentation. So I've already kind of split, uh, the data into a training and a validation, um, uh, data sets. Uh, so one which we'll use with the GEPA and the other one, the second one, to validate the result at the end. And the way I did the split is, uh, based on different tasks that are created in, um, uh, in the TaoBench. And if we look here is basically we have a, a training set with 480 traces, uh, with, with around two third, uh, that are compliant, and a validation set with 112 traces that are compliant. As I mentioned, the data here is not, not very, very nice because, uh, there are some redundancies. So there are sometimes the same task, uh, that is, um, being run with, with the same model multiple times. So there is a little bit of redundancies, but there are no redundancies between the training and the validation set. So we look here how the annotation, uh, looks like. And again, uh, basically we have an example of a compliant annotation and a non-compliant annotation. Basically a trajectory that, uh, that kind of adheres to the policy. That's the LLM as a judge that we want to learn. Another one that does not. And we can see here basically it describes like, okay, it's compliant because, uh, it correctly identified, uh, the basic economy reservation, while here it did not identify, uh, the user membership as regular. And, and this annotation is actually quite important for us, for the LLM as a judge to learn, uh, the policy. Especially in the case here, um, this policy adherence, it's kind of a very complex system, uh, that the LLM as a judge need to, uh, to learn. And without kind of information about, uh, what is compliant, like why is something, uh, correct or not correct, it would be, uh, quite impossible for the, uh, GEPA algorithm to, to reach, uh, a good LLM as a judge,right? I mean, it would be the same for a human,right? If you gave me all these trajectories, told me this is, uh, kind of correct, this is non-conform to policy, but you did not give me, uh, more information, it would be very hard for me to, um, to basically make a judgment, learn how to assess the policies. So having this information and the quality of the annotation, as I mentioned, the quality of the data is really paramount of being able to learn this. Uh, and again, obviously this is kind of bit of a complex, um, LLM as a judge to learn. So, uh, first thing we start is, is with a naive, uh, judge. And, uh, basically this is the seed judge, uh, we start with. Um, and, and it's, it's something that, that actually I engineered, engineered. I'm going to, uh, talk a little bit about it later with the learnings on like how exactly did we reach this. And basically you can see here is like it evaluates whether the customer service agent violated policy. And it tells that you should start by assuming that the agent is compliant and, uh, only change to non-compliant if there is a specific reason,right? I mean, here we're, we are starting with an LLM as a judge, which means that, um, like the seed judge, the initial judge should actually, uh, in my opinion, start by, by saying everything's allright,right? I mean, if I don't have any rules, uh, it should say everything is allright. I started in, in like in another example when I started as I created an LLM as a judge that says, okay, you should check whether the agent violated policy. And what you end up being is basically the LLM with its own biases trying to make that decision by itself. Says, okay, this is violates, this doesn't violate without having any information. So without telling it in the beginning that you should start assuming that it's compliant. I mean, if you don't have any reason to, to believe it's non-compliant, it should be compliant. Um, then, uh, basically you start with, with some kind of random, uh, LLM as a judge that would be very hard to fix later, uh, unless you end up with one prompt that discovered this thing,right? That it starts with a compliant. So the, uh, I discovered here that the initial seed is actually very important. In this case there might be simpler scenarios where you don't need this, but in this case it's kind of, uh, quite important. So if we take this and we run this on the validation set like this, uh, initial, uh, LLM as a judge, uh, we basically find 61% accuracy, but, but we look like the bias it's actually most of the time saying it's compliant, which is actually what we want,right? I mean, if that's, I would say saying it's compliant 98% of the time is actually the unbiased thing to do, uh, the logical place where to start. Um, so, uh, we run the metrics in the beginning,right? The accuracy 61%, 98% is saying like the recall of, of compliance with very low to recall for non-compliance. Um, and I think as I mentioned, this is quite allright. It's biased towards compliant. And I've had experiments in the beginning where it was kind of almost random, but then it doesn't learn at all. And we can look into why or where it goes wrong. And, uh, basically by looking, uh, the places where it does, does work and we see like it says compliant but it's not compliant because it doesn't know the policy,right? So we start here like the main code of kind of optimizing the judge with GEPA. And what you will see is that I have actually, um, wrote, um, uh, reflection template. So that prompt that GEPA uses to reflect and to improve, uh, or to sample new candidates. I did not use their default, but actually I wrote one. Uh, I tried in the beginning by using the default prompt in, in, um, in, uh, in GEPA, but the results were not, uh, as good as I expected. It was very hard for it to, uh, to learn. And what I tried to do is, is to provide, um, basically a bias and prior within the reflection, uh, template. So you see here, for example, I mentioned obviously that, uh, it's basically, uh, the judge is reading an airline customer service, um, and it needs to kind of decide which is the basics. But then you can look at, um, more information. For example, uh, that our annotation that this, uh, kind of reflection template sees includes also, uh, the judge verdict, the ground truth, like the annotation. Like this is an important information that the reflection template should look into and improve. Um, and, uh, basically I explain to, to, to the LLM how to do this. You can add rules, restructure existing one, uh, reward things for clarity. Um, and, and try to think about it that the, um, the reflection template should basically create some real policy rules,right? Uh, it should find theright policies and abilities. And I think adding that, uh, was very, um, important to kind of improve the quality. Uh, otherwise the default reflection template did not understand, uh, that, that it should kind of try to, uh, learn the, the policy more or less,right? With the LLM as a judge, uh, to some degree. Um, so that's the second thing, uh, kind of I changed and I mean, honestly I iterated only a little bit on it. And then we run the optimization. A run optimization is basically a wrapper around, uh, optimize anything and it just, uh, uh, kind of, uh, parameterize it since I ran like multiple experiments, uh, to find the one. And it's actually, uh, it would be part of the GitHub repository. It's actually something that you can play around with and start with when you're exploring, uh, the space of design, uh, and GEPA. And you can see here it basically tries, um, to build, um, the configuration based on the parameters like the quirks and then calls optimize anything, uh, with these. And for the, uh, kind of, uh, the make evaluator, it's basically, uh, it calls the LLM as a judge and all adds all the side information. So it doesn't only provide the trajectory, but also kind of the annotation, which is quite important. Um, and then we run this. Um, as I mentioned, it takes around an hour to run. Um, and you can see here the, uh, the basically, uh, the results. This is the optimized, uh, Rubik. And you can see compared to the default where we started, it's, it learned part of the policy criteria like the flight cancellation and refines, uh, flight modification, um, uh, how to communicate and so on and so forth. ### Results **Mahmoud Mabrouk** [32:46] Now if we look at the results, like with the kind of evaluate, uh, Rubik to evaluate it, we see that the accuracy increased from 69% to 74%. And, uh, we removed the bias,right? So what especially changes the recall for the non-compliant and the precision for the non-compliant, which was basically zero in the beginning. Now, uh, the LLM as a judge is, uh, has less bias. It's 64%. So previously it was 98%. And it's really learned parts of the policy. So looking at the results, as I mentioned, like for the validation set, we had, uh, quite a lot of improvement, like 40%. And, uh, for the training accuracy, it improved by nine points. And we can see that the pareto frontier, interestingly, accuracy is now 100%, meaning that for each task there is one candidate that we generated that solved it. Uh, the issue the algorithm faced is how to merge all these candidates and all the information to have one prompt that solves everything. Um, and it struggled to do this. So at the end of the day, we improved the LLM as a judge. We improved its accuracy, but it's very still quite far from kind of 95% accuracy or something that is really well aligned with the human judgment. Obviously here I didn't invest extreme amount of time in it. And as I mentioned in the beginning, the quality of the data is also, um, would be better, I would guess, in other cases. It's really a tricky, uh, example. Um, but, but nevertheless, it took actually quite a number of iteration. I think that's the biggest learning to, uh, to reach this LLM as a judge. It's not an algorithm that you just take and it works from, from day one, unless for kind of, uh, toy examples. ### Lessons Learned **Mahmoud Mabrouk** [34:39] And, uh, I wanted to show a little bit in the end, like what are the experiments I, I tried in the beginning, uh, that failed and how did I think about fixing them. Um, the first thing was actually, uh, using smaller or older model, like using, uh, GPT-4.0, uh, for both the refiner and the LLM as a judge. And that was a complete failure,right? Uh, smaller models really are very bad, at least in this example, to be either an LLM as a judge or a refiner. Um, uh, for the LLM as a judge providing all this policy, especially it has a lot of kind of complicated logic, it just failed and could not improve it. Uh, I tried other models. I tried Mini and, uh, Nano and Gemini and DeepSeek. And, and you see that the best kind of results with this kind of, uh, using, uh, Gemini for reflection and, uh, Grok for a judge. But I would say also using, uh, GPT-4 Mini for both is actually quite good and they result quite well. The other thing was actually trying how to try to debug it. And, and what I tried to do from the beginning is really to not start sampling, doing big experiments from the beginning, but trying first, uh, kind of a small iterations, looking at the reasoning LLM, looking at the candidate, how do they improve, how many improved, and understanding what's happening. And that actually what, uh, uh, what allowed me to kind of think about improving the refined prompt, um, and, and basically adding some, uh, prior there to, to kind of help solve it. Basically what I did is I stopped at the first iteration, found some example, and then looked a little bit kind of fine-tuned, um, and clothed code, uh, that refinement prompt to, uh, to basically allow it to improve, um, uh, the candidate. And, and as always in machine learning, like what you always try to do is to overfit for the training data, not trying to run the whole algorithm, but really trying to find a way so that it works. And I think what we saw with the pareto frontier reaching 100% is we almost overfit to the training data, but obviously for the merge there are like things I think we can do to improve. And the final thing was kind of this iteration on the seed prompt. There I actually, um, iterated on multiple seed prompts and, and there were two families. One which, as you have seen here, is, uh, uh, kind of was not, did not include any information about the agent prompt because we have access to that,right? And the agent prompt does have access to the policy. And one which had access to the policy. So basically it was this prompt that I've shown, uh, but then the policy of the agent like really copy-paste it. And interestingly, uh, the, uh, prompt that did not have access to the policy did, uh, better because, um, my hypothesis is that if you have access to the agent policy from the beginning, then it's very hard to fine-tune it. You're already stuck in a local minima that you can not improve on. But if you don't have access to the policy and, uh, yet obviously you have access to the annotations that describe in this case all the policy, uh, or a large part of the policy, then you, you are able a little bit, uh, to, um, to explore the space of the prompts much better. Um, and finally, the last point is beware of the cost. I mean, um, even these small experiments I've done, I think they cost like $200, $300, uh, in tokens, especially since, uh, the trajectories are long. So there is a lot of input tokens, uh, in this case. Uh, but, um, the models that are used, um, are actually quite expensive,right? Uh, GPT-4, I mean, I tried a little bit to play around with GPT-4, but that ate a lot of money. So I stopped the experiment. But even GPT-4 Mini is quite expensive to some degree. And then if you go, uh, Nano, at least from what I've seen, it doesn't work,right? Also, you, you go to kind of smaller model, cheaper model, it does not work. Usually what they say you should use, uh, kind of a bigger model for the refinement prompt and smaller model, uh, for the LLM as a judge. I think it makes sense, especially if you're running LLM as a judge, uh, against a lot of traces, like in the case of online evaluation, it's obvious that it's, it's a worthwhile investment of spending money on the optimization to lower the cost on the long term. And I think there is a lot of use cases where it worked. So again, first, uh, overfit to the training data, start with a small iteration, uh, visualize. So basically instrument the traces. I instrumented in using Agenta in this case. I tried to look at them, tried to look at the prompts that have been generated and understand how the algorithm is working before, uh, increasing the sampling. And in this case, I think we had, uh, around 200, 300 iterations per experiment. Um, in addition to that, there is actually a number of parameters like the batch size and so on that you need to, uh, fine-tune to get the algorithm to work. So that's it. Thanks a lot for watching. I hope that has been helpful and that you'll build good LLM as a judges that helps you to improve your applications. Uh, I'd love if you check out Agenta or open source LLMOps, uh, platform. Um, and you can follow me both on LinkedIn and X. And finally, if you're thinking and working about auto optimization, uh, about how to, uh, optimize prompts, uh, feel free to reach out or to write in the comments in YouTube. ### Outro **Mahmoud Mabrouk** [40:49] Have a great day. Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.