Will Agent evaluation via MCP Stabilize Agent Networks? - Ari Heljakka

Hi, this is Ari Heljakka, the CEO of Root Signals, the platform for measuring, controlling, and optimizing LLM-based agents, chatbots, and workflows, both for AI product companies and enterprises. And today I'm trying to answer the question: whether agent evaluation by MCP or model-context protocol could stabilize agents and agent networks and swarms.

So most of us are already thinking along the lines that if you only had a stable agent swarm, you could, in principle, solve any kinds of knowledge work problems. And the only limitation here is that those swarms tend not to be stable when you try to solve increasingly complex problems.

And the reasons relate to both our inability to observe them perfectly, and the dynamic environments, difficulties of testing them comprehensively beforehand, etc. And also, it is often not clear whether the agents are making progress consistently toward the goal in the cases where they just don't achieve it on, basically, one shot.

So how do we start solving this problem in a more robust manner? The general answer to the question is going to be evaluations, or evals, but not in a way that you would think. So firstly, many people tend to oversimplify the question of evaluations.

And if you just go out there and you basically just take some eval stack and you add it, then the outcome is like this: that alone will probably not take you anywhere. Just the same

kind of outcome if you don't actually systematically use evaluations in a way that not only improves those agents and workflows, but also those evaluation stacks themselves continuously develop and

grow more aligned with your actual business requirements. So the reason this is difficult is fundamentally because if you want to evaluate all aspects of agent behaviors and internal representations, you have a very complex landscape, as shown here. On the links below the video or attached to the video, you can also find my previous talk on this same topic where I went through these in more detail.

But in short, you have both evaluations for the left-hand side, which is, in a way, the agent represents reality: how it models the reality and discusses the reality with the user, and how it's grounded in reality in terms of what is true.

And on the other hand, on theright-hand side, you have behavioral aspects, such as whether the agent actually inferred theright goals, whether it's making progress toward those goals, whether it selects theright tools on the path to get there, etc., etc.

So in principle, you would want to evaluate all these things, but at least getting started with some would probably take us somewhere if we are just able to do that consistently.

So first, to get started, as an example, we need to have some kind of a clear framework for actually setting up the evaluators. So for example, in this case, we are looking at a reservation agent for a hotel.

And for this purpose, we would have evaluators such as the ones mentioned here, like policy adherence to that specific hotel's reservation policy, accuracy of the outputs of the agent to the user, and then, of course, the appropriate behavior overall.

So you need to set up these evaluators somewhere. This has been set up in Root Signals, but wherever you do, make sure that you have both good visibility to how you can create large stacks of evaluators and how you can maintain them and improve them systematically over time when you have to run dozens of them all the time.

So what we want to achieve is a stabilization loop where the agent attempts a certain task, the output of the task will get evaluated by the evaluation engine, and then feedback in the form of a numeric score and also an explanation of what went wrong comes back to the agent, or the explanation of what went well as well.

So using that information, the agent can then improve its own performance. So this needs to come from the eval engine. And in order to attach these agents, the new, latest, greatest method is going to be the model-context protocol, or the MCP.

So let's look at some practical examples next. And we will start with something very simple.

So for this first experiment, we will not even have code. We will just be looking at text. So if the agent, or whatever workflow you're running, would output this sort of sentence, like: "Root Signals MCP service awesome, lets you access evals nicely."

And now you want to measure and also improve this specific text that might be coming out of the agent. So for this purpose, you firstly need to find which evals you want to run on it. So we can actually ask the cursor UI, through MCP, to firstly list the charges that I have available.

You can also list evaluators directly. Charges would be collections of evaluators in this specific terminology. So I've already attached the

free Root Signals MCP server that you can get from GitHub, and I just added it on cursor. And what this MCP server lets you do is gives you access to these evaluators and these charges that I have created for myself earlier, and then some universal charges, universal evaluators, that can be used anywhere.

So importantly, the first one here that I happened to be interested in in this case was this marketing message quality charge. So it was just found by what I wrote here. So I can then very freely, or loosely, just ask cursor to measure and improve this marketing statement on the left by just saying that optimize the message by using the marketing message optimization charge.

And at this point, cursor is going again through the MCP interface back to the charge and evaluator stack. It will find again that specific charge I was interested in. It will score this message on the left, and then it will try to figure out how to improve that score and score it again if everything goes nicely.

So let's see if it's able to actually do that. And this is actually now real time. Okay, it's finding various issues with it, and then it continues the process of trying to improve. Let's see if we can get the scores.

Okay, it's already suggesting an improved version of that based on the scores. But let's see if we also see the scores from here. Sometimes they are shown immediately. Sometimes they are shown later. So apparently, this time it didn't show the original scores for us, but now it's running the final evaluation to get us with the final scores.

If you don't see the scores, you can actually ask it to show them. Now it's showing them. So okay, so persuasiveness, quality of writing, engagingness, numbers shown here. You can see numbers that it started out with. You can see final numbers.

Now it's only showing the final numbers between 0 and 1. So this is basically the same thing that will happen inside your agent, but now we just call it manually to make it easy for us to see. For the agent, you can either exactly explain which evaluators and which charges to run, or you can just let it pick them itself.

So next, we'll look at an actual agent example.

And now I want to be really clear. We are no longer talking to MCP through cursor, so I'm removing this from our field of view. So I have now an actual, very simple hotel reservation agent here running on Pydantic AI.

And this just happens to be implemented or actually viewed through cursor. And in fact, I'm going to run this separately on the console. And what happens in this reservation agent is that the user is going to ask something a little bit dangerous.

So this is a reservation agent for Shire Hotel. And the Shire Hotel happens to be next to the ACME Hotel. So in no circumstances is the Shire Hotel reservation system supposed to be recommending or preferably saying anything about the ACME Hotel.

So the user is now asking about the Shire specifically, but actually hinting that it would also be interested in the hotel next to it. A bit maybe artificial example, but just showing this very precisely what is the problem.

So let's see what happens when you run this without the MCP.

So currently, the MCP is off, and let's see what the agent returns. Okay, it will politely say that there will be rooms available a bit later, but it's also saying that the ACME Hotel is also a great option.

So this is what they don't want to see. So let's go back. We will turn on the MCP server here and see if that makes a difference. It just set this to true, and it will then add it will actually use this

MCP server here for evaluation. So let's go back to then running this same command again. And while it's running, it's calling the MCP server here. So you will see that there will be a lot of activity happening where evaluators are being listed, and there

will be calls back and forth through the evaluator lists and running many of them and some of them several times, even if we want to make maximum use of this.

Allright. So as you can see now, the agent came back, and now it's not talking about the ACME Hotel again. And we can see that it was invoking this Hotel Shire booking policy evaluator, even though we didn't even ask it specifically to do that.

So it can, in most circumstances, pick relevant ones from the list, but you can also enforce specific ones if you want to be sure. So this seems to have worked. And then let's go back to the big picture.

So what we have now seen is agent being attached to the evaluation engine through the MCP and getting the feedback and improving its own behavior. And you can see how this sort of approach would scale to more complex examples.

So then

to summarize on how to approach this, you should first start by making sure that your evaluation library, evaluation platform, whatever you're using, is sufficiently powerful to support all this, both in terms of diversity of evaluators that you can create and also their lifecycle maintenance and optimization on both the letting you both optimize the agent itself, but also you need to be able to optimize the evaluators because you will be running a lot of them, and they will cost you money.

Also, it's a good idea to start with running the MCP manually offline, like we did with the marketing message example, so that you first understand roughly what is happening, and then you're able to get better and better transparency to the ways in which those evaluators can work and not work.

And finally, then you will be attaching the evaluators through the MCP to the agents. And this promises to make everything more controllable, more transparent, and also dynamically self-improving and self-correcting. So this is all. The Root Signals MCP server is available for free.

Others will surely be appearing to allow all kinds of evaluation platforms and frameworks to be implemented as part of your agent stack. So I thank you for your interest, and happy hacking.