Harnesses in AI: A Deep Dive — Tejas Kumar, IBM

Hello, everybody. Everybody's head turned up. Hello, hi.

Hey.

How was lunch? Was it good?

Yep.

You didn't like it, huh? It's like British food. Anyway, hi. I'm Tejas. I'll be your first speaker this afternoon. Tejas, that's pronounced like contagious. Don't worry, I'm not. Hopefully my joy in AI is. And I've had the privilege of working at a number of different places over my career, in one form or the other.

It's just been an absolute joy to learn from the best. Today I'm an AI Developer Advocate at IBM, where we do things with AI, believe it or not. We train frontier models, we build harnesses. It's really a fun lab to work in.

But that's not what I'm here to talk to you about today. Today I'm here to talk to you about AI harnesses. AI harnesses. Before I move forward, I would love to just have a show of hands. How many of you are confident in your understanding of AI harnesses?

Like, you're like, I could present this on stage today. Look around. Look around. No, seriously, look around. That's why we're doing this talk. OK, this is my hope. I want you to, if I ask you this at the end of the talk,right, I want you to be like, oh, I get it now.

That's the whole point. I have literally nothing to gain from this other than I shared knowledge. OK? Because also, this term is kind of everywhere. You may have heard it used like 52,000 times today. And it means different things to different people.

Because like in the machine learning world, it means like a glorified test suite for machine learning models. But in the AI world, it means something different. And so today, we're going to understand this in detail. It's a deep dive, but it's 18 minutes long.

So let's move forward. I want to start by talking about why harness. Like, why do we use harnesses? And the reason for this is because we pay rent to companies that give us compute, give us inference, give us tokens in return.

Some of you maybe work for companies that have frontier models, like Anthropic or Google or whatever. And you maybe, what was the term? Token billionaires, yeah? I'm not that. I am maybe with Watson models. But the vast majority of us aren't token billionaires.

We pay rent. We literally $20 a month for Claude Pro. And then you get a context window that's limited, and you get like, you don't get the full hog, so to speak. And the model you rent is a black box.

Like, they could at any time, I'm not saying they do, but they could, if Opus is somehow not available, they could serve you Sonnet, even though it says Opus. You would never know,right? And so it's just a big, there's too many variables that we cannot control.

So why harness? Because the name of the game with harness is reliability. I really hope I'm not supposed to stand in front of this white line, and then I'm just nodding the camera. Anyway, whatever. It's reliability. It's making sure that the agents we build do what they do, period, irrespective of the black box model, irrespective of the thing we rent, and so on.

OK? Now that we understand why harness, let's talk about what a harness even is from first principles. Like, let's take it all the way back to harnesses that we know and understand. If you've ever climbed a mountain or something, or you've seen someone, this is a harness.

It's like mountain climbers literally will harness themselves to what? To a mountain, because it's stable. And they can't go off the rails, literally. They anchor themselves in something stable so that they can't drift too far. OK? That's what a harness is by design.

When you have any dog owners here, you have dogs? You walk your dog on a harness. OK? That's why. Because your dog doesn't go and bankrupt you with tokens. OK? That's what a harness is. But the problem is, if we think about what harness, there's really two types.

There's one from the machine learning world, which, as I mentioned, is kind of like a test suite and a test runner. You give a model some inputs, and you see the quality of the outputs. This is not ML Engineer Europe.

We're going to talk today about the agent harness that is common in AI engineering. OK? So what is an agent harness? An agent harness, and this is kind of the money shot here. The agent harness is, I'm not making money off this.

That's just an expression. The agent harness is everything around the model that gives it grounding in reality. It's literally the thing that ties it to a stable environment. OK? And agent, so Claude Code, for example, can be considered an agent harness.

And some of you will say, oh, no, it's a coding agent. Absolutely, it's a coding agent. But it's a harnessed coding agent. An agent harness has more or less the same typical suspects, moving parts. Number one, it's got a tool registry.

Almost every, like, so Claude Code, Cursor, Codex, they have tools to read from the file system, to write, to execute bash commands,right? They have a tool registry. They have a model. And some of them allow you to choose a model.

Some of them allow you to not. They have a model. They have primitives for managing context. Almost every harnessed agent runtime today will compact its own context,right? That's the job of the harness. Guardrails are another part of a harness.

For example, Mac steps. Anyone using Mac steps? Do not do more than five tool calls. That's a guardrail. And so if you do that, you just kill the run,right? An agent loop is another part of an agent harness, which is crazy.

This is what some people I've spoken to preparing this talk will say, wait, isn't a harness just the agent loop? No. It's the stuff around the agent loop. In fact, it could be a loop around your agent loop.

It could be an NM loop. And we'll look at that a little bit in some code. And then finally, there's a verify step. This is, for example, in a coding agent, after the work is done, a verify step would be, hey, let's run lint, let's run tests, let's make sure nothing broke,right?

So almost every, I'll use coding agents as an example, but you could have a harness for anything. And it's amazing, because it really grounds black box models in a stable environment that you control. OK? I'd like to show you a demo.

And what we're going to do together is we're going to build a harness, a baby's first harness. Let's call it a poor man's AI harness together, so we understand from first principles how this works. We're going to build a computer use agent that has a job.

The job is, go to Hacker News and upvote the first post. OK? It's a computer use, it's a browser use agent. We're going to use a really bad model intentionally. We're using GPT-3.5 Turbo, which is like 2023,right? But we're going to harness it so that it can actually do the job.

And we're going to save money. So let's, I've spoken too much. Let's just get into the demo. And so let's, welcome to my project. This is my project. Hello, everybody. This is the entry point. Can you see that?

Is it too, yeah? You want it bigger? Let's do bigger. OK. So this is not, actually, this room is too bright. Let's do light mode. It's not my nature, but sometimes, that's better, yeah? OK. So we have a model, and we're trying an old, I'll just, sorry.

We shouldn't have seen that. No, we'll try an old model. And this is the prompt. This is the task. This is literally my prompt. Upvote a story. I just described it. For the purpose of this demo, we will not change the prompt at all.

Because a lot of us think, hey, my agent is not doing what it's supposed to do, so I just need to prompt it harder,right? That's not always true. I need to change the system prompt. We're not going to touch any prompts here.

We're just going to build a harness, and the outcome will change. We log some things to the console, and then we start a browser session. OK, what's a browser session? It's literally just Playwright. Not Playwright MCP, like Playwright, Playwright, where this is just a class I made with an open method that launches Chromium and gets a context and makes a page.

And then navigate, we're just literally calling the Playwright functions, yeah? This is just traditional engineering. So we create a session. We open the session, meaning a browser window and a context. And then we create our tools, and we give that browser session to the tools.

And we create a context, and we give the task, meaning the prompt here, to the context. Now, create tools is literally what it sounds like. It's, here, there's just some types. And create tools is a function that takes a browser session and gives you tools.

And these tools are not, I didn't invent this. This is from OpenAI's SDK. OK? So you have the name, the description, parameters, and execute, the way you actually call the tool in your runtime. And there's just tools for, I made this.

It's very easy. So that's my tools. And then create context. You may think, whoa, context engineering. Absolutely not. This is my context. There's nothing here. It's just a system prompt. Literally, the most basic system prompt and the user's task.

This is basic, basic. And then we have run loop, which is just running the agent in a loop. So what it's doing here, we can actually just look at this too, while true. So it is an agent loop.

And we get a response from the agent, and we see if the response says stop, meaning if the LLM says, I'm done, then we return the value. If we get any other response, we don't do anything except add these events into a trace.

So we just push history into a big list of history. Does that make sense? And so that's all we're doing here. This is just a loop where we just collect events until we're done. OK? So this is super basic.

Now let's see how it works. So I'm going to come over here, and I'm going to do, are you OK, sir? Do you need water? I'm going to npm run agent. And so it's going to open Chromium. It's going to, OK, Hacker News, so far, so good.

Click upvote. Oh, no! So we hit a login screen, and then it kind of panicked and crashed. But look, it lies. You see this? This is a problem. And so what's the solution? Prompt it harder. No. Change the system prompt.

Always log in with these credentials included in the system prompt. No. So how do we then solve this? And look, because of my login, we can actually see it just clicks the upvote button and then considers it a success.

It doesn't verify. This is the job of a harness. OK? So now incrementally, we're going to slowly start building a harness. And so let's just, I'm not going to write code here. I'm not going to live code, because we don't write code anymore.

We inspect diffs,right? Anyone write code by hand? You don't, maybe actually you do belong here. Anyway, so I'm kidding. So this is the first change we're going to make. This was our index file. And we have this run loop that I showed you, but now we're going to add one thing to it, which is default guardrails.

We're going to create some guardrails. OK? What do our guardrails look like? Well, let's go and look at it in the editor with guardrails over here. And so we have some types. But these are our guardrails. We have two.

Max iterations, meaning if you do more than six steps, I'mma kill you. And max messages, meaning if you have more than this many messages, I will compress the context. These are just guardrails. OK? A little utility to combine them, and we just, we can compose them here.

We could do like as many as we want. So now, let's go back to our changes. That's the guardrails. If we go back to the agent loop, we actually use the guardrails here in this diff. And so we include the guardrail functions.

And we can see that here, what we're doing is we're checking how many messages have we accumulated, and we just like trim the context if it's too much. But what I did want to show you is here at the end, we push context size, just some more metadata about what we've done with our guardrails.

OK? Our context compressor is extremely basic and extremely naive. This is what it does. Let me actually open this with syntax highlighting to spare you. This is what it does. So what we're doing is if we always keep the system prompt and the user prompt, and the most recent two messages.

So if the guardrail is triggered, we always remove everything after the system prompt and the user prompt in the middle, and we keep the last two messages. This is super naive. Don't do, there's better ways, but this is where babies first.

We're getting there. So we're starting to have a harness, but it's not called a harness. But this is really like a pregnant harness. Like it's almost born. OK? And so what we're going to do is let's just call it a harness now.

So I'm going to show you another diff where we, here, check this out. Index, we've deleted almost everything. And we've moved it into this file called harness. Let's go look at our entry point now. Index, it's all gone.

So the prompt is there. But this is, it's like 19 lines of code, and we just have run harness. We've taken all the logic from here and hidden it in a function called run harness. And as you'd expect, run harness does exactly the same thing as we did in the index.

OK? Nothing new is here, except maybe like a print function, which is just console log. Is this clear so far? Yeah, we just moved stuff. Now that we have something called a harness, we can actually use it. And let's solve the problem of lying first before we solve the problem of logging in as me.

Yeah? Because it says I upvote it. It did not. I want to know. So what we're going to do is we're going to add some guardrails and have it tell the truth. Like if you failed, tell me the truth.

How might we do that? Well, we'll check it out here. So many, many things changed. Or not. I don't know. Hang on a second.

Yeah, OK. It did. Many, many things changed. So we run harness, and we added a third argument here, which is a verify step and max attempts. Max attempts goes to our guardrail. So if you took more than three tries to do this, just give up.

And if we go to the harness, we added a lot of things that are just manual code. This is not different prompt. This is my logic. The main logic is run harness no longer wraps over the code we moved.

But we moved that to a different function called run harness attempt. So if we come to run harness, let's go here. I need to check the branch out. Sorry. Yeah. So now, if we go to run harness attempt, I'll collapse this.

I'll collapse this. I'll collapse all of these. And if we go to run harness attempt, now this is the same thing from our index. We just moved it into a function called run harness attempt, because our main run harness is just a loop that runs no more than three times.

OK? Is this clear? So we're just enforcing the max steps, but at the harness level for safety. Then we have run harness attempt that calls it. We have this function called verify successful upvote. I wrote this. This is deterministic.

That's what I want to show you. What does this do? Well, we see if, remember we were tracing in the agent loop? We were just adding history events. So we reflect on that, and we see if there was a browser click on the upvote.

And if it's successful, but really successful, then we say true. But there's a huge butt here, which is we have now cases for failed login. If there's a tool named harness auto login, and if the message starts with failed, then we return early and we say, no, no, this failed.

We're removing the lie. OK? Similarly, unrecovered login redirect. We look over our agent loops tools that we've been pushing into. And if we see that the harness auto login didn't run, and now we're on the page that is the login URL, then again, we just fail.

OK? And so what we're doing is we're just adding, like if this happened, if this happened, just failed, return early. Is this clear? This is what a harness does. And so let's run this now with the harness, npm run agent.

And now it's going to go into Hacker News, and we're going to repeat the same cycle. OK? It's going to come here. And now it still failed, but look, it stopped lying, because our harness checks the tool history and actually sees what happened.

This is what a harness is supposed to do. Great. This is already like half the battle won, because step one to solving a problem is admitting you have one. OK? Test-driven development vibes. So now that we're failing correctly, we can succeed.

And I'd like to show you that in the last diff, and then we'll finish the talk here. So number four, we have a whole new function. It's called login handler. I'll add some syntax highlighting here so you don't go blind.

But here, create login handler. This is all it does. It runs every agent loop just before we push to the traces. And this is what it does. It checks the browser session's current URL. And if we're not on a login page, it just says, cool, I didn't, return, I have nothing for you.

This computationally is not costly at all,right? If you're not on the login page. But if you are on the login page, then we fill in a temporary, this can be an environment variable. It can be secure. You get the idea.

But we fill in credentials and submit the button programmatically from the harness, not from the agent. Deterministically and securely, because this file has access to any secrets I wanted to. Right? And so this, how is this called? Well, this is called in the agent loop.

So if we go back to our agent loop, and notice we were pushing traces, yeah? This is where we push the traces. Just before, if we have a login handler, we call the login handler just before this in the agent loop.

What does the login handler do? Well, if we're not on a login page, it does nothing. If we are on a login page, then it quickly will inject credentials and submit the form and then take you back. It will also add, as we can see here, it pushes a message into the queue saying, hey, I'm the harness.

I logged in. You're good now. Is this clear? Yeah? So the harness is literally harnessing the agent to something stable, something deterministic. That's what it's for. OK? Let's run this now and see what happens. So npm run agent.

It's going to open Hacker News. And when it gets to the login, now that harness step, it logged in, and it upvoted the first one. And it closed. Amazing. So successfully upvoted, a little snitch for NYLX, rank two, succeeded after six iterations.

And I can click this and go into Hacker News and actually see, indeed, it was upvoted. And I can unvote now, which means it was upvoted. Right? And so the agent used the computer, logged in as me, with my harness that I just made here on stage.

That's the purpose. Is this clear so far? Do you understand now the role of a harness? Look at you nodding. This is music to my ears. Fantastic. Something to my eyes. I don't know the, is beauty to my eyes?

Kind of weird. We don't have an expression for that. Let's land the plane. I'm done. I think my work here is done. What does this look like in practice? Why do I care so much about harnesses? Because they run the world.

Models are non-deterministic. And you want to do more with less. You want to use a cheap model. Use like Qwen or something, or even something smaller. Use GPT-OSS. It's free. And with a great harness, you can go very far.

That's why. At IBM, we create an open source project that we deploy in the enterprise that allows very large companies, huge companies, in their private data sensitive areas to perform RAG operations on all kinds of things: Teams, calls, and PDFs and invoices.

We build, it's called Open RAG. And it's RAG, I don't know if RAG is cool or not anymore, but Open RAG has a hell of a harness that provides enterprise level security to like asking questions with internal very, very siloed data.

And that's kind of where the harness engineering comes in. So let's summarize. We covered a lot of content. Was it a deep dive? I think it was a deep dive. It was a deep dive in like 18 minutes or so.

We went pretty far. It's not, it should not be lost on you that I did not touch the prompt once. I did not change the system prompt. We just built a harness, and the outcome radically changed. And of course, we can add secrets.

We can add tokens. Yeah, we did a lot. In the end, I hope you understand what a harness is, the value it can present, and how you can use it. What's next? Look, I don't have a crystal ball like everyone else here.

But it's not lost on me that 2025 was the year of agents. Yes? 2026 is the year of harnesses. I'm pretty sure everybody, how many times is this word used here? I think, I would hope, I think it would be pretty cool if 2027 was the year of dynamic on the fly generated harnesses.

How cool would that? Like you tell an agent, hey, do this for me. Buy me a flight ticket, whatever it may be. And then before doing the work, the agent creates a harness. This is similar to plan mode.

Any of you using plan mode? But on steroids. The agent creates an actual harness, self-aware. It knows, oh, I can maybe hallucinate here. I can maybe, creates a harness, does the job, and returns back to you. Guardrailed and everything.

That is so cool. Dynamic on the fly harnesses. I would, I think this is honestly the next logical step towards AGI, and I would love to see it. I don't know if this is just me being, you know, weird guy with ideas, but I think that's kind of the direction.

So with that, I'm almost out of time. I would be really remiss if I didn't spend the last like 30 seconds saying thank you so much. The slides are on GitHub, as am I. And so I'd love to chat more.

Thank you.