# Your coding agent doesn't always follow your rules — Talha Sheikh, Checkout.com AI Engineer · 2026-07-08 Talha Sheikh from Checkout.com argues that the real value in AI coding agents lies not in the code they generate but in the verification layer that checks it. Drawing from his experience with Claude Code, he built Vector Harness—a tool using Claude hooks to deterministically test agent outputs against a config file, retrying failures automatically. He contends that model capability does not equal reliability, and that even with better specs or models, deterministic verification is essential. Sheikh observes that Anthropic, OpenAI, and CUDO are all adopting similar harness patterns, shifting the focus from instructing agents to enforcing rules. His central claim: work on the harness, not the code, and you can even use smaller, cheaper models with enough guardrails. ## Questions this episode answers ### What is Vector Harness and how does it make sure coding agents follow my rules? Talha Sheikh built Vector Harness (originally Vector V1) to deterministically verify the output of tools like Claude Code. It uses hooks to trigger checks after a session, running tests defined in a config file. If a test fails, the harness tells the agent to retry until the output passes all tests, ensuring the work actually matches the required spec and not just the agent's own claim of completion. [2:05](https://aie.addtry.com/2eb22668-c770-4f81-ba70-540be80630b1?t=125000) ### Will future AI models become so smart that we no longer need verification layers? Talha recounts an Anthropic engineer telling him future models would eliminate the need for enforcement, sparking a “crisis mode.” He counters that capability improvements do not equal reliability, and points to Glasswing’s Project Metos as an example of a touted all‑solving system. Even with perfect instructions, verification remains essential because telling an agent what to do is never the same as confirming it did it. [3:09](https://aie.addtry.com/2eb22668-c770-4f81-ba70-540be80630b1?t=189000) ### How can adding a verification layer cut the cost of using AI coding agents? Talha explains that with a verification harness providing strong guardrails, you can use smaller, cheaper models like Haiku or open‑source alternatives instead of always relying on expensive frontier models such as Opus. Investing more effort in the harness allows drastically lower per‑task costs and even enables asynchronous operations, creating a cost‑efficiency spectrum. [4:43](https://aie.addtry.com/2eb22668-c770-4f81-ba70-540be80630b1?t=283000) ## Key moments - **[0:00] Trust Issues** - [0:51] "All I want to do is play Cyberpunk on my Xbox while I have Claude Code do some work for me." — Talha Sheikh - [1:19] "I am the enforcement. I am the enforcement layer." — Talha Sheikh - **[1:38] Enforcement Layer** - [2:18] Talha Sheikh created Vector V1, a tool using a config file to define tests that deterministically verify Claude Code's output, retrying on failure - **[2:57] Sharing Vector** - [3:09] Anthropic engineer predicts that future models will be so smart that enforcement layers like Vector won't be needed - **[3:28] Crisis Mode** - **[3:45] Reliability Gap** - [4:03] Increased model capability doesn't equate to increased reliability, says Talha Sheikh - **[4:14] Guardrails** - **[5:17] Fragmented Efforts** - **[5:41] Pattern Emerges** - **[6:34] Design Contract** - **[7:11] Industry Adoption** - [8:11] "Slow the hell down" — Talha Sheikh cites a keynote warning about the need to verify AI outputs - **[8:26] Value Shift** - **[9:00] Q&A** - [9:29] Q: What do you say to allegations you're a top token spender? A: I need those tokens to build a verification layer ## Speakers - **Talha Sheikh** (guest) ## Topics Agent Evaluation ## Mentioned Anthropic (company), CUDO (company), OpenAI (company), Work OS (company), Claude Code (product), Vector (product) ## Transcript ### Trust Issues **Talha Sheikh** [0:15] Hello, hello. Um, have you ever given a task to Claude Code and you give it a feature and you're like, "Okay, cool, can you build this for me?" and Claude Code starts putting it out into subtasks and you see it, okay, this is pretty cool. You see it running multiple sub-agents and you're like, "Allright, that's really cool." And you can see it, like, ripping through all of your tasks, sub-agents being completed, and it gives you, like, a final output: "Task completed." Like, amazing, great. But when you actually try to run it, it should be like, "Oh, well, it's, it's not." Something has failed. "Hey Claude, can you fix this little bit thing?" "Oh, well, let's try it again." "Okay, it's fixed. Everything should be working." "Oh, no. Actually, just this tiny little thing is just missing." So, and that's, and that's what my talk is about. All I want to do is play Cyberpunk, uh, on my Xbox while I have Claude Code do, do some work for me. And what I realized was, the problem is, the problem is that I kept on, uh, telling Claude, like, "Hey, fix this, fix that, fix this, fix that." Even though if I give it a spec, if I give it some instructions, if I give it little to no instructions, every time there is something that I need to tell it. So what that means is, like, I am the enforcement. I am the enforcement layer. I have to tell Claude on what exactly you need to do and how exactly it needs to be enforced. So the agent says it's done, and but you have to check it anyway because there's nothing else that can check it for you. ### Enforcement Layer **Talha Sheikh** [1:38] So what I wanted was something to be very deterministic. So when an agent says it's completed, have this enforcement layer deterministically check something like whether it's actually been done or not. Actually, the way I wanted it to be done. Because it says it is done, but is it the way that I want it? So I needed some way, deterministic way to do that. So I tried it. I built my own vector, I call it my own product called Vector V1, and it deterministically checks Claude's output. And the way it did that is through using Claude hooks. So that way, whenever Claude finishes its, its session, it automatically, the hook calls my vector product or program and it checks it for me. Cool. And this is how it essentially looks. So I basically give it a config file, define all of my tests over here, like what I needed it to be checked. And if it fails, it can actually keep on telling Claude, like, "Hey, look, this is failing. Try again. Try again. Try again." Sorry if it's, uh, a little bit, little. So you can see one of the test outputs over here. So it's like, okay, first the test, test passed, first failed. Then it retries again, then all of the things pass. Okay, cool. So what that means is, it's not about whether Claude can actually do the task, it's about trust. Can I trust Claude to actually do everything for me? And by the way, when I say Claude, I'm just talking in general about LLM agents in general, is when I give a task to a coding agent, does it actually complete it? So, yeah. So, and then that's something that, and then what I started doing was started telling this about to people about and going to different events. And it was, it was really cool. Like, "Hey, look, what about this verification feature that I built? ### Sharing Vector **Talha Sheikh** [3:09] It was so good. It was so amazing." And then I met one of the, uh, Anthropic engineers. And, um, he, and, and they just told me that we're not going to need this anymore. Like, we'll have, like, another agent or another model that will be so smart that you won't need enforcement. Okay. Um, so crisis mode. Did I just waste my time? What did I just, like, what was, what was the point of all of this stuff? But let's dig in a little bit deeper. And then also, they released this project, Glasswing, that shows project Metos, which is supposed to be so good that it will solve everything for us. ### Reliability Gap **Talha Sheikh** [3:45] So what I started, when I started thinking about it, like, okay, what is it that is actually happening? When a new model comes out, doesn't it increase in capability? But that's not necessarily the same thing as reliability. Sure, the models may, may become a little more capable, but are they more reliable? The other thing is, like, another argument is like, "Oh, well, I can have the best spec. I can have the best MCP servers. I can have the best sub-agents. I can get all theright context to it. Amazing. We should do that." But inst giving Claude instructions is not the same thing as giving it verification. So you can give as much instructions as you want, very good instructions, very little instructions, but you still will need to verify. And what I realized was that just what I realized was that having these small guardrails, or having as many guardrails as you want, technically you can use a smaller model, like a Haiku or even like an open source models, because it's got these guardrails on, it'll most likely be succinct and get you to the output that you want. ### Guardrails **Talha Sheikh** [4:43] So in theory, what it means is that if you use a frontier model like an Opus, uh, Opus model that can get you a task, okay, cool, that will be the most expensive one. You can have Vector with a little bit of guardrails, but it gets you a little bit cheaper. But if you put on more guardrails, that means it invests a little bit more time in the harness itself, like you can reduce the cost drastically. Or maybe even use, like, async tasks as well. So, okay, I was feeling, I was feeling good and I started talking about, about Vector again to different events. And as, as I spoke to more and more people, um, ### Fragmented Efforts **Talha Sheikh** [5:17] what are there is missing something missing here. What when I spoke to more people, what I realized was everybody's building their own stuff. Anthropic is building their own stuff. My company is building their own stuff about enforcement. Facebook is building their own stuff. Meta is, every company is building their own thing. So if I had built, if I built something that is specific to me, then I can't really share it with others because everybody has their own way of doing it. And what I enforce doesn't necessarily mean that somebody else would enforce the same thing. So what that meant was, what I realized was, okay, so it's actually a pattern. So it has to be a pattern that is applicable to everyone. ### Pattern Emerges **Talha Sheikh** [5:54] So what that means is that we can, it has to be language agnostic. It has to be something that can be shared by everybody else, and everybody can bring their own version of enforcement to it. And that's, and, and it should run on every level. So it should start, start off with in conversation. When a conversation ends, you can have checks when, uh, before committing. You can have checks when you're part, is part of a multi-agent workflow. You can have it, uh, on checks on asynchronous operations or asynchronous agents. And as well as you can have a check that non-deterministically calls, like, um, LLM and LLM as a judge sort of a thing. And it can run on any, any different language on any different code. As long as there's a capability to run it deterministically, we sh we can have that. ### Design Contract **Talha Sheikh** [6:34] So what, what I realized was, what we needed was essentially a contract that just says, like, "Hey, given this task, I want you to fulfill this." What is in the middle that you can, the developers themselves can define. So this idea is really cool. And it was like, okay, so we're moving towards, we want verification always. Allright, cool. Um, so a lot of different companies have actually started doing this as well. So Claude Code, uh, Anthropic has recently released their new thing called Executor Advisor pattern, where you've got one agent that actually does all the code, all the code work, and then there's advisor that, you know, feeds in, essentially creates a feedback loop. ### Industry Adoption **Talha Sheikh** [7:11] Or in other words, verify. Anthropic, uh, sorry, OpenAI built their own harness engineering, and it's the same idea. Like, you give an agent a lot of, a lot of things to do, but how do you verify it to work? You give it different tools. You give it different contexts. And that's essentially what a harness is for OpenAI. There are companies like CUDO who are, who are over here that provide very comprehensive code reviews. And again, it's the same thing. The agent has done all of its work, but do you trust it? No. So what do we do? You do a very comprehensive PR review with all the different issues and findings and create this feedback loop. Some, um, something from today as well from Work OS. So it says enforce, don't instruct. So it is all about, like, running these checks deterministically. When I say checks, it's just about the verification. Another one, which is my favorite, is, uh, one of the favorites, like, you still have to go slow. And the reason for that is not because the, the agents themselves are not able to produce code as fast as they want, but it's because the verification layer. You need to verify that everything is working or not. And my favorite is, is this one in the, in our keynote. It's to slow the, slow the hell down. So, so what is the shift that we're seeing here? ### Value Shift **Talha Sheikh** [8:26] What initially what we thought was, like, the value is in the code that we create. But it's actually now, in reality, is, uh, what we're seeing here is the verification that we design. So it's not about can you code, but can you verify. So TL;DR is, uh, work on the harness and not on the code. So you work on the verification system and that produces a little bit better, better in outputs. And that's it. Thank you. ### Q&A **Talha Sheikh** [9:00] Any questions? I've got 40 seconds. Yep. **Guest** [9:08] Does it exist? Is it public? **Talha Sheikh** [9:10] Yeah. Yeah, yeah, yes, it is public. Um, it's called Vector Harness, but if you, if you send me a message on LinkedIn, I can share that with you. **Guest** [9:17] Can you put the LinkedIn back up? **Talha Sheikh** [9:19] Oh, there you go. **Guest** [9:21] Thanks. **Talha Sheikh** [9:25] Cool. Yeah. **Guest** [9:29] You mentioned that adding the verification layer allows you to use smaller models. Uh, what do you say to the allegations that you're a top token spender at your company? **Talha Sheikh** [9:40] I need, uh, I need those tokens to build a verification layer. Cool. Um, I think that's it. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.