Mergeable by default: Building the context engine to save time and tokens — Peter Werry, Unblocked

Allright, thanks everyone. Sorry about the wait. Um, this is going to be a bit of a strange session because there is a workshop component to this, so I guess everyone will be coding on their laps. Sorry. But anyway, sorry, I'm Peter, and this is my colleague Brandon.

So we're going to break this session into two different parts. One is sort of a talk that I'm going to give about what context engines are useful for, and how you might go about building one, what to think about.

And then we'll launch into the second part of it. So, just briefly, quick agenda. We're going to talk about three myths that are circulatingright now about context engines, and then I'll go over a couple of lessons, or a few lessons, that we learned along the way building one of these things.

And then finally, we'll do this. So we're going to build a social engineering graph. This is a component that is super useful in a context engine. And first, just to show hands, does everyone know what I mean by context engine?

Or does anyone want clarification on that?

Yes, ma'am.

Okay. So in the world of AI agents, you have agents that, when you start off and you start coding, they are basically at ground zero. They have no context about your code, your organization, nothing. Okay? So typically what happens is, the first thing they do is they start to rip around your code base based on the task that you give them, to try to gain some understanding, sort of background understanding, before they start to do their task.

So context engineering is kind of the art of supplying all the context that you need, and most importantly, none of the context that you don't need, in a highly optimized way so that when the agent starts to run, it executes the task in a streamlined way that's in line with your organization's best practices and expectations and so on.

Okay. So we'll get to this after.

So not long ago, as in like four years ago or less, you were the context engine. Okay? So when your agent needed something, you would prompt it, you'd grab the issue ticket, you'd hand it all of the information that it needed to start its task.

And in many cases, even when it was ripping around, getting background context, when it got to the end of its task, sometimes it got it wrong. In fact, many times it did, and you'd have to kind of like reset it, re-guide it towards the solution that you were thinking of.

Or if it completely missed the mark, you'd have to be like, no, not the JavaScript dummy, it's the Python source code that I want you to look at. So

let's just remember how you built context in an organization. So we're taking AI out of the picture for just a sec, and I just want you to pretend that pre-AI, you just joined an organization. Let's remember how we built it up.

So

over time, you would accumulate this kind of context through experience,right? You would start a job, maybe start code-splunking a little bit to figure out

how things work. You'd maybe latch onto a mentor.

And eventually, you'd experience real things, like incidents and outages and things like that. Those are sort of the pain things that stick with you. Those are the battle scars,right? And that is what constitutes organizational context. It's the learnings along the way, the why did we do things the way we did it.

And now you're good at your job because after all of that experience of pain, now you know what questions to ask. You know where to look when an incident happens. And this is the goal. This is what we want to get for our AI agents.

So I'm going to just lift this adoption curve from Vaseem Aldath. And I may have butchered his last name, but sorry, sorry, Vaseem, if you see this. So let's start at the beginning here. This was like four years ago in 2022.

Everyone remembers fancy autocomplete,right? So back in those days, context windows in AI were pretty limited. I'm not sure if everyone even remembers this, but it was like 8 kilobytes, or 8k tokens, I should say. And that's not a ton.

And so tokens were highly optimized, and agentic IDEs like Cursor focused just on the code that surrounded the code that you wanted to go in autocomplete. So basically, they took some code before, they took some code after, they put it into a model, and they said, this user is working on this piece of code, what's the most likely next thing?

And that's what was printed out. It got progressively better than that. ASTs were integrated, language servers, and then you were able to basically pull like colors of source code and pull all that into context. And then the LLMs were really good at completing code.

So at those levels, you were the context engine. And in many circumstances here, this is kind of where most people are here. They're at the parallel agents hooked up with MCP and skills. Okay? Just super curious, has anyone gone beyond curated context into the last few degrees of agentic freedom, shall we say, where you have background agents running in the cloud doing stuff in YOLO mode?

Is anyone experimenting with that? Okay, cool. That's very cool. That's bleeding edge. But let's just take a moment to recognize that bleeding edge today is like yesterday's news in six months. Okay? So I'm Canadian, so I'm going to say this.

The puck is going down the line towards background agents for sure. And one of the things that we run intoright now is this, we're becoming the bottleneck as humans,right? I'm not sure if people have tried managing parallel agents and working on several tasks at once, but everyone's starting to feel this like cognitive disconnect because you're context switching all the time, and it's just really, really painful.

It is very difficult to move from that mode where you're the human managing context into the background agents mode unless you have some kind of context engine that knows how your code operates, how your organization works, and understands the motivations for historical changes and things like that.

So Andrey, he nailed it.

Systems are intelligent. We're reaching the exponential on intelligence for code pretty soon. Everyone's seen the release about Mythos, even though we all haven't had a chance to really try it out yet. The promise is that from a code intelligence perspective, this thing is like pretty much close to perfect.

But so now the bottleneck is context, of course.

Without context, I'm just going to reemphasize this point, you'll probably end up in doom loops. Does everyone know what a doom loop is? A doom loop is like when you're struggling with the agent. It's not quite doing what you want, and you have to keep iterating on it.

The worst-case scenario is you run this thing in YOLO mode, and it finishes the entire task, and it's completely wrong. And you have to go back and correct various stages. So when you have a context engine, you can get there faster.

The problem is that access doesn't equal understanding. So we have customers that are on various parts of, I'm just going to go back to here. We have customers that are on various parts of this journey. And one of the interesting things that we've noted is that people feel that they understand their organization best.

So when it comes to feeding theright context to these agents, people will try to build some semblance of what a context engine actually is. They'll maybe build a RAG system, or they'll build some way to like feed organizational data to an agent.

So unfortunately though, access doesn't mean understanding. So what that means is you could just wire up a bunch of MCP servers, and it's not going to be able to understand what the relationships are between all that data, how it got there, and why it is the way it is.

And then there's another problem, which I'll talk about a little bit later, called satisfaction of search. So just remember that term. I'll come back to it. Okay.

So I just wanted to show you this. This was something that we actually implemented, and we did it in two parts. One was just without any context engine, but wired up to a bunch of MCP servers. It did a pretty good job, but then when we reached the end, it missed the fact that we had some legacy stuff that depended on this old method of intelligence size to Anthropic.

So they have adaptive thinking now, but it used to be you had to supply a token budget, and that's how like you could increase the size of the thinking window. So we had some code that kind of like depended on this, and there were reasons for that that the agent didn't understand or see, and so it just basically clogged all that code.

But when we added the context engine, then it saw all those reasons and implemented it theright way. So it made the appropriate changes in theright places, included backwards compatibility for the code that was using the old method.

Okay, so now for the myths. Myth one, naive RAG over my docs is a context engine. So if you implement, say, like vector search, or just a couple of search methods, you're going to run into this, you're going to run into a few issues.

One is the satisfaction of search problem, where the agent will search like crazy, consume your tokens, and then in the worst case, you'll reach compaction. Okay? So without being able to find the end game.

There are a few different other techniques, like you need to have personalization when you build a retrieval system, because if you just RAG all your data, especially for very large organizations, there's going to be things like conflicts that you have to resolve in the data.

It won't be focused on the task that you're trying to perform. It might pull in relevant code from other parts of your organization that, especially if you have a really big org and you've got tons of different repos, it's just going to create a huge mess.

So you need to have some element of personalization. And then here again, connect a bunch of MCPs. I'm just going to reiterate this point. I'm done. Nope, definitely not. So that is the thing that really puts an emphasis on the satisfaction of search point, and I'll explain that in a sec.

And finally, a bigger context window will solve this. So way back, when the models were starting to get big, people were really excited about a million tokens in your context window. The first models that tried this, I think it might have been Claude actually.

Was it Claude?

I think it was Codex.

Oh, or OpenAI? Okay, okay. Gemini, yes, sorry. I'm so sorry. So yeah, Gemini, first model to try this, and it was really good at finding needle in the haystack. So you could feed like a huge document to it, and as long as you knew what you were looking for ahead of time, it could find it.

But it wasn't good at all at reasoning across different data sources, understanding the real meaning behind a problem, and then recommending the appropriate solution. So none of that was possible. Obviously, things have gotten much better. Now the problem is, most organizations have more than a million tokens worth of context.

So trying to fit all that into their context window isn't going to work anyways. Let's project out to the future and imagine that you could fit like 10 million tokens, 50 million tokens. At the current rate of memory consumption just to operate the models, that's not going to be possible for a really long time.

Even if it was, and you fit all that context in your context window, you're still going to run into problems with understanding what's true, what's false, how to select theright information. Okay? So now I'm going to come back to this second point here, satisfaction of search.

This is a term that actually comes out of the medical field in radiology. And the idea is that when techs are looking at X-rays, and they're looking for the cause of symptoms, they might find something on the X-ray that explains those symptoms, and then they stop.

And that's kind of like a dangerous thing medically, because there might be other indicators for things like cancer that get missed. So satisfaction of search is a real problem in radiology, and there's lots of protocols to prevent just stopping as soon as you find the first thing.

This is what happens with agents. When they search around in, say, Notion and your code, Confluence, they'll stumble across what looks like the thing they're looking for, and they'll stop, and then they'll proceed. But the real golden nuggets of information might be in a different place that the agent wouldn't think to look, like in a past Slack conversation or an incident report, something like that.

So here's the classic iceberg meme.

Code that compiles, that's like the baseline. Does the agent produce code that compiles? But everything that is actually important happens underneath here. So understanding the user's original intent, what was rejected in the past by the team and tried before, but failed, how are you going to surface that kind of content just by looking at docs and code and stuff?

So you need to understand that somehow.

And even worse, like it's sometimes hard to know when things were deleted, like in the absence of information. So you need history as well, leading up to decisions.

So this is why we think you need a context engine. A context engine understands who you are, what team you work on, who you work with, who the experts are in your organization, and what the decisions were that led up to the current iteration of your code base.

It's able to resolve conflicts. So this is like a truth and false type situation, what's true, what's not. Sometimes that truthiness is a gray area,right? So the context engine needs to also understand when to instruct the agent that it wasn't able to resolve the conflicts, and then learn from additional user input.

This third point is super important, of course, in any large organization or enterprise. There's often repositories that not everybody can access, secret projects, that sort of thing. So it's really important that you flow the access controls up. I'll give you an example that everyone will appreciate, which is Slack.

Our context engine integrates with Slack or Microsoft Teams. And when you have private channels, that's really highly sensitive,right? Like you could be discussing HR information or maybe something that you just really don't want everyone else to see. And so when Unblocked answers questions, it will use private channel information, but it will only use that information if the person that's asking the question has access to it.

And then those answers are not public. Okay? So they're private to you.

And then finally, of course, delivering theright contest at theright time. And this is about token efficiency. It's about getting to the answer as quickly as possible. So here's kind of a high-level overview of how a context engine might work.

On the left, we've got data source inputs. So things like planning tools, docs, conversations, code, PRs, basically like anything that's relevant to getting work done at the engineering level. And then on theright side, we have the outputs. So this all can flow to coding agents, MCP, or CLI tools.

You can custom build apps through the API. We have a code review component that just plugsright into your SCM and provides code reviews, and of course, integrations with social messaging apps.

So these are the kind of like broad six requirements that we think are important. There's actually much more than this, but these are the high-level things. So again, unified system context. This is about building relationships between data. Okay?

But it's more than just recognizing when one piece of data is related to another. Like for example, in Slack, you might have conversations about PRs. That's an easy linkage because you're posting links back and forth. So that's easy.

What's less easy is understanding the reason why decisions were made, or your organization's best practices,right? So to understand that, you have to go a little deeper, do things like distill historical pull request comments on PRs, and try to distill those down to their core essence.

And then when you see repeated patterns, you can pull those patterns together and store them as quote unquote "memories" so that when someone is working on a similar piece of code, you can load those memories and then the agent can see that and go, "Oh yeah,right, this is the way this organization does this particular thing."

Conflict resolution, super important. We took initially kind of a naive approach to this at first, and based it just on recency,right? So we would bias towards newer stuff. Unfortunately, when you have in the fullness of all your context, recency is not enough.

Often you have people writing documents or chatting in their messaging platforms, and they might be saying things that are not like completely aligned with how the system works.

So then we started to bias towards code. So we had recency and we're like, the main branch is definitely your source of truth, but not always, because sometimes what's important is what happens next, not the way a system currently works.

Like when you're working on a task, what you really want is for the agent to understand where you're going, not necessarily where you've been. Where you've been helps it understand what not to do. Where you're going helps it understand what you should do.

Okay? So in the Slack case, looking at the conversations that your organization's experts are having is more important than just understanding what every random engineer is talking about.

Targeted retrieval and personal relevance are very related, so I'll just talk about them together briefly. So again, like when you're pulling context in, it's important that you're only pulling context in for the relevant task at hand, and probably relevant to you.

So here's a technique that's kind of interesting. You can understand what repos a person works on most by the number of PRs that they submit contributions. And then if you're doing vector retrieval, you can do a deep retrieval on those focused repositories, and then a wider retrieval on the rest of the source code, and then sort of bias the selection towards the focused repositories, because that's more likely where someone's going to be working and spending their time.

And then we've talked about data governance, so I don't think I need to go over that again. Super important though.

This was just a little experiment that we ran with a larger task. I fully admit that some of these numbers are a bit wonky. This is basically like Claude outputting numbers. So don't trust it. Just trust the vibe of the thing and not necessarily the numbers.

Basically what it's saying is that when we started out without the MCP server, or sorry, without the context engine active, it really missed the mark on a lot of stuff. And that's just because it didn't understand how the existing implementation really worked and why it was the way it was, what was tried before and failed.

And so it made a lot of those same mistakes. With the context engine turned on, obviously, it nailed it. The key numbers though are the time and the tokens that it took. So without the context engine, it took two and a half hours to finish this task with 21 million tokens, which is a lot of tokens.

But with the context engine, it took only 25 minutes and 10 million tokens. So it's a pretty dramatic difference. Okay, so the hard lessons. These are just samples, by the way, but these are ones that we thought were kind of interesting.

So first of all, initially we optimized for access, not understanding. So our first premise was if we just wire up a bunch of tools and provide a knowledge graph, it will be able to traverse the knowledge graph and execute a bunch of retrieval-specific tools for particular integrations and so on, and figure everything out.

That does not work. So you'll have to go a little bit deeper than that.

Second one is we hid conflicts instead of surfacing them. So by hiding conflicts, I don't mean that we just ignored the conflicts. What we did instead was we tried to resolve those conflicts using those naive strategies, and we didn't surface the conflicts that we weren't able to resolve.

So this was a really good learning is that a context engine, I mean, we'll get there eventually probably, but it can't always tell what the truth elements are. And when it can't, you should surface that and learn from it.

That's the key thing.

And then finally, I think a lot of folks try this. This is a really bad idea. So when a context engine supplies an answer, do not cache the answer and try to serve that same answer up again to a similar question.

The reason is fairly obvious in retrospect, but everything changes constantly,right? Code changes, docs change, the reason for things change. So this just doesn't work. The other thing is if you try to

use the previous answers as context for new answers, you regress towards a mean. So if the model is like misbehaving or doing something bad, and you continuously bring that into context, you're obviously going to pollute the context.

And this is what happens.

Okay, so let's now talk about where AI forward teams, like those that are doing this like cloud-based agent thing, are using and taking advantage of context engines.

Definitely, and especially during the planning phase. Okay? This is where you get the biggest bang for buck unquestionably. Get the context engine involved, use a skill to bring it in, connect it to the MCP server, and watch it do its thing.

This is where you get the biggest bang for buck. It's also useful to do this during review. So you get planning and review at the end. Because if you get an agent to do review, it's basically just going to pay attention to the code and try to understand where the breakpoints are, security concerns, that kind of thing.

But without the organizational context, it doesn't understand the motivation for it. So that's the really important thing. Pick enrichment. This is a super cool use case. So you create a ticket for a new feature, and then you just ask the agent that's connected to a context engine to fill in the blanks.

Works. Triage. I use this all the time. When I see an issue in production, I just whack it into my agent connected to the context engine, and it just like instantly brings up all the past issues related to this and starts operatingright away.

Increasingly, we're seeing this one, incident management. Okay? So we just wired up Datadog and this, sorry?

Sentry.

Sentry and Datadog. Sorry. And this is already proving like super cool use case.

It can see the signals, and then it can act on all the signals and relate that to code, relate it to past incidents and discussions that you've had in Slack. Having all those things come together at once is almost like magical.

And finally, I think this one's actually my favorite one, and it's the one that customers use the most, is customer success and sales and engineering support. So what a lot of big teams do is they have engineering support channels where other teams can come in and ask questions.

If you put a context engine into one of these things, you can have it automatically answer a lot of questions and save engineers a ton of time.

Allright, so how teams make a context engine their own skills. So definitely build skills that you can use to curate context in a GitHub repo. And you can build other skills around it, like typing ticket enrich, give it the issue ID, and then it can use the context engine to build the enrichment.

Workflows like this one prepare an incident timeline, and then you can just send it off to your agent again, context engine, blah, blah, blah. It brings everything together. Magical.

And this thing here, you can wire this up to all kinds of agents. I've got one of the things that a lot of customers like to do is wire this up to Claude code in their CI system. We actually do have a code review component, so you don't have to do this if you're using Unblocked.

But people use this for other things, not just code review. As soon as you wire up a context engine in the background, give it an API key, let it run on its own, it can do some pretty insane stuff.

So I'm just going to show a quick

example of what wiring up a context engine can do. So this is a PR that my colleague wrote, and Unblocked like went through and provided a kind of review to this thing. And at the bottom of this review, here's the review part.

You can see that Richie, who was the author of this PR, was like, "Very cool, this is something I would say." Now the reason for the comment, which was, "You've basically duplicated a bunch of tests," you can kind of dry that up a little bit, is because this was a best practice that was distilled from a bunch of other PRs.

And the funny part is that the author of those PRs was Richie. So he's the one that actually instilled the best practice in the organization. So that was just a cool little moment when we discovered that. Here's another example.

So this was a fairly long transcript. I'm not going to show the whole thing, but we sent it on a mission to do a big large task. Without Unblocked, it took quite a while. Like you can see, transcript's quite long, and it missed a whole bunch of stuff.

With Unblocked, it was a lot more compact. It got to the answer like very quickly and correctly. And just because we're now AI forward and lazy, we took both of those transcripts and ran them into Claude and just said, "Hey, Claude, why don't you just do an analysis of both these things and give us your result?"

So it went through. I won't bore you with the details, but just to say that at the end, the verdict is that the context engine plan is what I'd ship with. This other one is good for a prototype, but it's missing a whole bunch of stuff that is important to this organization that was previously discussed.

Okay?

So this is essentially what I've been trying to say. AI generated code should just feel like it was written by someone that's been in your team for like 20 years. Okay?

If it doesn't yet, that's fine. It will. If you wire up Unblocked, you'll see like a huge difference in performance of agents. And if you're building one of these things, absolutely. Like take all these things and build, and let's see where that goes.

So just before we get into the workshop component, maybe we'll just have like five, ten minutes of Q&A.

I'm Brandon.

And this is Brandon. So he'll help with.

I swear I'm on this. Oh, maybe. Is that with the recorder? Yes. Oh, thank you. Sorry. I'll give you a buck.

Thanks. So it's clear what it does for you and what kind of problems it solves. But to me, a big question mark is, what is the thing? What is the artifact that fits the bill? Is it like a program you install, an API that's hosted remotely, or an MCP server?

What is it?

It's all of those things. So a context engine, I'll explain what Unblocked is. Maybe I can just show a quick demo of it. So broadly speaking, there's a bunch of different surfaces to a context engine. You want to get it into your agent flow, and you can do that with an MCP server.

You can do that with a CLI tool, for example. We also have this dashboard surface where you can ask questions about your code. This is a pretty basic one, but you can see it understands who I am and what I've been working on.

And then we have Slack connectivity as well. So you can bring Unblocked into Slack, drive it in conversations, and have it auto-answer things. Does that make sense? Did I answer your question or?

I think my question will get answered in the demo or in the.

Okay.

But yes, API, CLI, MCP.

Yeah.

And product. It's mostly API though.

Yeah.

Oh, sorry. I got carried. I forgot that.

Thank you. So my question is, so as far as I understand, it's like a knowledge management and retrieval application.

Yeah.

And does this relate somehow to things like LLM Wiki like it was made popular recently by Andrej Karpati or the decision traces and context graphs.

Yeah.

Which was discussed a lot a few months ago.

Yeah. So you can think of all of those things as kind of useful components to a context engine. A context engine has to do much more than that because so agents are really good at recursing through a wiki, for example.

It depends on how you build this wiki because there's a bunch of things like organizational memories, best practices, experts in your organization that are used as pivot points for context retrieval. So a wiki doesn't solve those problems unless it has like a you could build a structure with it.

And I think Karpati discovered that if you treat a wiki as kind of like a file system, you can break it down and have the agent whack through it like a file system. By the way, agents are like highly optimized for file system traversal.

The compilation step.

Yeah, the compilation step. Exactly. Yeah.

Yeah. I'm sorry. Everyone else is ready.

Sorry. And maybe the same question, but is it a general purpose context engine or is it targeted against code? Because will it be useful as, say, as a business domain expert or sort of building up a business domain and then having this context engine use so all my other agents could use this as context for their business?

Or would you say that this is more like just for the code part of it?

So it's definitely engineering focused. The integrations are focused on engineering activities. So SCM integrations and other tools that engineers use. We are increasingly seeing customers using it for other purposes. So business intelligence is a key thing. And that's usually useful when people in business functions are trying to get an understanding of the product and its function.

We don't have like, say, Salesforce integrations wired up for that. So you couldn't use it to understand

anything that's sales related. It's really just primarily an engineering focused context engine. That's not to say that that won't change.

Too many.

More questions?

Yeah.

On the governance thing, if you're respecting access rights, how can it do sort of synthesis across stuff and then develop new knowledge internally that it could then surface to people?

So yes, you're correct to point that out. The synthesis is compartmentalized. So there are places that are compartmentalized like individual repositories. That's kind of the level of access. So if you can synthesize historical data based off of that and then correlate that with public Slack information, then that's one way to do synthesis without crossing the organizational boundaries.

So

the other way is to look at and tag when

synthesized information crosses those organizational boundaries. And you can take something like a group ID approach to that problem where you attach group ID tags to the synthesized information and then only retrieve it if the person that has access to that can build it out.

So first take the compartmentalized approach because that's where you'll get the most mileage. And then you kind of build up from there. I mean, this is the core problem with using a technology like GraphReg,right? Because GraphReg is like a pyramid where it builds up in layers and then basically summarizes at each layer.

But that unavoidably crosses permissions boundaries. So you have to create compartmentalized pockets. Yeah.

More questions?

That's a good question.

Time to hack after this, I think.

Yeah.

Yeah. So you've talked a lot about like all the different sources of information that you consume and putting them all together. When it's like synthesizing those down, is that still sort of like naive RAG vector search, all of that stuff under the hood?

Or is it like agents deciding what is appropriate or probably like combinations of all of them? But what is that sort of step?

Yeah, you'reright. It is a combination of all of them. So knowledge graph, like knowledge graph buildup happens in a bunch of different ways.

Yeah.

The PR thing that I showed you, for example, is like first you build a naive knowledge graph procedurally, and then from there you can use an LLM to distill down and summarize and build up those types of techniques.

Our context engine builds first like a knowledge graph from the base trying to leverage like all the different entities. It's kind of like a page rank thing where it builds up the relationships procedurally. And then, of course, it vectorizes data.

And then there are procedural tools that fetch data at runtime. A lot of the distillation for

conflict resolution happens in two places. So one is like during data ingestion time, there are tags that relate data to each other so that we can see if we can deconflict at that level and then like rank against each other at that level.

And then, of course, at runtime, you have to pass the thing to a judge with the criteria, and then it does additional deconfliction in real time. Does that make sense?

Yeah.

Okay.

One more question and then we'll answer.

So I was curious. You say conflicts, but at some point you get conflicts that something means revenue for one company and means revenue for another company. It's a totally different meaning how you can recognize that. So how do you get humans in the loop?

How do you use their ontologies and how can you do you use it when you run into it? So I'm very curious about that, actually, how you do that.

Yeah. So I can show you just a quick thing here. So you'll notice that at the bottom, the references that were used for answers are delivered both like to the human in this interface, but also to the agent.

Yeah.

So if the context engine isn't able to do the deconfliction, then at this point here, the human can step in and guide the agent when there are enough.

How does it do that then?

So you can literally just reply and say like, "That's not correct," or you can come here and.

Oh, yeah. Sorry. The player's been there.

Yeah. Or you can do this like not helpful and give the reason why. Like it is a bit of a manual process at this stage, but the signals that build up over time.

It's funny,right? So you're in production with some company, you might have you catch a lot of human intelligence by this,right?

Yeah.

That's amazing. Yeah. For a typical customer, how much do you have this volume of that? The metric?

Oh, it's huge. It's amazing. Like I was actually really surprised by how willing people are to give feedback. Yeah.

Can you give an example? One project that you catch the thousands or hundreds of these human feedback things?

I mean, at small team size, it's in the hundreds. So small team size being like 20, 30 people. At large team size, 100 to 200 people, it's like hundreds and hundreds of.

Oh, wow.

Of feedback. Yeah.

That's cool.

People just really like to interact with agents and tell them in natural language what's wrong. It's just a totally natural thing to do. Yeah.

Good for us.

Cool. Allright. Are we good for Q&A? Then we can get on.

We can ask questions as we hack, but.

Yeah. Let's get on to the workshop part of this. So we have created a actually, what I'll do is I'll just do this first. So you can do this now if you'd like. I will come back to the slide in a sec.

So the idea here is we're going to get everyone to join a Slack workspace that we created, and then we're going to get everyone into a repo where this sample code lives, and then we'll just start hacking away on it together.

Okay?

Hopefully, this is working. The clarity around.

Yeah.

Light mode. Thank you. One time, I need it.

Hopefully, that's working. Oh, I got some people coming in here.

Nice.

When you drop in, you'll see an AI Engineering London channel, hopefully. Hopefully, there's a link to click there. And please, in this thread, drop your GitHub URL name and I'll invite you.

The Unblocked link will not work until you do step two to get into the GitHub org.

My code isn't loading.

Oh, no.

I've got many people coming in, so I'm really hopeful that.

Is it network? Yeah.

Might be network. We will find out.

Okay. While folks are doing that, I'm just going to show you what we're getting into here. So this is the GitHub organization. What we're working on is a social graph builder. So what this is going to do is look at a source code repository.

So you can run this on your own repo. It's not going to upload anything. It's all local so that you can see this thing building up against your own organization. And it's going to do a bunch of things.

We're going to get basically a social graph out of it, and I'll show you what that looks like. And we're going to understand who the experts are and which parts of the code they work on. And then there's going to be a little interactive visualization thing.

So what the goal of this exercise is, is to get this thing up and running and just start hacking away on it. So like start submitting PRs as soon as we get this going.

So this is what it looks like. This graph here is our organization, Unblocked. And what you're seeing here is a relationship graph that shows who's reviewing whose PRs and who's getting reviewed, essentially.

This thing is

a distillation of all the different teams within Unblock. So this is roughly accurate, actually. Well, not roughly. It is pretty accurate.

I did this all the way back to the start of 2025. When you run the thing, I'd recommend maybe doing it for a shorter timeline because it will be a little bit slow. If you go all the way back to 2025, it could take like 15 minutes.

But it's effectively distilled who the teams are. And the only AI step in this is to label the teams. You don't have to run the AI step if you don't want to. It'll just use the parts of the code that people work on the most.

This tab here will show the experts in the organization and what they work on. So this is just broken down by project area and path and shows like what areas of the code have good coverage. Coverage is defined mostly by whether a high-contributing organizational expert is present and whether it's an actively contributed to part of the code.

And then finally, we'll have this interactive graph that breaks things down by team area. And we'll show like who the major contributors are. I'm over here on the AI team. Yeah. So that's it. Let's get everybody in and we'll start hacking away at this.

Are you sure to get them to your?

Yes, absolutely.

Let's put that back.

Yeah.

Many of you should have an invite who have put your GitHub already in. So please give it a check. GitHub is the worst.

So it's private or allright?

Yeah.

Yeah.

We will. It is an MIT license, so we will be making it public later, but for now, we needed it locked down.

I found it. It's just like.

Oh, did you oh, you're still not in Slack? That's crazy. What's your app? Just what's your GitHub thing? I don't have like.

It's KAS.

KAS?

It's G-U-U-L-E-M-M.

Oh, okay. But you should have an invite at the moment.

Yeah, of course.

Julian?

Yes.

Sorry.

Julian?

Yeah.

Oh, there.

Yeah.

I never realized this was probably faster.

I'll show you.

I'm comfortable.

I won't put it in.

What have you done? Let me type it.

It's a bit weird.

I keep trying to type Kiwi. That's not theright.

Yeah, no. It's Griswold. This is Dutch.

Which one?

I also have this slide that my daughter wrote for my daughter, so it's just ticket that.

It's yours? Allright. You got this one?

Julian.

And that was by the way teams with Ron Arwen.

So I think that the rest of this session is going to be now just like hacking away. So in a sec here, I think I'll take this down if everyone's got it so that Brennan and I can concentrate on working with you guys to build features.

It's going to be fun.

Oh, when you submit PRs, by the way, you'll notice that Unblocked is sitting there as a code reviewer. So don't feel badly if it sprays on your PR a little bit.

It depends on how much slop you send.

Oh, hell.

You got one hack of a username. Good work.

It's not hacking so much.

Is everyone good with this? I take it down? Okay. Cool. That is it.

Brennan, you're on top of the invites? Okay, cool.

There's a few more. I'm on to Chris. Chris has given two invites, so.

Oh, yeah.

That's okay. I'm going to send both. Don't worry.

Amazing.

That's just where I am in this list.

Did the Windows open?

It was a custom acronym.

Yes.

On email.

Were you able to make any pull-up?

Oh, forgot to mention a couple of things here, actually. Yeah.

Coming back live? Yeah. Good. Just a couple of things. So if you're looking for something to implement and starting with coming up with ideas and stuff, there is a set of sort of predefined issues that you can hack away on.

So you can just grab one of these, whack it into Clyde and see how it does when it's connected to the context engine.

The MCP server for Unblocked is here. So if you want instructions on how to wire this up to Claude Code or another agent, then you can grab it from the instructions from here.

Allright. So I'm at Lars. There's two more in here. So I'm still going, by the way, for those just adding their names.

Sorry, it says the event here is on 2026. I think the rest of them

what's going on, Brennan?

Oh, sorry. Just one of the usernames is invalid.

Oh, okay.

I'm just adding a new.

Allright. Kat, you should have an invite. Andrea, you're coming next.

I'm just two behind, it looks like.

Christopher, did you not get his invite yet?

No, no, I didn't.

Oh, that's weird. Let me double-check. You should have one, but.

So is it in both PRs?

Yeah, you should have an email. I'm up to like 31 of you. So hard work. Hopefully, you're good to use. It's like five clicks to add a member. I'm like, GitHub, we need time.

Agents need that for you.

I was going to say it. You should be able to use the CLI for this. What's going on?

We don't have the agents for it yet.

That'sright. Not yet. No, they keep putting it in my PR, and I don't want it there. Copilot's going to review for me. Very poorly, but it will review.

Andrea, you're good.

Yeah, for sure.

Hold on. Let me grab you the mic.

Hopefully, that's on.

Does it work? Yes. So I guess that context engine works very well for asynchronous agents so that you don't need to specify things on your keyboard because they can fetch what they need. That's one of the main uses, I guess.

And so it plays very well, I think, with agents like Copilot on GitHub. Do you see if you can share it, which agents are used most with Unblocked? Because on the wild, as developers with our laptops, I think Cloud Code is much more used than Copilot.

But maybe you see a different picture.

Okay. So I'm going to take this off the screen for a sec

and try to see if I can pull that up for you.

But the answer is yes, we do know roughly what that breakdown looks like. So let me grab that.

I have invited everyone who put up.

Oh, wait. There's a thread. Hold on.

Okay. I think this gives you kind of the rough picture.

Okay. So this is kind of the rough picture here.

Unfortunately, because of the way that this is, I should probably extend the screen. But I'll just step over here. So Claude Code is by far the most used.

This is the next one is Cursor. So that seems fairly obvious. This last one here is kind of a catch-all. But what's really interesting is that a lot of people use Claude Desktop, which was very unexpected, but this is the case.

And then VS Code and Codex account for a much smaller component. But yeah, it seems like everyone's using either Cursor or Claude Code.

I would have expected more of totally asynchronous agents, like something that people would just run from a PR. Okay, you can run Cloud from a PR, but it's less common. Maybe sometimes you use Copilot because it's built in.

Yeah, actually, this one here, Claude Code, may capture some of that traffic. So that's probably what you're seeing because people will wire up Claude Code in CI and do things like that.

Thanks.

No problem.

I've got a potentially dumb question. There's no dumb questions. Well, we'll see.

Actually, you know.

Judge you soon.

I had a teacher in grade three that used to tell me there are no dumb questions, only dumb people. Go on. I could be one of them.

From your point of view,right, you can use subagents from an exploratory standpoint. How does that plus memory plus just storing snippets of information that might be able to I'm thinking of the social graph that you just showed,right? Even in an organization that's like several thousand people, you would be able to store that in a very small file.

You would, as the graph that you showed. Oh, I see. The social graph component, yes. Yeah, it can be compacted. I'm trying to understand how this compares. What's the kind of USP compared to the exploratory agents and repeating that?

I see what you're saying. Okay.

So there are two components to that. One is that an exploratory agent would have to do this every time. So when it starts from ground zero, yes, it might be possible for it to reconstitute a sort of social graph hierarchy, but it would have to do two things in order to do that.

One is it would actually have to write code in order to constitute the graph. At least the way that agents are today or the way that the models are today, you wouldn't be able to just have it run basic tools around the organization and figure out the who's who.

It would have to write kind of like what that social graph algorithm is, run it, and then get the distillation out the back end. So at that point, you're basically getting close to that component anyways. So you short-circuit it and just run it and use it.

Maybe I should explain some of the motivation for that thing, actually. I realize now that I may not have done that effectively. Social graph is not just about conveying information about who the experts are. It's used within the context engine as a pivot point into more important context.

So understanding who the experts are in a particular code area acts as a jump point because another part of a context engine, which happens at the ingestion and processing layer, is distilling the we call it bottling the expert, but it's essentially distilling what that individual has worked on in the past, where they sit in the kind of hierarchy of the organization, the decisions that they've made based on Slack conversations that they've had, based on their PR comments, all this kind of stuff.

When you distill that down and you pass it to the agent, then what happens is, let's say that I'm a new employee and I'm coming to work on a particular area of code. There are a bunch of different ways of loading context for that code.

One is semantic search via vector search,right? So that's kind of layer one. Another layer is pre-built memories. And then the third layer is

unbottling the expert for that area of code. And getting that expert's learnings into context is a really powerful mechanism. It helps drive the rest of the retrieval in an agentic loop, and it helps the agent directionally, like where to go next.

Does that make sense?

I think everybody's in now.

Awesome. Okay. So I think if we're all in, then

the next thing here is, once I get this back up on the screen.

I'm still sending invites. I saw someone just put, "So please keep coming." But we can keep going.

Yep. So feel free to basically just fire this repo at your agent and get it to run it. If you literally just say to Claude Code, "Run this against my repo," be sure to give it a time range or a PR limit.

Otherwise, it'll go off the rails and take a really long time to finish. So just say, "Process the last 300 PRs or process up till September 2025," or something like that. There's enough information in the README that it should be able to just do it and just run it against your repo.

This morning I get cloned, said, "Read the README and make it happen, Claude." And it did.

Yeah.

Can I ask another question? What's your roadmap? If you have a roadmap, what's your plans for the coming year or something?

For Unblocked?

I've been speaking for this long.

Is it about Unblocked or about this sort of side project?

I mean, Unblocked.

Yeah.

So I mean, I've sort of alluded to this before, but where the puck is going is with fully autonomous agents. So we're very focused on making sure that autonomous agent flows are highly optimized. As I was saying at the beginning of the conversation, you cannot run those things effectively without very finely tuned context.

Yeah. Yeah. When you think about it, it's like agent swarms. It's going to be input-output. There will be an IOPS at some point on a context engine, and then shaping that token efficiency.

I read things like tracing. And you get runbooks out of those. Is that the path you're investing in, or what is it?

Yeah, we're.

Retrieval, what?

Are you talking specifically about incident management then?

Sorry?

Are you speaking specifically about incident management and that sort of thing?

Oh, I'm speaking about your I'm thinking actually more from a business perspective. How can we extract business knowledge that's really deeply embedded into systems nobody knows anymore? And some people think they know, but they don't know. And documents, human knowledge,right?

Tested knowledge.

Yep. So I mean, there's two ways of servicing that, either at the product level or through the context engine itself. And increasingly, what we see is that people leverage agents to do their work even at that level. So they'll go to Claude Code, they'll connect to the Unblocked context engine, they'll be like, "Do this thing for me."

And then the context engine will find all the things that it needs to do that task, and it'll surface that data. Yeah.

For us, that means the first near-term roadmap is API.

Yes.

It's like CLI.

CLI API.

API endpoints and the API layer to your good question earlier about how you expose this thing. It's like API, CLI, MCP, etc. And some first-class products that we saw in the QA experience in Slack or Dashboard. And then the code review is one of our first-class apps as well.

But when you use these in your org, it's just good to basically make sure that that access is available to obviously humans and agents. Most are it's over an API some way.

Cool. I'm going to lift this off again. Hopefully, people will start submitting some PRs and we can.

Yeah, rip some PRs. That's a good question. Test the agent.

Once you're in that GitHub org, let me actually repost it in the Slack channel because that link will.

So this org will stay up until the end of the week, at which point we'll basically bring it down and release this as open source. And everyone that contributes obviously is going to get credited, so your name will be on it.

Should we set up the repo locally and only then start doing it, or what's the first thing we should do, basically? So I've just finished setting up.

Yeah, just clone the repo. The easiest thing to do is to take an agent like Claude and point it at just launch it from that repo, from that directory, and just say, "Please bootstrap and launch this product," and away we'll go.

If you guys run into any kind of technical things, we're here, obviously. Yeah.

Hold on. Let's get you the mic. Oh, you got it?

I've got a lapel now, so.

Awesome. Cool. Yeah.

Can you hear me? Yeah, perfect. So on the slide where you had the performance and you guys were like 80% and without Unblocked, it was 20%. And now I see that, well, you are basically hooking up Unblocked to Cloud Code.

So in a way, is it a fair comparison to say, "I will use vanilla Cloud Code with access to the MCP and to the skills, and then I will use Cloud Code hooked with Unblocked with the same MCPs and the same skills."

And here you can do the performance comparison, and here you still have a lot of alpha from, I guess, whatever you are cooking inside Unblocked. Was it the comparison that was done, or was it done without was it done with a vanilla Cloud Code but without context?

No, it was done with MCP servers like GitHub and Slack wired up.

I see. We basically got parity with all the MCP servers of every SaaS vendor in one. It was like vanilla Claude, all MCPs, and the other one was Claude with Unblocked only, and then do the task.

Same context, like the same context files?

Same prompt.

And same access.

Yeah. It's pretty fun. Yeah. Oh, thank you.

Maybe two questions. So one is, I see that a lot of these social graphs are built with the traditional network kind of calculation and the statistical aspects of networks. Is this the approach that you began with and it already worked the best, or did you because most of the memory systems, they work more on filtering out episodic memory, something else, something else, something else, and this is like really scoring, really nice scoring system.

That's the first question. Is it also with the Unblocked? A second question, you mentioned that it works with Teams, Microsoft environment. I wonder what differences did you observe between building social graphs for different environments? Because on GitHub, I imagine it's very different than on SharePoint, Teams, etc., etc.

Is it also like this network stats-based, or is it something different?

So I mean, our first implementation was incredibly naive,right? It was just using the numbers of PR contributions and comparing that directly with the number of PRs reviewed by each person. So just a simple numbers game. That didn't produce accurate team clusters.

So then we got onto the algorithms that you see here. Unblocked does a little bit more than this. So this is kind of like a middle road. Another strategy that Unblocked uses is

experts by vector clusters. So when we ingest the source code and vectorize it, we understand who the most contributors are for that piece of source code. So when we look up individuals, we can see what they've been working on and what the clusters and proximity are, and then relate people based on their cluster proximity.

So that's more of like an ML-type approach. And then there's a final layer, which is

a sort of AI LLM-heavy layer that does distillations of a whole bunch of different context elements, things that people have worked on in the past, conversations that they've been having in Slack. And then when you take all that and you weigh it against the procedurally generated graph, you get a much more accurate distillation.

This one here, you'll notice some people will get pulled into team clusters that you know are operating across many different teams, for example, and this won't account for that.

And the differences between different environments, like GitHub versus Microsoft versus Slack,

do you need different algorithms, different weights, let's say, that you extract there? I mean, I don't want to take out all these.

No, no. This algorithm is purely SEM-based. So the algorithms for, you'reright, like Slack, Teams, they're quite a bit different because you don't have these review points. So then it becomes, who's the most active in particular channels? And then you need a distillation or a summary of what that channel is about, and you need to vectorize that, and then you need to score it against the most frequent contributors.

But it's not enough. You have to relate that back to the SEM data in order to figure out who the real experts are. One of the problems that I've personally experienced in some organizations I've worked at is that you get the noisy junior engineer,right?

So they're very noisy, they love to talk, but the signal-to-noise ratio is not great. And just because someone's not saying a lot of things doesn't mean that their messages are not impactful. So part of this game is about assessing the impact of when people say certain things, how does that relate to the PRs that get spawned off as a consequence?

How many of those PRs get merged, that sort of thing. Yeah.

I sent a message to Slack, but I don't think there'sright access to the repo.

Oh, is there not?

There should be.

Okay. Check that.

Well, you should be able to open a pull request. You can't push to main. So if that's the situation, but I mean, we'll check.

Yeah, you should be able to create a branch.

Oh, no. He can't fork the repo either.

Oh, yeah, forks might be disabled.

This will be open source at the end of the week, and all your contributions will be on it.

What's really fun is using that social graph tool later against your own repo and showing your team. Yeah. Oh, sorry. Oh, I'll do it. Yeah.

Come out.

I like that Unblocked tried to answer you for that question.

Oh.

You see that? The Slack auto response?

Yeah, I like that.

Sorry. Are you in here?

That's a camera. Sorry.

Oh, it's okay. I was just

Oh, okay. Let me check to see that should not be the case.

Okay.

Let me know if you still need a GitHub invite.

Just check the members. I think there might, yeah, there might be an issue here. Just a second.

Oh, these were direct assignments. So I think we have to pull people into the project because they're not org assigned.

Oh, GitHub. I love you.

Zero nines of uptime.

Yeah. We'll fix this one here.

Yeah. I think slam everybody in.

That's crazy.

Oh, come on.

You got enough?

I'm trying to, because now we just need to add people.

Good saves. Good collaborators.

And then.

Just start adding people here. It's super beautiful.

Unblocked. You all haveright access. It is the name of the company.

Just validate that for us, if you would.

Yeah, please. Let me know. Oh.

Perfect.

Allright.

Allright. We're getting real PRs now. There we go.

Nice. Nice.

Hell yeah.

Now let's do fun things.

Nice. Looks good to me.

What?

I think we have our first approved PR.

I'm just sending ridiculous chats to Unblocked so you can see it try to answer questions in Slack as PRs come up.

I'm going to see what it says about this.

Ask it to.

It's like, oh, let me think about it.

Oh, did you ask it about the PR?

Yeah, but the PR, I think you accepted, so we'll see what happens.

Yeah. I mean, it did approve it, so Unblocked was like.

Unblocked's like, this looks good to me, man.

Unblocked was down.

Only visible to you. Oh, no.

Oh, it was such a good answer, though.

Nice PR. Good job, Unblocked. Great answer.

Yep.

Oh, yeah, yeah. I'll put it back up. I'll put it back up. One sec.

Where did it go? Actually, I lost the

okay, thanks.

It was over here.

Oh. Yeah.

We had a sneak peek of how it looks in the SaaS inside. I mean, how you can handle the sources or maybe see something on the app.

Yeah, for sure. Yeah.

If that's possible.

Oh, yeah, yeah. Yeah, of course. We were focused on you building, but yeah.

Sorry, what am I supposed to do?

Oh, no, it's okay. I mean, let's go.

Oh, sorry. Oh, sorry. So this thing that I showed before, it is the project that exists in that repo. Oh, so the idea is think about features that you want to add or things that you want to fix or new components, and then just hack away at it and submit a PR and.

Sorry, I didn't get that part.

Yeah, I'm all good. Do you want to open up a terminal session and show the MCP?

Oh, sure. Yeah.

Because people can obviously use it, but they don't have all our source.

Yeah. Because explain what I'm coming from. I'm a consultant and really like the idea, but if I wanted to try it or propose it to a client, I cannot show the context graph or maybe get an idea of how it works in internals.

Because I really like the idea. This is a big problem.

Yeah.

Without seeing the product, it's hard.

Well, I mean, one thing that you could do if you're visiting clients is you can ask them if they run the tool on their

repo, and then it will generate this result for them so they can see on their own project what the value is,right?

I think, Peter, I think he's just asking about our product specifically, not this.

Oh, Unblocked. You're asking about Unblocked.

My bad, man.

We're driving this way.

Yeah. Sorry, sorry. Single-track mind. Okay. So your question is, how can you demonstrate the value of Unblocked to customers or?

See the value.

Exactly. I mean, I guess that one big question for customers is how to use this information and be able, for example, if there's a conflict of intent, maybe to data source and conflict information. And how to do that.

Sorry. You can make conflicts emerge in your app, and then there is the compliance layer, which is very interesting for corporate clients. I was thinking how this is translated to a UX because

many people understand it's mainly for coding. And whether this is for technical people or maybe people overseeing some engineers or the engineer itself, I mean, just see how your platform works. But if it is out of context, I mean, it's okay.

No, no, that's totally fine. So this dashboard is kind of like the

sort of front-end customer interface to the product. So you come in here and you can ask any question about your code base or your organization and get an answer for it here. This isright now attached to sorry, I lost my cursor.

This is attached to this test org that we have, but I could use it against Unblocked and I could say,

I have a little hot thing here that I can show.

Sorry.

Oops. Yeah. So the SourceMark engine is an internal component that we use to track source code changes through time, including where

changes move between files and so on. So as a demonstration, you can show off I mean, you can book your customers into a demo with us and we can demonstrate this, or you can wire it up to your own organization and demonstrate this flow to customers and try to find use cases where data sources conflict and demonstrate that.

The challenge with context engines is that it's really hard to demonstrate the value to someone without actually wiring it up. So there is a little bit of overhead there where people have to connect it to all their integrations.

Now, the good thing is Unblocked has a free enterprise trial period, so people can try out the products in its fullest form before

paying for it. Yeah.

So if some of that information is incorrect, you can just reply in the chatbot or flag it in the references.

Yeah, exactly. Yeah.

So you can reply here or you can say not helpful and explain why, and then it will distill it for the next round.

So it will adjust some weights or confidence scores internally.

Well, internally, what it does is it constructs task memory. So it looks for those kind of repeated signals. And this is actually where the experts graph comes in. It's used a lot. The experts graph provides weight. So when an expert comes in and says that's not correct, it's going to get some more weight and distill a memory for it.

If it's just a new engineer that says that's notright, then that's not really a trustworthy source yet. So you have to have a trustworthy source to base that on. Does that make sense?

Yeah, it makes a lot of sense. It's like social network somehow.

Exactly. Yeah.

Thanks.

No problem.

Cool.

What does your memory source look like?

Oh, under the hood? Well, when it's presented to the AI, it's presented as files. But under the hood, we store it in database tables and stuff. The memories are constituted from a bunch of different sources. So they're not just flat file-based.

The whole memory construct will be hydrated at runtime.

And what you just give your agent tools to query your database based on whatever pressure and users are.

Yeah. So yes, there are a bunch of tools for data retrieval. For memory specifically, you can't really leave it up to the agent to do memory hydration because that's kind of like part of the seed context. In order to get the agent to go in theright direction, you have to seed it with the appropriate data.

And experts context is a good jump-off point for the agent. So yeah.

Thank you.

Yep.

Is there any official benchmark that kind of track the type of value you try to bring? Because I feel like it's not exactly coding. Well, it is, but yeah, I'm curious if there's any public things that you're tracking yourself against.

So we do have some internal benchmarks. You'reright, it's a little bit squishy.

Have you heard Boris Cherny talk at Claude Code?

The creator of Claude Code?

The creator of Claude Code. Yeah. So he did this interview where they were talking about how they measure success for Claude Code internally. This may have changed because there's a lot of benchmarks now that they have. They have the shit talk benchmark.

You guys have probably seen that one. But what that really distills down to is vibes. And so the most important thing in systems like this is to capture sentiment. And so if your sentiment is trending upwards, then that's a good thing.

Our sentimentright now is on a scale of minus 100 to 100, somewhere around 60 score. So on a normalized scale, that's like 0.75 to 0.8.

So the vibe would be captured by something like maybe less back and forth on the PRs or maybe, I don't know, you having less back and forth with Claude to get your stuff done.

Yeah. So the vibes are they're people satisfied,right? So satisfaction can come from a lot of different sources, and dissatisfaction can come from a lot of different sources. So the way to think about that is that it encodes all of those things.

But you can capture specific metrics, and we do, how long things take. And we're actually currently working really hard to bring the response times down because

even though agents are here's the interesting thing. As we move towards a more autonomous universe, response times for MCP servers are actually less and less important. The more important thing is that they get the answer absolutely bang on.

And the reason is because the amount of time that a context engine spends collecting all that information and distilling it is a microcosm of what the full task takes to implement and to traverse. So if you can spend a little bit more time and cut the implementation down by like 60, 70, 80 percent, that's a huge win.

Right. And go ahead.

Sorry, very small follow-up. Actually, I'm curious, do you have any rough numbers on how much time does it spend retrieving context versus executing the task? To your point, is it 10%right now, 90%, or is it I have no idea.

I mean, I have my own experience.

Yeah, agent context collection is probably close to that number. It's like 90%. The actual code writing part is really, really fast. You can even just watch what an agent is doing. When it writes the code, that output tokens are, by the way, the thing that drags down the performance.

Everyone used to think it was input tokens. We've run tons of experience with this. You can bring the input token size up, and time to first output token now is pretty good. It's pretty highly optimized. The thing that really impacts performance is output tokens.

So you have to be judicious with the way that you collect and supply context back to the agent so that it remains tight on its output loops as well.

For one benchmark that Peter mentioned in the talk, we gave an ambitious task because obviously it's prompt-dependent, how much time you're adding with a context engine. But the ambitious task we gave was to implement the new adaptive thinking mode in Anthropic's toolchain when they introduced that, which, as mentioned, it went from a 25-minute wall clock time with Unblocked, with a context engine.

The other case without was two and a half hours. It was two hours and 25 minutes. But the main reason for that was we gave it all the data, we ran the prompt, and then its first output was totally wrong.

So the human had to loop again and be like, no, no, this, this, this. And the next output was wrong and the next output. So once you do four loops, you have a two and a half hour wall clock time versus obviously the 25-minute when it did not need that, when there's no corrections required.

So as mentioned, think of it as a waterfall. The more high-quality, correct, high-signal context you have upfront, the better every single thing the agent's going to do until it says it's done, whether it got itright or not. Yeah.

You

also mentioned that the token usage on tool calls and just information search really decreased. So I know that a lot of these tools that provide or aggregators for tool use, they have insane token usage. So maybe have some estimations on how let's say I need a Slack conversation, some summary from one conversation to another, or how people interact there would be like 60K tokens on Composio.

I wonder how many tokens it would be like using Unblocked.

Yeah. Lower. We're still very vibes there. It's hard to get real data from other customers or people in the market. But again, with that same I'm going to keep talking to the same task as easy. That one went from 21 million token total usage to 10 million token with the context engine.

So a part of that, though, is because you didn't have to doom loop. So of course, that increased a lot of the tokens expense. So we did drop it by 50% on a large task. Again, obviously, if you're like, yo, send her a div, you're not going to get a lot of gain.

It's probably in the training data. But yeah, any feature fix. So a lot of again, a lot of what people are putting through Unblocked or what an engineer is doing every day, it's very rare that you're doing a task that's so, I don't know, minor that I mean, then again, I've asked Claude to do Git push.

So I'm not the only one, I bet. I was like, you do it. It's like, why did that cost me 30 cents? I don't know.

It'sright up in the pipeline.

Yeah. I did all the effort to put my GBG keys in theright place. So I'm like Claude, go.

Any more questions while y'all ship? Any confusion, anything I can unblock for you? It's my purpose in life.

Sorry, you may have answered this question already. So are you using knowledge-based RAG in Unblocked, or what exactly is the tech that you are surfacing?

Oh, so many things. I can come talk to you at the side. I'll take my mic off. I'm just going to answer that question.

Sure.

That was just the start.

See, incredibly talented individuals.

Oh, it's real time, basically. So I guess there's two parts to that question. One is how much or how frequently Unblocked updates the data on the backend. So it's real time for many of the integrations and then on a cron job for others.

Because for those particular integrations, they don't have webhooks, basically. Yeah. But so that means that rebuilding the graph data has to happen on a very frequent basis.

No, it's incremental. So our social graph build algorithm has an incremental component to it. So we don't have to rerun the whole thing. But also, social graphs are less sensitive to frequent changes in data because it's unlikely that a single change is going to make a huge impact on the experts graph unless your organization is brand new.

So for

yeah.

Yes. Yeah. So as an example, we do best practices distillation on a much lower cadence, basically week by week because yeah, it just doesn't change that much. Yeah.

Well, oh yeah. Repeat your question. That's a good question. So I want to make sure we get that one done.

In

terms of customer privacy, data retention, kind of from my point of view, I'm thinking of enterprise SaaS or even on-premise type deployments, which I'm not suggesting that you I'm just thinking of that customer kind of modality. Yeah. Do you get pushback?

How do they feel about you holding data? It's another processor in the loop.

Well, so the privacy discussions happen at the organizational level. So

we don't actually run into a lot of friction. There are definitely environments like in government and at banks that have super sensitive needs. And so for those needs, we have an on-prem solution. But it's definitely not the path that I would recommend, like staying cloud-based.

We have very large enterprise organizations that are entirely cloud-based, like fully cloud-based.

The secret sauce is kind of less encoded in source code now and more encoded in the reasoning. So organizations tend to be a little bit more sensitive around things like Slack data, for instance. But the way that we store data, like we have a whole white paper about how we protect customer data, and it's never been a problem.

Yeah.

Pardon me? Yes. We do have an on-prem solution, but as I say, it's not the recommended approach. But for sensitive environments, for sure. Yeah.

I missed why it wasn't recommended. Sorry.

Oh, why it's not recommended? Well, the cloud-based integrations

get updated more frequently, and so there's software patches. It's a little bit harder to maintain within an organization. There's one customer, it's a bank, where administering the platform becomes quite difficult because they have network isolation. And so now one of us has to sit within that network and administer the platform, or we have to train individuals within the company to administer the platform.

So it's more of a maintenance and hand-holding exercise.

But yeah.

Yeah. Exactly. That's exactlyright.

Yeah. Thank you. Thanks for coming.