Demand-Driven Context: A Methodology for Coherent Knowledge Bases Through Agent Failure

Uh, thank you. Maybe we can get started. First of all, thank you so much for coming for the workshop, especially ones who didn't get the seat. I promise you, I'll do my best to make it entertaining, especially for you who are sitting here.

Thank you so much.

You're the volunteers,right?

Yeah. Actually, it makes sense. So now I know, like, why the tickets got sold out,right? Which workshop actually sold out the tickets?

So, uh, let's start with my introduction. So I'm Raj. I work as a staff software engineer at IKEA. I work for a domain called Delivery and Services. Basically, we are like almost more than 100 engineers and six FedEx teams altogether.

It's like a mini company within the company itself. I'm very interested in architecture, neuroscience, and linguistics, and now AI. So if anyone has some cool projects, because everybody is building cool projects these days, please find me after this meeting.

So, quick pulse check with the audience. Who is visiting London for the first time?

Okay, cool. Welcome to London. Who is here from the engineering background?

Also with vibe coding and prototyping? No, all of them. Okay. So, who actively uses agents like Copilot? Okay, this is going to be tough for me then. So, extensions? Okay, so everybody is pro. Okay, fine.

Not so much as I wanted to.

Okay, so much tension now.

So, you're going to sit here in this hot room for more than an hour. So first, I will give a bit of introduction of what I'm going to present today. It's basically on agent and the context management. I divided it into three parts.

One is the situation, which all of you already know. So I'll keep it tight and short, like five minutes. Then I will talk about the problem. This is where I'll spend a bit more time on the slides because I think, like, nobody is actually seriously looking into the problem, and I want to bring it up.

Then less slides and more into some kind of a hands-on, how the actual demand-driven context actually works. All good?

Okay. So let's start with the first one. How many have seen this movie? Memento.

Oh, yeah.

Okay. Okay. Okay, cool. So I'll give the gist of the movie. So this guy is very skilled, very talented. The only problem he has is he can't hold memory more than 15 minutes. So every 15 minutes, he has to take his notebook, watch his tattoos that he put on his phone, and figure out, okay, what I was doing before the 15 minutes.

And he does it again and again. If you relate to the AI and AI agents things and all, it actually fits exactly how the movie is and how the agents areright now. If you go and watch this movie, you don't need to watch YouTube or blogs to understand agents and MCP.

This movie actually tells you about everything, literally. And as this guy has a memory problem, in the same way, the AI that we got introduced a couple of years ago is very good with reasoning, computation, code generation. It's benchmarked as, like, above the par.

The only problem is the institutional knowledge,right? The domain knowledge that you have, that's the only thing we have to be a bit problematic. So, from AI to agent, if you look at the evolution, it exploded. So it started from prompt engineering first of all.

Then there was RAGs, now MCPs, then multi-agent. Now it is deep agents. I recently found out, like, using Replit actually, you can build a full-stack app in 10 minutes. That means by the time you make the instant noodles, you already have a million-dollar app already working on your laptop.

So we got it to this point, like, it's extraordinarily good.

Now, that's AI and agents. So let's talk about enterprise AI.

Okay. I don't know how many of you have this question, but most of the enterprises, I see the question is, okay, AI is pretty smart. It's doing code generation, full-stack apps, reviewing your PRs, doing incident management, all those things,right?

So if it's doing that much, why is the Jira tickets or epics not moving on the dashboard,right? Why do I don't see the delivery actually? So everybody is speaking about, look at, like, three minutes, everything is ready. Yeah, okay, fine.

Why are my Jira epics not moving? Because that defines the business delivery, and that defines the return of investment,right? And as you see, like, it's from the McKinsey this year, like, 88% of all companies use AI, but they only see, like, 6% of value creation.

Okay. So I think this is the problem that we have. I have four Jira tickets, different ones, sample ones. And you can see the green ones that I have marked as basically which LLMs are already trained on, like API standards or, like, things where they already know it's a general knowledge,right?

That's fine. Those tasks from the ticket, it can pick up and it can do it. Now, there are second part orange ones, which we have to teach them actually. So you have, you know this, but do this in this way.

So all this kind of an orange color things will fit into, you know, your agent extension, like skills or, like,

but the red ones, that's what the institutional knowledge is, which sits within the company and within the people. So unless if it picks a task, if it picks a ticket, it has to fulfill all of them. It is so good with green ones and orange ones, but it struggles with the red one, with the institutional knowledge.

And what I believe is,right now, the coding agents are getting so, so better. I feel like if there is an AGI coming, the first AGI will be a coding agent for sure.

So, to fix the giving the institutional knowledge to the agents, so we have an industry solution already. So this is basically your return of investment on AI pipeline will look like,right? So you have LLM model quality, you have agents, and you have agent harness, and your institutional knowledge sits under Confluence, Jira, SharePoint, GitHub, all those things.

And basically, retrieval layer is what industry is telling us will fix that issue. So you build that retrieval layer, and then it will fetch all those things and give it to an agent, and the agent should be able to do it,right?

So basically,

so 40%

factual accuracy can be achieved through RAG or, like, knowledge graphs actually, but with a documented knowledge base.

Now, basically, if you build a retrieval layer, it has to work,right? Now, let me ask you a question. How many have you built retrieval layer things, like RAGs and MCPs? Okay, cool, all of them. Okay, now the question is, how many did you build?

How many MCPs did you build? How many have you built more than 20 MCP servers? Okay. Okay, so nobody beats my record then. So what I see is, mostly in the enterprise organizations and all, people are building, like, 10 to 15 or, like, 20 MCP servers or, like, RAGs, knowledge graphs on top of their institutional knowledge and the agent,right?

So the assumption is, if I can build all those MCP servers and give that agent, I don't need to work anything, like, it will do. But the thing is, when you plug in those MCP servers, basically all this data coming out is mostly undeterministic, it's unreliable, and it's untested,right?

So especially in engineering, nobody does evals actually. It's more like a data machine learning concept, but we don't do evals. So for me, if you plug, sorry,

if you plug an MCP or, like, a RAG and all, we see whether the output is coming or not, rather than is it really valuable actually, is it really solving the problem or not. That's the main problem that I see.

I'm not saying pointing other people because I was that person. I was like, okay, let me build all MCP servers, plug in my institutional knowledge. I'm going to prove that point that agents can semi-autonomously continue and fill those Jira tickets and finish it,right?

But every time when I build those MCP servers, 10%, 20%, 30% of the time, it was accurate, but the rest of the time, I was doing the data entry job for them actually. So I was filling the gaps, asking the questions.

So basically, I'm doing more work than actually doing less work. So I think this is the main problem. And I actually was in this fourth stage where I literally started to write the domain context with handwritten actually. So, okay, let me write everything and prove the point, but I got really exhausted of doing it.

Okay. So how, I don't know how many can you relate with this pie chart, but most of the enterprise, the institutional knowledge is kind of something like this. So 20%, if you see, it's outdated. 20%, it's unreliable. 20%, 10% is always duplicated with different places.

And the major problem is 40% of the knowledge is always tribal knowledge, which means people know how things work. So it's never documented actually. So in this situation of an enterprise, and you build, like, 100 MCP servers and plug into that monolith, it doesn't matter how many you build, it won't work because basically your whole institutional knowledge is a monolith.

I think, like, because you're all from the engineering background, so you already know the transformation of monolith legacy system to microservices,right? So in the same way, unless we break down that monolith knowledge base into some kind of a context blocks which are useful for agents, then only we can actually make it useful for them, for the agents, and actually make them semi-autonomously can actually do the tasks.

So that said, we are going to talk in this workshop mostly on that monolith, how to break it, what is the approach to break it, and how it is useful once we break it. And this is a job we need to do because the LLM providers will focus on the LLM model quality.

The agents will focus on the harness things. And there is a big

retrieval market of 9 billion. They're focusing on retrieval. But nobody is going to come to your company and fix your knowledge base. You have to fix it yourself,right? So how can we do it?

Okay. So the demand-driven context is what the, as a solution, I was trying to propose,right? So basically, if I have to give an abstract of it, what it is, like, we have monolith services, and we have this process of breaking them to microservices.

We have waterfall model, which we transformed into agile. In the same way, when you have a monolith of institutional knowledge, how do you transform into a context blocks using an approach? So this is an approach of how we can do it.

Before starting, just not an idea. So we already tried with some datasets and tried to prove this approach works. And in March, we have published a preprint in RXV. So if anyone is interested in reading academic papers, you can find it with the demand-driven context.

Or, like, I can also give you a link after the workshop.

Okay.

So how does it work actually?

When we are giving institutional knowledge to agents, basically what we are trying to do is we are trying to do a push strategy,right? So we build everything and we push it toward. So in this approach, it's more pull approach, which means, for example, let's say a new joinee has joined your company,right?

How do you onboard a person? So you onboard them for one or two days. You give some initial orientation, and then you tell them, like, okay, these are the Confluence links, these are the GitHub, this is some kind of a documentation you have to follow, things and all.

Then you just assign a task to the person. But you're not going to tell, okay, go and get graduated on this knowledge and come back, then I will give you work,right? So you'll just assign the work item. And when you assign the work item, the person will start asking questions, fill the gaps.

If the person is very much into documentation, he will also fill the documentation for you. He gradually gets his knowledge of the institutional knowledge,right? In the same way, we don't push all the knowledge to the agent, rather than we start giving problems to agents, like work items, and let them actually pull the information from us.

And once you pull the information, also ask them to document it

in a better way, rather than in a monolithic structure. So if you, so that's the four layers. So you have a monolith, a framework, and it actually pulls and actually creates a good, better context blocks. You can actually relate it more into a legacy monolith to microservices directly if you have to have a knowledge of it.

So this is how it works. So this is one cycle of how

we solve a problem to an agent, and in the first attempt, the agent will fail to do it. So it will say, you know what, you gave me a problem, but most of the documentation, I couldn't be able to find anything, I couldn't be able to do it.

Then these are the things I need to do to finish this task, and it gives a checklist of things. So we fulfill the checklist, like, we fulfill this checklist. So once it is given, the problem is solved, it will take that knowledge, and also it will update, that means curate the knowledge in a particular place so that it can reuse or, like, other agents can also reuse.

This is one cycle. So the idea is, if we can do it in multiple sessions with multiple problems, so it will gradually curate your knowledge, monolithic knowledge base, and also document it for you.

You can also relate it to TDD. So how many are do TDD or, like, nobody hates TDD,right? Before I-.

Yeah.

Okay. Okay. So in the same way,right, in a TDD approach, what we do, we just write the failed test cases. We don't build the product first of all. We just write the failed test cases. We see what is the code that is missing for the failed test case to pass, and we just give that code, and we gradually build a product based on the failed test cases.

In the same way, we give problems that agent will definitely fail, and we gradually fill those gaps, and at a certain point, it becomes semi-autonomous with good institutional knowledge already.

Okay.

So I think I can jump into some kind of a demo already. So I will use terminal, so don't hate me. I think, like, all of you are from engineering background, so I think, like, you'll like terminal. Let me switch to terminal.

Okay.

Okay. So on the far left,right, what you see is how under the hood it works actually. So when you have given a problem, how does the agent feel fail? How does it demand for the knowledge that the problem has to be solved?

And a human, like a domain expert and all, fills those gaps, and then it will curate a new knowledge base for you, which is much better. And then the agent succeeds, and you repeat on the next problems. So that is how one cycle things and all.

So how it can be implemented, it can be implemented using any agent. There is no, it can be implemented on Claude or Copilot because it's an approach you can do in any way you want. At work, I use Copilot.

So I implemented this using Copilot, but because everybody, I believe, loves more Claude code, so I created this demo with Claude code. And you can see it's just a combination of skills, rules, agents, and hooks, and some kind of a place to save the knowledge base.

On the middle pane, what you're seeing on the top is your monolith, basically. This is a representation of your Confluence, Slack, GitHub, and all, but just for the sake of demo, I just put some flat files that look like them.

So that is how your monolith knowledge base will look like. On the down, what you're seeing is on a live, so when it's solving a problem, how it is actually adding the new knowledge to it. So this is how it is.

So let me, so what I'm going to do is I'm going to go to an agent.

Okay. I'm going to basically give an incident problem to do the root cause analysis,right? So, okay, what I did is, you remember in the previous slide, there is a Jira ticket sample that I showed,right? It's a combination of knowledge that is documented, not documented, things and all.

So this incident also represents the same kind of combination. So there is some knowledge that is documented already on your monolith. Some, it is not there or outdated or things like that. And most of it, it doesn't, wouldn't be able to find because it's never written down actually.

So when I gave this problem, so it uses those skills that I have developed using this approach, and it will try to actually first go to your monolith actually on the knowledge base and try to find information on what is there.

So

think about it like this. So first part is retrieval. That means it's already doing what RAG and MCP is doing, first part. But what else it is doing is after it fetches the data, what it will do with the data actually.

So that is a missing part. For example, when you give new Confluence links to a new employee, the employee goes there, looks into it, but doesn't find information. But he doesn't stop there actually. He continues asking questions to solve the problem, then just adding more knowledge and things and all,right?

Those are the next steps missingright now. We just stop at retrieval. So this is the next three steps it does actually. So you can see the confidence score is almost one to five because it says

these are the particular terminologies. I don't understand actually these terminologies. And this business logic is not needed. So one thing you need to look at here is whatever it has said, this is the undocumented information. That means it was never written down.

So unless you don't do this way, you will never know what is not documented. For example, if somebody says, like, okay, there is documentation missing, we need to write. Okay, what do you want me to write actually? There is so much in the people's head, I can't write so much,right?

Somehow it has to surface. So when you give a problem, it actually surfaces what is not documented. And it tells me, okay, this is missing, I need to have new information there. So what I will do is, so it does all the three steps.

So then what I will do is I already have a pre-prepared answer.

Very high-level prepared answer I gave it to it of, like, what is the missing information. So okay, this is the missing information you asked me to solve this problem. Can you solve this problem now?

I didn't expect this one. Okay. Notifications is yes. I'll just say yes.

What if it asks also what fictional name should I use for it?

Ah, okay. I didn't see it. See, when I did the test run it didn't ask me the questions. Let's see. It knows it is a demo and-.

It's not for.

I tried to. Okay.

No, it is already, what it is doing is already, so you can see on the live it is already adding the entities that the new knowledge base has come into the place.

So the knowledge base is managed as a file system,right? Or as a system of files?

For the demo, I'm just showing it as a file system, but it's basically your MCP servers. The data will stay in Confluence, Slack, or things. You can just plug in and use the same MCP servers or RAG and all.

So it no need to be a flat file.

So you treat this as your. Systems layer for this agent,right? It's like a mem, do you use any, like, a memory tool for this?

Yeah. I will show you on the next slide, like, how I'm going to save.

Okay. So it started from 56 entities or something with this one,right? Now, one problem actually surfaced six entities that are never been documented. And when I gave that information to it, it is able to actually discover, curate another five or six new entities that were never documented.

So it does discovery of the gaps. It also gets information from me and also stores information, new information, and all.

This is one. Okay. Next, let's see.

This is a busy window. I tried to actually do things, but it didn't work out. Okay. So what you are seeing on the window is, like, 14 incidents. You have seen one problem that I solved with an agent,right, the communication.

What if I took, like, 14 incidents and I just go and have 14 cycles of this thing and how it does? So if you see on the left side, it was the first incident,right? Soright now it has 1.5 confidence and everything is critical.

So basically, nothing is documented. So everything is critical, high, the data is missing. So I started giving answers on the first incident, then I repeated for the same second and third and continuously for, like, 14 incidents. But on the 14 incidents, it basically actually able to go to a confidence level of 4.4 because first it discovered on every incident, it got the list of answers for me, and also it documented everything for me.

So it gradually from 1.4 to almost, like, five range of knowledge, it improved. So if you look at the traditional way, in traditional way, what we do is we solve all the context problem,right? We have to deal with it first, then I have to give it to agent.

In this one, we are moving agent from a consumer to a knowledge manager. So you just don't consume from me. I'm going to tell you, but the whole knowledge management is also your job and you have to do it for me.

Okay. I think we can get back to the slides a bit. Okay. So what we have seen is I have run one cycle and also I have shown how it looked like when I run in, like, 15 or 16 different cycles,right?

But if you want to do it manually, it would be really painful because I tried it. After 15 cycles, like, nobody would like to actually sit with an agent and, you know, you have an incident, but you won't be sit with your agent and keep actually asking questions and telling you about your problems,right?

So that is super painful.

But the thing is, we can automate this process. So this is where actually it's really good and gets interesting. So here is the thing.

We already have all the work items,right? We have Jira, we have incidents, we have customer support tickets, like that. All those kind of work items already there,right, sitting in the archive. So why can't we take them and actually use the framework and validate across your monolith database, run an automation, and see actually what is the state of your context actuallyright now?

Okay. Let me show you how it looks like. So rather than actually doing it manually, at a scale, if we do this approach, so how does it look like? So the demo that you're seeing is almost like everything is preset.

For example, I have the demo. I have, like, a platform operations agent and I'm saying, okay, these are the recent incidents. Let's say I have 20 recent past incidents. I have, like, an MD file or a JSON file,right?

It has all the details of description of it, things and all, comments and everything. And the rest of the files are your knowledge base. So it's a file system, but you can also actually connect with the same way the Confluence and things and all, just for a demo purpose and just showing it as a flat files.

Now, what I'm trying to do is I'm going to take all those incidents and validate each incident across my knowledge base and ask the agent, okay, tell me how much of the document is good, how much of the documentation is I can't trust it or, like, old or outdated, and how much is actually missing, not documented as per those incidents.

So let me

run it. Okay. It will take some time. So it will take three steps actually. So one is it generates probes, which means a basic test it will write to actually test your knowledge. Then it will run those tests and then analyze the gaps actually.

Okay.

It's a little bit hot in the room actually.

For identifying the gaps, it's just like a clever prompt, I imagine.

Okay. For example, let's say you have an incident called the notification service is not sending customer messages to

SMS service,right? So the notification service then you mention that the agent sees, is there a documentation related notification service? So it doesn't find. That means you never wrote a documentation on notification service. I do understand what is a customer SMSs, things and all.

The customer notification service, when you mention, it's a gap because it's never documented. Or it takes the customer notification service, goes to Confluence and sees, like, the documentation, how old it is. If it says, like, it's like one year old, it will tell you, look, I looked into it, it's like one year old.

I don't know whether I need to trust this documentation or not. Or, like, incomplete documentation. So if you see it scored each incident it took it and it looked at all the knowledge base that you have connected and have a consolidated list of, like, scoring of, like, okay, partially the agents can handle the basic edge cases of the incidents that you give because your knowledge base is not complete.

And it will show you how much of the tribal knowledge is missing, system information, business process, what are actually you're missing from your institutional knowledge when whatever is not documented. These are the probes.

And it will also identify what is critical and what is high. This is really important because let's say there is some kind of an example of notification service which I mentioned,right? It is repeatedly appearing in, like, 20 instances.

This

is the first that you need to fix as per your documentation. So it will also help us actually understand when you're

breaking down your knowledge base, you need to understand what is critical actually, what I need to focus on first, what makes value for me. So it will organize into critical, high, medium.

So this is what, like, I showed the flat files, but you can also connect it to the various data sources that we have. So the step one is basically what it does is demand extraction. That means every incident it will extract the checklist of information, what is missing.

On the second step is what it will consolidate everything, what is missing. So it will create, like, systems and APIs and all, and how many are clean, how many are stale, which is incomplete, what is entirely missing, something is tribal, those kind of classification it will also do.

And it will create a Kanban board for you. So what happens is, so if you want to fix your institutional knowledge base, basically you just, just like Jira tickets, we finish it, we actually has to document these missing pieces and all.

And the moment you started to, so it also saves in the context lake, so it also has to build its own knowledge base. And you can see the performance. So once you're fixing those tickets on the Kanban

knowledge. So

that's how, so what we've seen is, one is the approach first of all, which means not the push approach, but the pull approach, how to do it. One cycle or multiple cycle, how it look like. But if you put it in a scale of automation, then how much valuable it would be.

Okay. Now, the important question is.

Sorry about this.

Double mic. Yeah, it's cutting out a bit, so I'm just going to put that one on as well.

Should I do a voiceover later, actually?

Yeah.

Thank you.

Welcome around.

I think you need to repeat that from the beginning.

I have the patience. Who has the patience for a second?

So the question is, so I was all the time I was talking about, okay, it receives the context, we give the information, it will store it, but the question is, where does it actually store it? So I have a very opinionated opinion, hear me out.

I prefer it has to go to a GitHub repository because eventually somebody will actually come up with a, you know, 20 million seed funded SaaS solution for you. But before that, I prefer to actually put it in GitHub as a repository.

Why? Because if you look at it at a scale, if you want to do this, there will be multiple agents, multiple teams actually contributing to the same knowledge base. And there will be conflicts and resolutions,right? So the easiest way to do is using GitHub because it actually comes with inbuilt PR processes, review processes, things and all.

So if multiple domain experts are sitting and uploading the files or, like, agents are contributing to it, the most efficient way to manage is in a GitHub, something like a structure like this. And the other advantage is also if you put it on GitHub, you can also publish it to Confluence later or, like, Slack, wherever you want to publish it to, another solution that you want to use.

So I prefer to have it on GitHub, but if you want to directly integrate it to Confluence and all, you can also do it. Next is a Meta Model. How many are aware of the word Meta Model in the...

Okay. Maybe I can quickly show you how does it look like.

So Meta Model is basically something like this,right? So how does your domain actually structure around, like, is a business process or how it is related to a systems, how systems are related to an APIs, and how is this business jargon or, like, tech jargons are actually linked to which one.

So these kind of a relationship, Meta Model is really important. It's not necessary for the approach that I have proposed, but it's an add-on. And why you need to have this one isright now, think of it as like a map.

Right now, your agents doesn't have any map actually to navigate with your knowledge base. Basically what you're doing is you're dumping, like, these many number of files and you need to figure out which file I need to need,right?

But your file structure is actually a representation of your Meta Model. It actually knows how to navigate. For example, let's say, can you fix this system? It will understand if I make changes, which business processes will be affected and which APIs I need to change or, like, touch these kind of things.

So it's also important to have a Meta Model. If you have it, then it will produce more value. So I strongly prefer to have a Meta Model along with this approach.

Okay. So the last part is what is the value it created. So there's a lot of slides that you have seen, a lot of demos that you've seen. So personally, I need to also share, like, what is the value that I see when I was using it or, like, the other people who I shared with already were using it, came back with the feedback and told me.

First, the most valuable thing is knowing the unknown. So what is never documented is something can be surfaced only by this approach actually. Otherwise, you will just end up in an endless myro board of, like, putting tickets on, like, this is missing, this is missing, I need to add it, I need to add it, and keep on doing it.

So this is the fastest and better way to discover with your previous work items and all, what is never documented, things and all. Second is basically I can now give work to agents rather than I do all those things.

Like, rather than I give the agents all this information, let it manage my knowledge management. I don't want to be the knowledge manager of it. So let it do it. So those are the two big values that I have seen.

If you want to use it, I think, like, you will also see those two as the most valuable. But these are the other things what I've seen.

Now, okay. So I also need to tell you, like, what is the drawbacks of also using it,right? First of all, if you are coming from a small team or, like, if you say, like, no, no, no, my documentation, my knowledge base is really good, I'm, like, super happy for you.

You're the lucky ones in this worldright now with agents. For you, it might not be really relevant unless you have a

very complicated documentation that you have. Second is I already mentioned the manually doing is very painful. I don't prefer anyone to do it. If you want to just write for testing purposes, you can also do it, but automation is the most best way to actually use this one.

This is very early, this approach. So by tomorrow morning on YouTube, somebody would have already posted something differently, better than me. So in the era of AI, nobody knows, like, how long a thesis or an approach or an app product want to survive.

So for now, I see this is the best approach. Okay. So the whole workshop, so we started with one pipeline,right, on the ROI. And. So the demand-driven context actually sits between this monolith and also the retrieval layer actually.

And what it does is it actually helps you build curated context blocks for you. You can also think of it like a cache database that you have. So every time your agent doesn't need to go and, you know, boil the ocean for fixing an issue, rather than if you have a good context block of information, most of the time, 80% of the time that can be usable.

Because what I also believe is it's always the 80-20% rule. So 20% of your documentation is most useful, 80% is some corner cases you have to look into it. So rather than giving 100% of things, you need to figure out what is my 20% of that that is super helpful for agent and have it like a cache database, the context block of it using it.

And rest of it, you can leave it like links. So whenever agent feels I need more information, then only it can go and check the whole monolithical institutional knowledge.

Okay. So from here, what you can take from this workshop is three things. One, I hope I made sense of this approach. So there is a GitHub repo which I detailed it out and also a starter guide on it one.

If you want to go home and try with it, you can try it. You already know how the framework works. So you want to go home and just remix the whole approach, you can do it and let me also know.

I'll leave this one and I'll join with you for contribution. You have a context gap scanner that I showed you which is live already with presets. I think, like, added, like, $20 on it. So hit it as much as possible.

$20.

$20,right? Okay. Okay. So after $20, so first term for first term.

So all these three you can use, you can take away from this workshop. Okay. So because this is a workshop, so I also would like to want you to try something. What you can try is three things. One is either if you say, like, you know what, I'm so tired already, it's almost like 4:00, it's almost about to go for a party, I don't want to do it.

So you can just go to the context gap scanner. Everything is a preset here. You can just try it out, hit it, and see how it works. If you think it can be done better, let me know so that we can work together.

Or otherwise, let's say now I'm very technical, I want to know how it works under the hood. This is a GitHub repository. It's under, maybe I'll just take this out.

This is a GitHub repository and it has all the information. Plus there is a starter guide also if you want to try it out. But if you still feel like no, I want much more simpler.

You can also try this one. So you don't need to do anything. Basically, take this prompt, take one of your Jira tickets or incidents that you haveright now. If you already built MCP servers or, like, any other kind of things, you just use that prompt, give it to your agent with the incident or a Jira ticket, and ask it, give me the quality of the knowledge base that I have as per this incident or Jira ticket in this way.

And see how much of it comes in the red, which is never documented. So you can try also this simple one.

I'll just leave it like this. Maybe you can take a picture. Maybe I can switch to the slide if anyone wants to.

Cool.

So while there's time for the hands-on, can I ask you?

This is your slide. Okay.

Yeah. So I see some bottlenecks in this approach. I find it very interesting. So my first question is, have you already used this way of working at scale? Because we've seen mostly toy examples,right?

Yep. Yep.

In your slides so far.

I used it, not at a scale. I started with simpler because you also need to see what is the scope of it. Let's say I have an enterprise and I try it at enterprise level, I can't do it because it's multiple domains, things and all.

Even if I do it at domain level, I need to understand, I tried it at domain level. Then even at a domain level, there is so much of a domain expertise I need to fill it up and fill those gaps.

So again, I cut down into maybe what is the smallest team that I have and the smallest team's Jira tickets, the smallest team's incidents, and the team's Confluence page with a bit of a scope. Then if I drill down the scope, then I feel like it's more fast, more useful.

But if I do it at a bigger scope, what happens is not one person has the whole domain expertise. So basically it again becomes like somebody has to come and five or six people has to sit down and start doing these things.

Yeah. I'm a bit concerned that this might denial of service attack your team members in a certain way because our LLMs are fine-tuned to keep eliciting information, to keep getting more information out of us to ask follow-up questions.

Oh, okay.

So I think it will be hard on the engineers that have to do the question answering. And secondly, the scanner is nice, but that still builds on the assumption that all of your team members and the rest of the enterprise are still using your enterprise IT, well, as planned, that they're actually filling in their tickets with all the details and etc.

And I know from practice that that is most of the time not the case.

That is true. That is true. I agree with you. Even if I go, my assumption is also even if I go to a leadership to buy in, like, hey, can you give me a bandwidth or, like, you know, I need these people to actually sit and fix the context, I don't thinkright at this point of time nobody will do.

But I think it will happen because slowly I think we are slowly moving towards an agent manager where agents are becoming semi-autonomous or autonomous and we manage them. But at this certain point of time, somebody has to fix that knowledge because it's not going to come from anywhere.

You have to. So then the enterprise focus will shift towards the gap. That's what I started saying. I don't think nobody is looking into the problem yet. Everybody is very focused with agent, how good the agent is, how good the retrieval is, but how good the context is, you're not solving.

I think, like, in down the line in a year or so, I think people will realize the importance of it and the Kanban board will definitely come into reality actually very soon. Yeah. Thanks.

Yeah. I think I should build on the same point. I think when we look at large, which requires actually the search of truth, it's not actually the documentation, it's actually the code. Just wondering, have you applied it to the code base?

I did. I also uploaded the code base, but I got a mixed result when I... So here is the thing. What happened is when I only use code base, it is particularly good. Or when I only use Confluence or, like, textual data, like, it gives a good result.

But when I combine it, somehow actually it conflicts because it creates a theory out of the GitHub repository, but the same GitHub repository documentation is also on Confluence. So there it gets a conflict of, okay, what is the source of truth?

Code says this, should I implement it this way as per the documentation? So then again, I need to create an additional skill or rules, like, okay, what is the ranking that you need to give? If you see it in GitHub, that means that is the source of truth.

Or if you see it, if you don't see it, then you have to look the information in Confluence, things and all. But those are still, I'm trying to fix those things actually. So seeing the gaps and fix those things.

But I definitely see that issue combining those two.

And the second question is, but interestingly, I was actually going to apply the same approach with skills because what we find out is actually, like, you have your kind of like your, like, the process starts, like, by running agents, which is bringing a context and identifying theright skills that need to be used,right?

Then you go and you do the task, then you fail. I think,right? The moment you fail, you identify what you need to solve. You go back, you curate, you fix. But then you fix in the knowledge kind of base.

Once we find out actually if we go back and fix the skill, that increments kind of the next iteration. So I'm not sure if skill, because I saw you had some skill, but skill is part of the iteration loop or outside of it?

I thinkright now the skill that I have built is static, but what you are more proposing, if I'm not wrong, it's like evolving skill,right? If the skill fails, it has to evolve. Right. I agree with you. I never tried it, but I think, like, it has to be like that.

Because I'm also more concentrating on how to do it at scale. The reason is also

I want the context to be fixed before retrieval itself, not during operational. So first when I started with it, I started doing it when operational, which means, oh, I have a work item, I will listen to it, it will fail, then I'll start giving context and all.

But it takes a lot of time. It takes a lot of patience for me. So rather than doing it, you know what, I'm going to fix the context, but before retrieval. So if I can, while I was answering her question, if you take a team, the context that you need to fix is very small.

So you can use a context gap scanner kind of a thing. And maybe if you're good, have a good domain expert, I think, like, a couple of weeks you can actually fix your documentation. Not like 100%, at least like 60, 70, 80% of a good quality that you can already build it.

So my proposal would always be don't do it at an operational level, at a real-time level, but do it before retrieval itself. That is much better than this approach. Yep.

I have a question that is really unfun,right? Especially at scale, if you have a lot of documentation and if that documentation is sitting in a GitHub repo or whatever, you may have situations where you ask questions that may need, I don't know, like five or six different docs.

So you will have, like, I'm going there, reading all the docs. Like, this takes time for retrieval of information and then it takes a lot of context also for the LLM to get theright information.

Right now, after Claude code announced 1 million of tokens in the context window, I know I don't have any problem. So I calculated it. At an average, it's like 96K tokens because I tried with different domains actually. Per domain, I see, like, around 96K tokens if I consolidate everything, like Confluence, things and all.

So easily if it's in the context window actually. I tried to do some experimentation around, you know, a graph, RAG, put them there rather than just take all the files, use a graph, RAG, understand the intent. But for me, just putting the whole contextright now in the window gives you more results than actually doing RAG.

Unless you have a very big, almost around a million tokens of a context that you want to fit in, maybe then you have to use a bit more retrieval mechanisms between it. But otherwise, I think, like, it should be fine.

I have a question. I opened your paper and could you explain this graph, like the comparison between different techniques, like domain knowledge strategy, knowledge access?

Which one? Yep, sure.

This one.

Okay. So I also did the citations from other papers. So not directly related, but you have the paper of ACE, which also does a similar thing. But ACE is not exactly into, how do you say, discovery and curation actually.

If I remember it correctly, maybe I need to refresh my memory.

What do you mean between the difference between domain knowledge and strategic knowledge here?

Okay. So strategic knowledge, okay. So what ACE and all are doing is when you are trying to have a conversation with AI, you can see in the Claude code and all, it updates its memory or, like, the relationship with your things like that,right?

And also from the chat history, it understands what is the most important context I need to remember, those kind of things. So when you are in communication with it, that operational conversations with AI improvement, they propose. So what my proposal is not based upon your conversation with an AI, but rather your domain knowledge which is documented actually.

Okay.

Somebody else has a question. Sorry.

When running to remote knowledge things on Confluence and other stuff, how do you ensure that your agent only points to the updated or, like, the relevant documentation? Because in your local file system, you have tags like outdated, which is okay, but remotely that's harder to tag, I guess,right?

Okay. So when I wrote a pipeline for extracting from Confluence, it also allows actually to give you a date and also last updated who created kind of an analytics on the space. So you can use it to actually put a threshold of, like, okay, on this particular date, whatever is old, consider it as an outdated one.

And let me know. Don't just consider it as an outdated one. You let me know because sometimes the document can be stale for so long, but it could be an important document actually. So it lets you know, but not, like, take decisions actuallyright now on this one.

So you decide which one is stale and which one is not.

You don't have, like, an intermediate layer where you're in the repo, you store, this is still an updated document?

Okay. So when it is curating the context, also it updates with a date and also the state of the document, like stale, active, and clean. So it also looks into, okay, this is stale, I know, I'm not going to touch it and I'll just go to look for any other new other documents are there in this one.

Okay. Thanks. Do you think about how to manage access or permissions later to this knowledge? Like if you have some knowledge in the company that is sensitive and only specific people can get access to it? Because currently, I guess you just have all the knowledge and everything is accessible, but.

Okay. So because it's not a product or a SaaS solution, it's just basically GitHub, for meright now, permissions and things are not difficult to implement because GitHub out of the box gives me who I can give the permission to is GitHub, who can haveright read access things and all, who can merge those things and all.

But in case if it evolves into a product and, for example, context gap scanner as a product and I want you to test it because the reason why I was using presets for this workshop, not actually asking you to upload the files is because I don't want to take your IP data on this one,right?

So unless it becomes a product, you don't have any problem, GitHub and all. But if you have a SaaS solution for this one, then it's between how the SaaS solution will manage itright now. But the approach has nothing to do with access things and all.

How you implement those access on the knowledge is up to you. You have a question.

So this is, of course, about documentation, but did you give any consideration about using it on some, like, central tooling that a company would use? Like, let's say that you have a platform team and you have a CLI that the different teams are using.

And so now it's used by different agents,right?

Okay.

And so the agents can also be like, well, this action is available for a resource, but I don't want to do 500 calls just because I have a list of 500 resources. It would be nice if the tool could do that.

I don't know if you've given any consideration to this.

I think that is how it has to work. In an organization, you need to have a central solution for it. But how you want to do the solution is up to the organization. For example, we are doing Agile,right?

So Agile can do by Scrum, Kanban, or, like, Lean or something. And also you can do different apps to do it. The process is the same, but how you do it, which method you will choose, and which app that will you choose in your organization is different.

In the same way, what we have discussed is the approach. If you want to put it in the organization, you can use the approach and you can do it in whichever way you want.

My point is more like, so with this, you can identify gaps in your documentation,right? Could you use it to identify gaps on your tooling, so to say?

Okay. When you said tooling, it's the agent.

Internal tools that, I don't know, maybe a team is building for the rest of the company.

Just the infrastructure in general,right?

Maybe, yeah.

Can you give me an example of, like, how it.

Let's say that, I don't know, you build some sort of abstraction on top of Kubernetes. You don't want your developers to necessarily know what to do with that. And then you have a different CLI or you have something,right?

But then, like I say, I don't know, maybe you thought that they would list one of your custom applications or corporate applications one by one, but a team has grown into using more of that and suddenly they have a lot and they don't want to do that many calls or perhaps even the agent is like, well, this is inefficient.

I would like this internal tool to work in a different way. And in that way, you would identify, like, gap in the tool or a performance improvement, kind of like this does for documentation.

Could be extended, actually. So because we have seen the business processes also,right? So it can also document business process. But business process is nothing but how the process in the application, it actually runs and does things,right? So you can extend it to also find out the gaps in the business process or, like, how it works.

It could be an extension to it.

Okay. Yeah. Thank you.

How would you ensure that Mendel won't kill you?

Sorry?

How would you ensure that Mendel won't kill you? Because the knowledge for a company changes with time. So if the answer for a question today is B, tomorrow B1, on Friday it could be C,right? So you have to, at C, you have to identify B1 and B and override them.

If it's stored as a text, that's a problem.

Okay. So when I showed the context gap scanner, you also saw, like, an indicator of duplication,right? So if today you have a document, tomorrow you have version 2.0 and something else actually, it will find out the same information is having in three different.

It also will find it. If you have only one, it is changed, it will take the latest updated one because as a human, you changed it. So it will take it as a source of truth,right? But if you have three versions of the same document, that's a duplication and it will flag it as a duplicate.

Right. But if it's a search problem, how do you ensure the performance? Well, let's say you have a document of 100,000 words. You just change a word, like a password, let's say. It won't be there, but for example,right, it has changed.

So you have to find this specific word, compare those three documents, flag them and replace them, sort of,right? How is it feasible? Which tools would you use to ensure it will not kill you cost-wise?

I didn't quite get your question, actually. Is it, like, the token usage you're worried about? Like, that many tokens that we used, what is the cost saving?

Yeah. Precisely. It will grow, presumably be big, and if there are a lot of changes, you would have to maintain that whole database,right? So you have to have some sort of structure. I don't know, maybe GrabRAG or whatever,right?

My question is, well, because what you presented is sort of a happy path where you have a gap, you fill it, and then you reuse it. But in a while, you will have a bigger problem where you pretend to have that gap filled, but actually it doesn't contain up to that information.

To be done, it contains wrong information,right? So you want to preserve that.

Okay. Okay. So it can flag as per when it is created or the last updated. You can set such kind of filters. But let's say you have a latest document which has the wrong information,right?

But you don't know that,right?

No, that's true. But for example, as a human being,right? So you go and look into documentation. You told somebody to look into documentation. And the person looked into the documentation and as per the documentation, this is being implemented in this way.

The person will do it,right? It's not an agent or a human issue.

It's way too late,right? I mean, you solved one ticket, one zero ticket, and the agent is trying to solve the second one. The assumption is the knowledge is pristine. So it's solving it, but the solution is wrong. So that's too late,right?

So you say there is a scanner. Fine. Do you run it daily? How much will it cost?

You can have another process spawned that would try to see, like, last updated, try to see on some cadence, try to run a process that will update the knowledge.

Fantastic. How much will it cost?

For example, sorry, you're saying.

Less than doing more meetings every day to onboard someone or update knowledge.

That's a strong claim.

But I don't think it will cost that much as.

So previous of AI.

As I said, like, when I tested it, there is none of the domains which cross more than 100k tokens, actually. So I don't think we will, for example, contact GapScanner,right? I don't think, like, you have to do it, like, on a daily basis or anything.

Even if you run on a daily basis, like, 100 tokens and do one scan. For example,right, if you try to start hitting all of them, all of you, the context gap scanner, I think you can't even burn, like, $1.

I think so, if I'm not wrong. It already had, like, oh, you're ready. Okay. Okay. I'll cancel the subscription now. But I.

Because I'm going to go back on different two things. One is the complexity of the organization you're trying to solve. Like, a good example, like, say, from an airline industry. We're talking about, like, hundreds, if not thousands, of, like, repository systems, documentation, third parties.

So I agree with you. If you're on a specific domain, it's solvable. The moment you scale, you will have to solve this cost question, which is going to depend also how fast the, to his point, like, how fast the data change, which I think most of the time, not that much.

So the moment you get to, like, 80%, 90%, you just continue evolving the last part or something. I see it's different use case, different use case.

Yeah. Use case by use case. Yeah. Yeah. Any other questions?

Yeah. I was curious. So I ran the scanner for, like, one domain,right? And it has a bunch of recommendations. How do I know that that's enough? How

did you assess that?

It actually tries to detail out as much as possible. Right now, I haven't actually exposed everything what it did, just for the UI purpose. But all the per ticket, what it actually found, like, it writes, like, 100 or, like, 150 lines of Markdown files and save it somewhere.

So that gives you more details in case if you want to know, actually. So for the demo, I just put the, you know, nice UX stuff on top of it, but you also have detailed information at the test.

Anyone else has any questions?

Can I help you with one?

Yeah, sure. Go on. Easy one, huh?

At least try.

Yeah. If you start a new job where it claims or gets something for the 40% of the gap,right? And if I quote the translationright, your claim is you're defined, like, how many zero tickets you can see that gap.

Not just.

You said a couple of weeks,right? A couple of weeks and you will have filled that.

Not fill the gaps, but discover. But if you scope down to a team, within weeks you can do it.

Right. Because, I mean, if you wouldn't fill it and would have to keep asking those questions, then the solution doesn't make sense,right? So at some point, you have the knowledge base that's created, available, and delivers value.

Yeah. So you do it one time, first of all, or, like, multiple times at first. See the whole picture first of all. What is the state of your knowledge base? First, fix it at that level. Then you go into operations,right?

You still can actually also.

Yeah. I mean.

You can still continue doing it with the agent, with skills.

You kind of squeeze that 40% of the gap,right?

Yep.

And I think the same process of kind of providing that knowledge, because that knowledge, you say, is in those heads,right? It's just not documented. That filling happens pretty frequently when a new member joins the organization. So wouldn't a replacement that be just get Zoom calls, transcribe them, and use them as a source, assuming all the calls or the knowledge transfers or maybe introductions happen over Zoom?

Or Teams or whatever the communication. Like, if there's a new member joining, asking questions, and you have access to that.

You also have access.

That's also a solution.

Okay. So you mean, like, you can also give all the transcripts rather than doing the cycle, you mean? That can also be done if you only have the, all the time you have a discussion in the meeting, everything is documented in the meeting transcript itself.

But I don't think, like, that's the same case for everyone, at least.

But that should be easier in Zoom,right? Because if I come across a zero ticket that I don't know or I'm unsure about, I will call someone. I will call or chat with someone.

I think the amount of time people spending in Teams, if you do use the transcripts, actually, those are the ones, actually, which have more tokens, actually. There are so many useless meetings. The transcripts, actually.

Fully agreed. But the compression happens on you and then that part of the database.

Could be. Again, it depends upon institution to institution,right? So are you, like, more into meetings, have solving problems within the conversations, and those conversations has the data, or, like, your confidence or things has the data. If you have it, use those transcripts as your knowledge base.

And at the same time, like, the compression, actually, that works, actually. That is more useful. Yeah.

Anyone else? Any questions? No? All good.

Then thank you so much for attending this session.