Context Platform Engineering to Reduce Token Anxiety — Val Bercovici, WEKA

This is Val Bercovici, WEKA's Chief AI Officer, and I am joined by—

Callan Fox, head of the product management team here at WEKA.

And we're both thrilled to present Context Platform Engineering to you at the AI.Engineering Code Summit. Now, let's kick this off with an announcement we're making: we're actually open-sourcing our Context Platform Engineering toolkit. And this toolkit features a really cool load generator that Callan wrote, that lets you configure agent swarms and agent subtasks with very specific SLOs, being able to cycle through deterministic and random prompt cycles, and engineer context platforms with all sorts of model parallelism options, disaggregated or aggregated, prefill and decode options, and some really important memory tiering options we're going to be discussing here.

So if we advance the next slide, we'll see that this is an open-source toolkit that's already available to you on GitHub, so Callan and I really encourage you to just get on GitHub, download this, play with it, and give us your feedback.

Let us know what you need changed, feel free to contribute and fork the project, and advance the field of Context Platform Engineering, which we're going to be introducing to you later today. So, moving on, one of the key requirements for Context Platform Engineering really relates to the context engineering insight that our friends at Manus shared with us earlier this summer in their pretty infamous now Context Engineering blog, and they highlighted the fact that KV Cache hit rate is the single most important metric for production-grade AI agents.

And the reason Context Platform Engineering is so important is it dramatically simplifies reaching maximum KV Cache hit rates, as we're about to show you. On a more personal level, if we think about Token Anxiety, I know that each and every one of us, you know, feel that anxiety.

The reason Context Platform Engineering is so important is shared by the Context Engineering blog from Manus earlier this summer, where they particularly emphasize KV Cache hit rates are the single most important metrics for production-grade AI agents. And Context Platform Engineering, quite simply, maximizes KV Cache hit rates in a very straightforward manner.

On a more personal note, if you think about the concept of Token Anxiety, as we all regularly hit token rate limits, Context Platform Engineering helps you engineer platforms that eliminate token rate limits and help us be more productive with regards to developing our software.

Now, in the absence of Context Platform Engineering, we often resort to Context Financial Engineering, and that's fundamentally prompt cache arbitrage, where we balance the needs of pricing between the bookends of input and output tokens with these new token pricing categories that have appeared in the landscape over the past few months, focusing on cache writes and cache reads.

And we've got to be somewhat clairvoyant when we're doing this arbitrage to figure out how many cache writes we want to invest in for either 5-minute time to live; in some cases, with Anthropic, for example, we can do 1 hour time to live.

And that's all against, balanced against, the predictions we need to make on how many cache reads and cache hits we think we're going to have during those intervals. This becomes very, very tricky to be clairvoyant and predict the future, and I think it's much better to apply context prompt engineering techniques to overcome token anxiety and prompt cache arbitrage, than to continue to do the arbitrage in Context Financial Engineering.

And so, one of the ways we're going to be doing that

is looking at, and Callan is going to dive into this deeply, the cadence mismatch between the relatively slow human feedback loops for agents, and then the agent swarms and the agent subtasks themselves that iterate at much higher cadence, often in parallel, waiting on humans but conducting a lot of really cool work in the background, consuming a lot of tokens in the background, many of which are cacheable, but we just never know how the platform is able to respond.

And that's one thing we're going to be diving into here, is the fact that, if we go to the next slide, we're looking at fundamentally a token storage problem. And what we're going to be doing is explaining how the service level agreements we sign up to, when we subscribe to our various, you know, token tiers, or we actually commit in our instructions, in our agentic instructions, to specific token cache writes and cache reads, how those SLAs convert to service level objectives delivered by the context platform itself.

And more particularly, one of the insights that Callan reached from his research at WEKA Labs is that what we're doing when we actually subscribe to our token tiers, or we actually pay for particular token writes, is we're really purchasing cache, KV Cache slots, in token storage.

So there's definitely a whole science around the Context Platform Engineering to how context platforms take those SLA requirements, optimize infrastructure, optimize KV caching and memory tiers, and deliver specific SLOs to try and meet those SLAs as much as possible.

So with that, let me actually hand it over to Callan for actual research findings and lab, and test results from WEKA Labs.

Thanks, Val. So look, what I want to do is just go back to one of the slides that Val showed earlier, and what I'm going to do from now on is I'm going to focus on thatright-hand loop, and the first thing I'm going to do is I'm going to start by visualizing what that loop actually looks like, and then we're going to go into a little bit more detail.

So if you think about that loop as a column, and I've got a graph here that shows a very, very common pattern that happens in agents. So the salmon color is showing new tokens that the system's being exposed to, the gray is something that could be cached, again, within a limited amount of cache, we'll get into that shortly, the blue is the output tokens, and these blue dots down the bottom are showing when the user is actually giving responses in this particular case.

This is a really common example you get, where basically you start off, you consume context all the way up until you hit a high watermark set by either the model maximum length or by the inference provider itself, there's a summarization phase, and then you start a new cycle.

And everybody knows that summarization phase where sometimes, you know, the agent loses a little bit of its fidelity, a bit of its intelligence, and that's why we're trying to, you know, get more context engineering to a larger set of platforms, and we can raise that watermark.

So if we go into this in a little bit more detail, the question I often get is, okay, well, what is that, that's a lot of gray, what's that made out of? So here I'm able to get the data and actually look at individual prompts and what actually makes them up.

So when you look at agentic data, especially agentic coding, the actual user input is only a really small part of it, and you can kind of see it here just visually, that if you just scan across the lighter watercolors, the system prompt and the user text itself, and the rest of it is tool use and tool responses.

So this is, this one in particular is from Claude Code, where you're spending a lot of time, where the system is, you know, doing like, for example, a bash command, it's grepping something, it's getting a result, and then it's doing something else.

So where this really shows out in the data is if you actually look at the median time between requests, it may be, for a conversation, it looks like that, we have data for billions and billions and billions and billions of tokens, the median time is 10 seconds, 15 seconds maybe.

That heavily depends on whether the human's involved in checking every single tool use, but the mean time is in the minutes, because the human, or even hours, because the human time to respond is much, much, much higher. And that's what we're showing before of the two sides of the loop.

So the other thing that's interesting, and something that's very common today, is multi-agent. So you might have a core agent, which I've shown here as the orchestrator, and then you've got these sub-agents that are like spun up to do individual tasks, and depending on the type of agentic coding, or just any agentic software in general, these agents, or these sub-agents, may be short-lived, as in their context does not endure between one wake-up and the next, or there are some when they do endure.

And it's really important to use that agent, because it allows us to create, to effectively target more context at very particular parts of what the problem you're trying to solve, but as a result, you do actually end up using more context, and I'll explain that very shortly.

But if you visualize this gray section a different way, and I show you the colors, you can kind of see how there's this common relationship of the common context between all of them. Again, this varies a little bit depending on Codex versus Claude Code versus others, but you can see how it changes over time, and how the agents relate to each other and have this common understanding, and then back to the orchestrator to wake up the next agent.

The thing that we're here to talk about today, though, mainly, is that, like, while there's a lot of gray that could be cached, the reality is very different. So if you send this to an inference provider, what ends up happening is you don't actually get 100% of the cache hits that you could, that you could get.

Now, why does this matter? Well, there's two ways to look at this. If you're paying for API tokens, you're literally, it's literally costing you more money, because every time you see a yellow here, and this is just a simple example, you're paying input token costs, so you're refreshing your cache and you're paying a full hit for that, so potentially 10 times more than what you were if it was cached.

If you're a subscription user and you're thinking, well, I don't care about the cost, I don't pay for that, I pay a flat rate, that is true, but you're still, like we said before, you're paying for a subscription, and that subscription is rate-limited due to your cache usage, and you may actually hit rate limits further, or quicker.

So that's something that we want to be able to do, and we work with a lot of providers today to remove as much of this as possible. That's good for the user experience, and it's also good for the provider.

So why does this happen? Well, I mean, if you think about the last graph where I show the columns, they're not, they don't take into account time, they're just one after the other after the other, but there's obviously a temporal way to look at this.

So this is the way that I like to think about it, and I know this is a little bit more of a complex graph to look at, but bear with me for a second. So on the left-hand side, I'm talking about working sets, so that's the number of tokens that the cache system is holding in its memory based on different time to lives of the actual cache itself.

And then the bit at the top, the dotted lines, based on theright-hand secondary access, is showing the cache hit rate as a result. So the red is showing 1 minute time to live, and what you can see is there's prompts here at the start on the left where

it's thrashing up and down. And the reason it's doing that is the time between requests at that period is longer than 1 minute. So you're getting a period where you might take the cache, get a hit or two, and then drop the cache, and then you get another one, you got to refresh it.

So it just, it doesn't really make sense,right? You go to 5 minutes, which is the blue, and you can now write out more and more of those cache hits, and as a result, you get a higher cache hit rate.

You can see it at that very start up there, comparing the two. But then you're still missing many others, there's still many times where the time between a request is even larger, so the next one up is showing 1 hour, and while that requires the cache system to hold, you know, a little bit more tokens in cache, and eventually quite a fair bit more tokens in cache, it's got to hold it for a longer period of time, but the result to the end user is a better actual experience, and to the inference provider, which we'll show very shortly, it's a much better experience for them as well.

The problem, though, is to do that, you need to be able to hold a lot of tokens in cache, and you need good memory tiers to support that.

So the next thing I want to go into is that a lot of people think of cache hit rate isn't really something that a human's able to really internalize well. So another way that I can visualize it is by thinking about it in terms of the number of times on average that a chunk of tokens, which is a group of tokens, is refreshed.

So in this particular conversation that we're looking at here, you can see that this is showing the relationship of as I increase the time to live, how that affects my cache hit rate, but it also shows, based on the secondary access, that at 1 minute I'm literally prefilling like 15, 16 times the same tokens.

And over time, we can get that all the way down to approaching 1, and make significant differences to, again, the experience of both the user and the inference provider.

So with that, what I'd like to do now is go into the context engineering side of it, some of the lessons we learned, and just sort of really drive this home. So now I want you to think about what I think will be common in 2026 and onwards of people hosting their own, or having their own dedicated systems hosting for them.

So imagine you being an inference provider now. Okay, so now what I want you to think of is, think of yourself as an inference provider. Maybe you've, you know, worked with us or one of our partners to build your own self-hosted instance, and you want to get the most out of it.

What this graph is showing you is a relationship between a certain context length and the cache hit rate, and how many output tokens you get as a result of that cache hit rate. Now the first thing you'll see is it's not linear, and the shape of this curve will change based on the context length, based on the accelerators you use, there's lots of things that come into it, how you do disag and prefill, there's a lot of stuff that comes into it, but the curve is more or less the same.

And if I asked you, as an inference provider, where do you want to be, you'd obviously say C, and if you're in A or B, you're not making money or you're not getting enough value out of the system.

And inference providers that we work with, they have the same answer, obviously. So the question is, well, how do they keep in C? And this is where it goes back to a slide that Val showed earlier, where what they're doing is they're incentivizing users to stay within C.

And this is where we came to the realization that a lot of the times, because of how much cache hit rate impacts your actual output, that's why you're buying cache allotments in storage when you're actually buying subscription services, because it is so important to them that you stay in a certain cache hit rate band, especially for agentic workflows, otherwise they literally, you'll just melt the GPU clusters that they have.

And I think it's a really powerful thing to have in your head about how that works. So what we're going to do now is go through and think about, okay, well, what makes up this token storage? So when you think about the token storage, there's lots of aspects that the memory tiers that support the token storage need to be able to do, but to really make it really, really simple, it's literally as simple as, you need enough capacity in these memory tiers so that you can hold an optimal amount of cache.

If you think back to the slides I just showed, there's this point where having more cache helps you a little bit, but it kind of gets to a point of diminishing returns. You need to get at least to that point, and you need to be able to store extremely fast into it, because if you can't, you're going to be dropping KVs before they're in the memory tier, or you're going to be blocking GPUs, which is probably even worse.

And then the other way you need to do it is you've got to fetch from that token storage very, very rapidly, so that you can, again, not block the GPUs. They're the primary first-class citizen of this whole system.

So what does it look like? So there's a few different types of memory tiers, the most common, obviously, is HBM, and Val and I would love it if all our sessions were in HBM at all times, that's just not reasonable.

There's many reasons for this around how the batch works, which we're not going to go into today, but the point is that the main common way that this is done today is DRAM. And there's nothing really wrong with DRAM as such, it's sort of a means to an end, but it's quite limited in size.

It's okay in terms of performance, but the other thing is it's tightly coupled with the compute. So if you want to expand your DRAM, there's not really many good ways to do that. There are some technologies out there that kind of do this, but the way they're implemented, they kind of just hurt your performance.

And that's what I'm showing with pooled DRAM. You could pool more together, but it's, you know, it's kind of a, it doesn't help that much. So what we at WEKA did is we took all the durable advantages of our product, which has been, you know, tried and tested in AI training and HPC environments, and augmented memory grid is basically a supported, optimized connector between the inference systems and our existing product.

And because we're backed by NVMe, we're much denser, we're like 1,000 times, depending on how you look at it, denser. It's quite significant. And then I show another example of a storage at the top there, where, you know, not something sluggish, something that can still get 50, 60 gigabytes a second, but, and it has the capacity, but still relative to what we're talking about is still quite slow.

Okay, so then moving on to how do we test this. So again, we talked about how we've open-sourced this. Basically, Val already covered the main part of it, and the fact that it acts like it's an inference provider, it's trying to keep the load within two SLOs.

If you enable them, you actually don't have to enable them, and it'll just go as hard as it can, regardless of an SLO being time to first token or output tokens per request. But the main thing that it can do is you can either set a static number of coding agent users, or you can increase the number of those users over time, so that you can slowly utilize more of the memory tiers and be able to compare different configurations.

So there's two ways that it works. I'll just be quick through these sections, because you can read about this. I have a blog that explains how I do the testing that goes through all of this in detail, and there's obviously the GitHub as well.

But basically, it can do the initial working set and then sequentially go through those prompts. So this will be very, very, very deterministic, because as soon as you overflow the memory tier, even the slightest bit, you'll see a massive drop-off in performance.

But the other way that it can be done, and realistically the more fairer way that it can be done, is you can increase the size over time, so the amount of concurrent users that you're accessing out of a pool, and you can randomly sample where in that sample set you'll get that prompt from.

So sometimes you might be hitting HBM, sometimes you might be hitting your memory tier two, let's say that's DRAM, and you get a really nice blended number. So with that, let's go in and show you some results and just sort of explain and show why we're so excited about what we're talking about today.

So this is showing three comparisons. Comparison number one is HBM with WEKA, that's the purple. There's orange, which is HBM and DRAM, and there's the, you know, orangey pinky color with HBM plus DRAM plus that other POSIC system that I talked about earlier.

The dotted line is showing concurrent users, so the amount of users that are in a pool, and that's increasing over time. So in that initial shaded area, you can see that all three of them get an advantage of HBM.

The primary hit out of cache hit rate is coming out of HBM. But then over time, as we increase the users more and more and more, you're overflowing what the DRAM system, what the DRAM memory tier can do, and both orange and the pinky color start to drop off quite dramatically.

We also, from a WEKA perspective, also drop off because we get less and less advantage from HBM, so we have to pull back our concurrency a little bit. The system does automatically, the benchmarking tool. But then once we've sort of got down to the steady state, all three start to like level out a little bit.

But the main difference is that once you get down to that steady state, we can maintain that at a much higher amount of users, at a much higher amount of output tokens. The other way that you look at this is that was a decode-focused role.

If you look at a prefill-focused role, if you're doing disag prefill, then the prefill is actually an even better result for us because the systems, the GPUs are so much more efficient when you're doing large batches of prefilled tokens with a single decode, then we can basically saturate things more fairly and it continues.

Now, the main difference between pink and orange is that we, sorry, purple and orange, is that we have a lot more cache, so we can hit a lot more. The interesting thing about the orangey pinky color is that it also has the ability to hit every single thing that it's possible, but it's not fast enough to get it into the GPU for it to make a difference.

And that's why we're sort of showing the difference between these three, because with purple you're getting the advantage of capacity, but at DRAM speed, so you can maintain that benefit for longer periods of time.

And then maybe, Val, I'll hand back to you.

Absolutely. That was a great walkthrough, Callan, of all of your research and benchmark results from WEKA Labs. So once again, we're thrilled to be announcing the open sourcing of this Context Platform Engineering toolkit today. Please do download it, use it, give us your feedback.

Again, feel free to fork it and improve it yourself. And we look forward together to just contributing to less token anxiety overall, less prompt cache arbitrage, and more context and Context Platform Engineering in the future. A nice QR code for you to find that even more information.

And at the end of this video, in the actual transcript section and so forth, there'll be links to all the blogs we referenced here. So thank you for joining us today, and we look forward to carrying on the Context Platform Engineering conversation with you in the future.