Buy Now, Maybe Pay Later: Dealing with Prompt-Tax While Staying at the Frontier - Andrew Thomspson

If you're building an agentic system today, or going to be building it tomorrow, I want to introduce you to the concept of Prompt-Tax. My talk is broken down into 4 sections: the pain of progress, what it's like to ship at the frontier, a bunch of advice in the form of battle-tested tactics, and then ultimately a series of questions about how to think about paying back the Prompt-Tax.

Let's begin with the first section, the pain of progress. Here's a nice animation for Peter Gostov of Moonpeg that shows you the ELO scores over time as AI labs are shipping new models. But what does it actually feel like to develop with this rate of progress?

Well, for me, it feels like having a birthday just about every month. Anthropic recently released things just in the last couple of weeks; Google Gemini had a plethora of new additions to their models and functionality, and OpenAI keeps shipping at a breathtaking pace.

But with all these advances, as you bake a lot of those functionality into your own applications, there's a hidden Prompt-Tax. So what is that actually like? It's like getting a new bike on your birthday. You think about riding fast, the newfound freedom you may get to go into your friend's house, but ultimately some unexpected things can happen.

You go for a ride, you don't know how to ride a bike, and you fall over and you hurt yourself. Applied to AI models, there's an incredible amount of new functionality that you can bolt into your own applications, but there are also unintended consequences by using this probabilistic system that can behave in ways that are unexpected.

Ultimately, shipping product at the frontier has this constant tension between the opportunities that new AI models can give versus the risks that you could introduce, either regressions or unintended consequences to your products. Let's look at what shipping at the frontier of AI is really like.

I'm the CTO of a company called Orbital, with offices both in New York and London. The mission of our business is to automate real estate due diligence so that we can fast-forward the transaction of properties. Why does this matter?

We all live in homes, we all work in offices, and every time those get developed and bought and sold, real estate lawyers are involved in the due diligence of those properties, which often involve reading mountains of paperwork, hunting for needles in a haystack in order to find red flags for their clients before those properties can be transacted.

Our agentic software helps to supercharge that process by radically reducing the time to find those needles in a haystack and compile all the needed paperwork in order to complete a real estate transaction. This is our team. It's neither New York or London, but it's sunny Mallorca.

We were at an offsite. We're roughly 80 people, half of that is the product engineering team. And in terms of how we're structured, we have a product manager and designer that head up the team. We then have domain experts that are embedded in that team.

We then have software engineers and AI engineers and a tech lead that oversees the delivery of how all of this sort of cross-functional team works together to ship working agentic products.

Let's rewind way back to January 2024. We had just developed our first agentic product called Orbital Copilot. It thought like a real estate lawyer, and we started selling that to lawyers inside law firms and to real estate professionals in various businesses that needed access to the work that Orbital Copilot can perform.

I'm going to show you a quick demo of the product. It's on sort of warp speed, so it'll go pretty fast and I'll talk through it pretty quickly, but I thought it might be helpful to see what a sort of long-running task usually performed manually by lawyers reading paperwork and compiling all of the information that is extracted manually, whereas now we have an agentic tool that does that automatically.

Here we go. So we start by first clicking on a report. We choose the appropriate occupational lease report. It then is going to ask us to upload some documents. In this case, I've uploaded both a deed and a lease.

One I think has about 30 pages. The other has about 70 pages, so about 100 pages in turn. It first starts to OCR all these documents because they have handwritten and typed text, so they need to be OCR'd and structured before they can actually start with the real heavy lifting.

And here it goes. So on the left-hand side there, you can see the agentic system has sort of created a plan, broken that plan down into many subtasks. Each one of those subtasks are its own agentic system with multiple LLM calls.

It's been given an objective to find the date of the lease or the current annual rent, and it's reading all of these legal documents that you see on theright to find the appropriate answers. And as you can see, each of them going green here.

We'll wait until the last one completes, and then a final report can then be manually reviewed much more quickly by a lawyer because all of the necessary information has been extracted. And as you can see here, the citations can be clicked on very easily so that a licensed real estate lawyer can kind of go back to the ground truth.

And then the final step was downloading the word report to get stored in the system and ultimately sent to a client in order to progress the transaction. Okay, so that agentic system, how has it actually performed over time?

Well, let's rewind back to when that blog post was written. We commercialized our agentic agent 18 months ago. We were basically only burning less than a billion tokens. We're now, within touching distance, 18 months later, almost consuming 20 billion tokens every month on the behalf of real estate lawyers who are working on the behalf of their clients to process these real estate transactions.

So 20 billion tokens' worth of work that was usually done manually by lawyers is now done automatically by our agentic product. We've also gone from 18 months ago doing exactly zero in revenue. We've scaled that aggressively too to now doing sort of multiple seven figures in revenue of annual recurring revenue over time, and that continues to scale, and we're immensely proud of that.

So where have we got to, or where did we start, and where have we got to? We started on GPT-3.5. We had all these system one models that we went through. One of my personal all-time favorites was GPT-4.32K, radically expensive, but enabled for the first time in a genetic system because of that increased context window.

We then migrated to 4 Turbo, 4.0. We're now using 4.1, and we also went to the system two models going from 0.1 preview when it first came out all the way to the latest sort of 0.4 mini and everything in between.

We've had to make some decisions along the way for better or worse, and I want to articulate three of them. The one was to optimize for prompting over fine-tuning. The reason for that is we wanted to maximize the speed of development.

We got feedback from users, we adjusted a prompt, and those prompts were pulled in real time into the agentic system, and so changes could much more easily get incorporated into the product, especially in the early days when we were finding product-market fit.

And now, once we've had it, we can continue to ship features at a blistering pace by just simply adjusting prompts over time. The second was this heavy reliance on domain experts. These are private practice real estate lawyers who worked in law firms who've come to work for Orbital.

They have decades' worth of experience, and they are effectively writing many of the prompts. They take the experience that they've learned over many decades and essentially teach an AI system about that expertise, so it has it itself. And the third, probably most controversial one, given how eval systems are all the rage, is we've elected to go for vibes over evals.

We very much would like an eval system. It's always sort of just within touching distance on the roadmap, but it's pretty incredible seeing the growth we've had both in terms of tokens, both in terms of revenue, both in terms of user feedback, that we've come all this way in 18 months on pretty much vibes alone.

And by vibes, I mean not a rigorous evaluation system that's objective, but more human beings, usually the domain experts, testing out a system prior to its release, sometimes just getting a more subjective feel, sometimes logging things down in spreadsheets and figuring out if things have regressed from prior model changes, but nothing that's terribly comprehensive and much more open to subjectivity.

So our prompts can be broken down into two areas. The one, agentic prompts. These are pretty much owned by AI engineers. These are system prompts. They're prompts that deal with helping the model choose what tools to use and when.

And then more domain-specific prompts used by our real estate lawyers that teach the system its expertise in the real estate domain. So this is the number of domain-specific prompts over time. We've gone from near zero in the early days to well over 1,000 prompts now.

The challenge with this is that more prompts equals more Prompt-Tax, and I'll dig into why this matters fairly shortly. So when a new AI model drops, what we do, we experiment with it rigorously in the week, whether that's the AI engineers or the domain experts.

Ultimately, what we're experimenting for is we often have features that we've envisioned. We want to unlock those features with new capabilities, and we want to see whether the models are now fit for purpose, or we could get inspired with new ideas based on our experimentation.

We then also need to figure out how much Prompt-Tax is actually required to migrate the prompts we have to this new model, as well as there's this inherent fear that exists. We don't know exactly what will happen if we ship this new AI model.

There's a lot of unknown unknowns, and so that inherently creates a bunch of fear. And it's our job to sort of pinpoint where that's coming from. Is it rational? Can we mitigate it? If not, let's ship. I want to stress that Prompt-Tax, at least for me, is not technical debt.

When I think about technical debt, often you're optimizing for shipping quickly, and you may fix later. By that, the feature that you ship may never end up finding product-market fit, and you may just delete it, or that little prototype that you thought was just going to stay a prototype becomes the core part of the product and needs to be fundamentally rebuilt as you scale.

Prompt-Tax, however, there's this desire to upgrade now. New models often unlock new capability, and baking them into your agentic product is immensely valuable. The problem is, because of these unknowns, you don't know exactly what it's going to improve and what it's going to break, and so it's this idea of, can we fix things on the fly?

Is there an optimal way to release this new model to sets of users, figure out from them, get their feedback on what's working well and what's not, fix that as quickly as you can, and then roll it out to as many people as possible?

That way you maximize the benefits of the new AI model while mitigating, hopefully, as many of the downside risks of the model regressing any prior functionality that users have relied on. Okay, third section. Let's spend some time on some of the advice that is in the form of battle-tested tactics that my team and I have discovered over the last 18 months.

When we've migrated from system one models like GPT-4.0 to system two models like 0.1 preview, there's sort of three things we've come across. This idea that in system one models we often had to be very specific about how to accomplish a task.

With system two models, we didn't need to do that. We needed to strip all that out and just specify what to do. Related, the second point of making them leaner, but specifically system one models, you often needed to repeat a lot of the instructions to make sure it followed those instructions as closely as possible.

With system two models, those could also be removed. And then the other idea is just fundamentally unblocking the model. System two models don't like to be given too many constraints. They like to be given a clear objective and then given time to sort of think through all of the permutations and rationalize and reason about the context you've given them in order to find the appropriate plan and ultimate end result to feed back to you.

We often, you know, we love system two models, but system one models can be really helpful. They're often cheaper, they're often faster, but one extra piece is that they often have thought tokens that either we embed those thought tokens to users, because especially in our domain, real estate lawyers would like the added explainability of understanding sometimes how the model has thought about a complex legal matter.

Or second, we use them for debugging that when something isn't working, based on our own experimentation or based on feedback from users, we can then dig into those thought tokens and see what might have gone wrong. The other piece is software developers have been really aware of feature flags over time to sort of progressively roll out new functionality to mitigate risk.

The same concept can be applied to rolling out new upgrades to AI models, but I think there's the change of version bias, this idea that there is inherently more anxiety to the system you're moving to than the system you were at.

Just because the system you were at, people knew it. Even if it had substantial downsides, you were aware of those. But when you move to a new system, it could have the same downsides, it could have other ones, but there's a natural anxiety that's heightened by moving to one of these new models.

And so this idea of sort of poking the bear, you may end up being fine if nobody knew that you were moving to a new model, but simply by articulating that there is a new model that you're moving to, everybody's on a heightened state of awareness to try to look out for issues, and those issues can sometimes outweigh the positives of moving to the new model.

We created a team mantra about betting on the model. A lot of people have talked about this. Do not build SaaS software today for features that will only exist today. Try to imagine where AI models are going to be three, six, twelve months from now, and bet that they're going to get smarter, cheaper, faster, and have more capabilities.

If you betright, you can build features that as AI models become more capable, they make your product better, and you're not stagnating by building stuff today, but you're growing with those models over time. And so by creating this mantra, we were able to talk about it in terms of meetings to just sort of short-circuit some of the thinking to be like, hold on, are we actually thinking correctly about this product feature, and can we think about it a bit differently if we think into the future as opposed to the now?

The other hack we have is using system two models to help us migrate prompts. A lot of the prompts have been created for a world with prior models, but when the latest model comes out, it knows itself, it's often better than prior models, and by feeding that domain-specific prompt into the new model, it can help us migrate those prompts and radically decrease the amount of manual human effort required.

We also need to make some pretty tough calls. Life is uncertain. You don't know what exactly is going to happen with this probabilistic model with new capabilities. Your users are going to feed it the same things they've always fed your system.

They're going to feed it new use cases and new prompts. And so ultimately, there's a lot of uncertainty embedded in moving from model to model. Ultimately, we found that you need a person, a team that is able to take on that risk, that there will be positives and negatives of moving to the new model, but you need to make that call, you need to ship, and then deal with the consequences after shipping.

Obviously, mitigate a number of the risks along the way, but there's this inherent anxiety, as I mentioned before, that you need to get over, and sometimes that just takes a little bravery to say, let's ship, it's going to be allright, let's go.

It also really helps to make sure that you have a strong feedback loop, whether this is manually given to you from your users, whether you can bake this into the product's UX via a simple sort of thumbs-up, thumbs-down feedback.

Get that feedback in the hands of your AI engineers and your domain experts as soon as possible. We've built a system today that feedback comes to us through the product. It immediately gets sent to a domain expert, and that domain expert evaluates it, figures out what prompt needs to get changed, makes that change, and then ships that into production to not only fix that for that user who gave us the feedback, but also fix it for any other user who is on the system.

And we can do that sometimes in minutes or hours, as opposed to the days or weeks-long cycles that are sometimes more traditional with bugs coming in. Demis Hassabis, the chief exec of Google DeepMind, put his finger on something that I think is really important in the Hard Fork podcast the other day, and I'll let him talk through it in his own words here.

One of the challenges, I think, of this space is obviously the underlying tech is moving unbelievably fast, and I think that's quite different even from the other big revolutionary techs, internet and mobile. At some point, you get some sort of stabilization of the tech stack so that then the focus can be on product,right, or exploiting that tech stack.

And what we've got here, which I think is very unusual, but also quite exciting from a researcher perspective, is that the tech stack itself is evolving incredibly fast, as you guys know. So I think that makes it uniquely challenging, actually, on the product side, not just for us at Google and DeepMind, but for startups, for anyone really, any company, small and large, is where do you, what do you bet onright now when that could be 100% better in a year, as we've seen.

And so you've got this interesting thing where you need kind of fairly deeply technical sort of product people, product designers and managers, I think, in order to sort of intercept where the technology may be in a year. So there's things it can't do today, and you want to design a product that's going to come out in a year, so you've got to kind of, you've got a pretty deep understanding of the tech and where it might go to sort of work out what features you can rely on.

And so it's an interesting one. I think that's what you're seeing, so many different things being tried out, and then if something works, we've got to really double down quick.

I think there's a real opportunity here for the AI engineering community. We used to talk about this with software engineers, that they were sort of product engineers and then engineers, and if you could be a product engineer who sort of understood the customer problem, it was incredibly powerful to sort of connect those dots, both the technical and the user need.

And I think AI engineers who are more sort of product AI engineers, the sort of bottoms-up understanding what capabilities these models have and turning them into product features that solve very real user problems. That connective tissue, the least amount of people, if it's embedded within one single AI engineer or a team of them, I find that to be an incredibly promising proposition for the future of the AI engineering community.

Last section, let's talk about paying the Prompt-Tax, or actually, given that this tension exists, maybe we should just ship now. Let me dig into it with a series of questions. Ultimately, the meta question I have is what gives us more confidence to ship at the frontier of this AI?

As AI is moving faster and faster, more functionality is coming out, everybody's agentic product surface area is getting bigger and bigger, having more confidence over time is going to enable us to continue to innovate. We've built our product for the last 18 months, mostly on vibes, but I'm well aware of this idea of will vibes ultimately scale as the product surface area increases.

At the moment, it's doing well between a combination of domain experts and user feedback that comes in real time and that we have tooling to act on it very quickly. That's scaling incredibly well, but at some point, I can imagine that becomes harder or breaks down.

And so the question is, is an eval system the answer to that? Maybe. Maybe it alleviates that. Maybe it bends the curve down and allows us to sort of push even further. But given all the edge cases, given the randomness or the sort of underlying probabilistic nature of these LLMs, and in our case, in real estate legal, we have not only the correctness of the answer, we have the style of that answer, we have the conciseness, and we have all of the correctness of all of the citations.

That is quite a complex number of things to embed in an eval system for all of the different prompts and use cases that real estate lawyers throw at our system. And I can imagine your own agentic systems, if you had to evaluate all of the different edge cases opposed to just the happy paths, it becomes prohibitively expensive, prohibitively slow, and it might even be a bit of an impossible task to create that many evals as your product velocity is growing over time.

The progressive delivery, maybe that's the way forward. Back to the Prompt-Tax idea, upgrade now and fix on the fly. Maybe fixing on the fly can be done by rolling this out internally first, then to a limited number of sort of progressive users and scaling that up to 50 or 100% of users over time.

Maybe it's calibrated based on the amount of feedback that comes in. You just incrementally turn it on for more and more users up until your own internal teams get swamped with too much feedback and then you dial it back and you play that game all the way until you get to 100% and feedback is fairly minimal that needs to end up changing the product over time.

There's probably a myriad of other tactics I either haven't covered here or that you've experienced yourself. I guess this is a call out to the AI engineering community of just what other battle-tested tactics are there? Which ones can we add here?

Feel free to reach out to me and let me know, and maybe we can collaborate together with them on a future presentation and share them with the wider community. So for me, the central thesis of this presentation, this idea of buy now, if you want to stay on the edge of the AI frontier, if you want to maximize the opportunity from all of the capabilities that these new models enable, you need to ship, you need to get that in the hands of your agentic product, you need to get that in the hands of users, and that anxiety may not be real.

You may not have to pay for any of the downside risk that doesn't occur. Or maybe you do. Maybe that progressive tactic of sort of incrementally rolling it out, fixing some feedback for one or two users before it rolls out to hundreds of thousands or tens of thousands is the way forward.

But very much for me, my emphasis, I want to stay on the frontier. It means I need to buy now, and whether I pay for it not later is still to be determined on a case-by-case basis. So that's the end of my presentation.

Big thank you to my team. Many of these ideas come straight for them, so thank you very much. And finally, I'm Andrew Thomson, CTO of Orbital. Feel free to email me or reach out to me on LinkedIn. I'd love any feedback about this talk, good or bad, and continue the discussion.

And maybe, just maybe, once we've built our eval system, we roll that out, I will have another presentation all about the trials and tribulations of that process and whether it's solved a number of the things I've raised here today.