Stop Ordering AI Takeout A Cookbook for Winning When You Build In House - Jan Siml

Many teams run their AI strategy the same way they order takeout: pick something that looks good online and pay the premium, only to discover later that it's half-warm, nothing like the photos. Why? The expectations are too high. Scroll LinkedIn or Twitter and you'll see the recommended prefix menu, with multi-agent graphs and bleeding-edge models.

That might make sense when you're cooking for millions of customers, but inside your company, that's like paying for truffles to garnish your instant noodles. We followed a different recipe, and it delivered millions of dollars of revenue. Today I want to share it with you.

So first the proof, and then the recipe that got us there. In Q1 2024, we faced the classic dilemma: build or buy. We chose to build. Two devs, and roughly 10 plus sprint weeks of effort. Later, we had a system that resulted in several million dollars of ARR and a group-level award.

I can't share the exact numbers; legal would satan me, but imagine a number big enough that finance stops asking why build. Let me show you why Twitter's recipe would have killed us. Giant evals, multi-agent systems, RFPed models. It looks delicious, costs a fortune, and delays a launch.

It's perfect for flashy SaaS demos, but an overkill for your in-house needs. But the question really isn't what to skip; it's when does it ever make sense. Think of it this way: SaaS is a hotel buffet. It's generic, but safe.

Your internal users want the grandma's secret sauce, not the hotel eggs. So let's see where that home kitchen crushes the buffet option. SaaS shines when you need vendor integrations or cross-industry best practices. Our kitchen wins when we already own the battery—I mean, the data.

Our colleagues know the exact keystrokes needed to close the deals, and we can involve them in double-checking the outputs. And the advantages don't stop there. Because we sit next door to our users, so a tweak ships the same days, and the UI speaks their language.

And the compute that runs on infra, we mostly pay for already. It all basically drops the cost to pennies. So the rule of thumb is: buy SaaS to explore the unknown, but build in-house once the workflow is yours.

Okay, so pantry's stocked, knives are out. Here are the five lessons.

They work best as a set, so we will unpack them in order, starting from the foundation. This is where in-house crushes SaaS. You can go absurdly deep on one painful job to be done without chasing total addressable market.

Pick something where you can easily pinpoint the value event, which is that dollar-based outcome that you're doing it all for. Remember that one? It's going to be important later.

We started with a simple sales alert use case and grew it from there. By going deep on that one use case, it was albeit much easier. We didn't stop at the alerts. What were they for? What else needed to be done?

All those were questions we had. And who knows best what's needed? Your users. Talk to them to really nail it. If you stay focused, you can keep things very simple and avoid anything agentic.

So metrics coming next. Spoiler: offline evals never sign a contract. Nobody at the board meeting asks for your F1 score or NDCG. They ask, "Did it move the revenues?" Don't get me wrong, evals are important, but they are smoke alarms.

You need to track the actual money. So instrument everything until you can say, "This AI task, let's do $20 here." Build your revenue funnel. Everything from beginning to end, to that value event that we talked about earlier. By the way, your users are your guardrails, so you can run ambitious experiments.

Don't overthink your evals.

And once you link your system to dollars, decisions and prioritization become a breeze. The conversation shifts to, "What's your idea, and how much would you sell with it?" And here is where it gets interesting. Managers will start asking for team performance reports.

You should automate them, but also prepare the leaderboards, because those can really create that healthy competition, get leadership invested, surface some champions, but also help those who might be silently struggling with the new workflows. So now that you're tracking the dollars, don't wait for users to come to you.

You need to become the chef who anticipates what the next dish should be. Because the best UI is the one you never need to use. This is your business, so you know what the next steps should be. So why wait for users to ask for it?

Just do it for them. In our case, we built a motion to send daily digests. Here is what you need to know today. We still had shadow UI, but it was the fallback for all the unexpected and unplanned tasks that came along.

So what's next? Now that we've stopped waiting the tables, now it's time to turn those freed up minutes into money on the register.

Your AI system needs to guide action, not just deliver information. Why? Because saving 30 minutes is worthless if users just fill it with an email sludge. So the real power here comes from you actually converting the time saved into time well spent.

Because you know what the highest value activities are, and the more you start building up those revenue funnels, the more you will start to understand where to divert that free time and your users' attention. Our proactive system was a hit.

It was surfacing things users wouldn't have thought of doing. And compared to the chat app, it had 20 points higher NPS and order of magnitude higher engagement. So you're making some money now, but it brings us to a critical decision point: where to invest your limited development resources.

You might not like the answer.

Good data consistently beats great models. This is the secret that you won't find on Twitter. We all love shiny things, but o3 is 60 times more expensive and order of magnitude slower compared to 4.1 mini. So the biggest impact, if you put it in production, will be on the cost.

We've seen the best results from simply adding more triggers to alert the users on, and going deeper into what they needed. Boring,right? But it worked. When we changed the models from normal to the mini-series and back, the only thing that changed were the costs and the evals.

So you need to build for what your users need, not what you want to try. When you focus on what users truly value, instead of simply chasing model benchmarks, something magical happens. A powerful flywheel begins to spin. Because

those tight feedback loops make users feel heard. So they start providing you with ideas for improvements. So you can run weekly experiments based on their feedback, which drives even more adoption, which generates more data for prioritization and more ideas.

The revenue flywheel starts spinning faster and faster.

So that's it. Let's recap what we've covered so you can apply it immediately.

Focus on one painful job to be done that has clear dollar value. Don't try to bowl the ocean with a comprehensive solution. Revenue impact trumps evaluation metrics. Track everything to the final dollar-based outcome and make decisions based on that.

Push insights proactively rather than waiting for users' questions. And simply being proactive isn't enough. You need to ensure that those time savings are channeled into the highest value activities you can find. And invest in the basics. It really pays off.

So in short, start small, follow the money, and let your users guide you. Thank you.