The Rise of Open Models in the Enterprise — Amir Haghighat, Baseten

Yeah. Hi everyone. My name is Amir, I'm co-founder and CTO of Baseten, the inference company, but I'm not here to talk about Baseten. I'm here to talk about the adoption of AI in enterprise, why we should care about it, and how it's going based on what we've seen.

So first, why we should care about it. Ultimately, we've heard this before as like, "Hey, is there a hype in the market?" You know, "Is AI hyped?" And like, it probably is. But the evidence that a lot of people point to, that there's hype here, is that the adoption in enterprise has been slow.

You know, I've heard this so many times that, like, you know, enterprises are slow to adopt, and that has an implication. If that is true, that has implications for, really, the impact of AI and how large it can be, and whether it is truly a hype or real.

And the reason is that enterprises are massive, like their reach is massive, they have all the money, and if they're slow in adopting, then that paradigm shift that we're talking about, that will be slow to materialize. So I'm here to tell you, based on what I've learned about adoption of AI in enterprise, why me?

Because we happen to sit somewhere interesting. We happen to sell to enterprises. And so over the past, the company is 60 years old, but over the past two years, you know, in particular, I've talked to, honestly, 100-plus enterprises, from software companies that are public to literally soft drink companies that are, you know, Fortune 50.

And I've seen patterns that I want to share those patterns with you. One bias that I have is that I don't sell a verticalized AI tooling. I sell a very horizontal AI tooling. And this is important. So enterprises are adopting vertical solutions.

You know, AI for sales, AI for marketing, AI for customer service. You just heard from Clay from Sierra. That adoption is happening. But I think for the true value to get unlocked, we need to see enterprises actually build with AI.

The analogy that I use is that if in the 2000s, enterprises were not really building tech themselves and were just buying Salesforce or, you know, products like that, verticalized products like that, then the tech industry would just not be as big.

Companies like Snowflake and Databricks and Datadog would not exist or would not exist to the shape that they do. And so I really think that the value is ultimately unlocked once enterprises feel comfortable to actually build with AI themselves as opposed to just buy verticalized tooling.

So let's talk about the journey, the journey that they go through.

They all start with OpenAI and Anthropic. You know, enterprises, they're like us, you know, for good reasons. It's just so easy to get started. It's just they do it differently from the rest of us in that they have their own dedicated deployments of these models on Azure or AWS

and for reasons around security and privacy and all that. And then they get their engineers. A lot of times they're more, you know, predictive ML teams to become AI teams and build on top of these. And they're happy with that.

And if they can continue doing that, they will. Because there's just a lot of inertia in sticking to that if it actually works. Sticking to closed models, which is so easy to use, API-based, you know, build on top of it.

But we're seeing cracks in that assumption. And so let me tell you what I've seen going back in time and how that's changed over time. So in 2023, I remember, like, going and trying to sell to enterprises, and the terms toying around came up quite a bit.

I heard this actually literally from the CIO of a massive insurance company back then. It's like, "Yeah, we put a dedicated deployment of OpenAI, of GPT-4 or GPT-3 so that our engineers can toy around with it." Kind of like almost dismissively talking about it.

It's like, "Hey, go build something cute." That started to change. In 2024, we saw actual production use cases, again, built on top of these closed models. I would say like 40, 50 out of the 100, like, had something in production in that year.

And then in 2025, this year, something changed. And it's palpable from, at least from where I'm sitting. And the change is that there are cracks in that assumption. There are cracks in the assumption that we can actually build on top of these closed frontier models indefinitely.

So what are those cracks? I'll tell you what the cracks are not. And there's some misconceptions over here. I'll tell you what they're not. So like, often people say, "Oh, because enterprises don't want to have vendor locking." I don't hear that, honestly.

Like, we go and talk to them. I think I know why. It's because, one, there's a few of them now. You know, you can go OpenAI, Anthropic, Google has been coming up pretty well. They're somewhat interoperable at a certain level.

Like, they all use OpenAI specs. So like, building on top of them, yeah, you might have to, like, do your evals again and do some prompt tuning. But generally, you can go from one to the other. So vendor lock-in is not something that I hear about.

Ballooning cost. I didn't hear that last year. And I know why. It's because when I ask them, they say, "Look, the price per token is plummeting." And we just talked about this, like,right before this. And that's they were saying, "That problem will just take care of itself."

Compliance, privacy, security. Also not problems because these frontier model companies kind of take care of that with the help of the CSPs, with the help of the cloud providers, so that these models are running in a dedicated way inside of their existing VPCs.

If these aren't the cracks in that assumption of just use closed models, then what are the cracks? These are the reasons that I have seen. And I'll go through them one by one and go through, like, examples of these as well.

And then at the end, also talk about, you know, if these are the cracks, then, you know, how do you get around them and, you know, there be the dragons. So one is around quality. Look, none of these enterprises are in any sort of misconception that they can build the next GPT-4 better than OpenAI can.

That's just not the reality, not as a general model at least. But for specific use cases, for specific tasks, we are seeing this where the frontier models are not necessarily theright tool. So the example, I've seen this in a couple of big health plans, is that they want to do medical document extraction.

So they have millions of medical documents, prior auths and medical claims, and they're trying to get, you know, CPT, like, procedure codes and diagnosis codes and prescriptions. And just giving that to, you know, Claude or GPT doesn't do it.

But they have the data. Over the years, they've collected a lot of labeled data, and they're like, "Oh, we can do better." And so that's, and they actually did. That's one example. Another example is on the voice side, in particular on the transcription side, like, again, staying in the healthcare space, like, understanding medical jargon or, like, you know, getting transcription models to understand medical jargon.

That has been another reason to not just use an API-based generic model, but in-house it and do better than what they can do with just API-based models. Another one is around latency. Look, these models, OpenAI, Anthropic, even, you know, the big players that serve open source models behind shared APIs, inherently, they're optimized for high throughput and high QPS at the expense of latency.

But a lot of times we're seeing more and more now where latency is becoming very critical, especially when, you know, the AI voices or AI phone calls, latency starts really mattering. Time to first token, time to first sentence really starts mattering.

And you have to just think about things differently. You can't just use the frontier models as-is because, again, they're optimized for something else.

Around the unit economics, there is, again, like I said before, like, pricing, they said, "This will take care of itself." Then came this year. And as you saw in the previous talk from Michael, the agentic use cases ballooned.

And when they balloon, it's crazy. Like, I've seen this. Like, every single user action can result in literally 50 inference calls. And so suddenly the thing that you thought is going to take care of itself is not taking care of itself.

That costs are really ballooning. And enterprises think that maybe they can do better on the cost and unit economics in order to show ROI, in order to show that the solutions that they're pushing are economically viable, they need to show, they need to reduce their costs somehow.

And they're realizing that they can actually run these models and pay for the compute and have that be a lot cheaper than paying per token and covering all the margins of someone else. And really going from being a price taker to actually be the maker of the price and being in control of that.

And then lastly, destiny. This one is a bit vibey, but I'm hearing it more recently that, you know, some CIO, CTO saying, "If we, the enterprise, use just the frontier models and so do our competitors, what is our advantage?

What is our alpha?" And maybe we should bring in some of these things in-house and to be able to even differentiate, not just at the workflow and application level, but also at the AI level. So now what? If those are the reasons why they want to adopt open source models and iterate on those and build those and fine-tune those and distill those, then what changes?

Well, what changes is that they go from a super simple world that just call an API and run with it to now you need to build inference. You need to build inference infra. And you need to make sure that it scales well.

And you need to make sure that you can move fast, that your engineers can actually deliver instead of having to, you know, hire a bunch of new types of people and then wait for them for a long time to actually build this infrastructure in-house.

So one thing that I hear quite a bit at this point is, look, I've heard this from enterprises. I hear this from startups too, actually, which is that, look, you know, we've picked a model, an open source model.

We've heard of ELM or SGLang or TRTLM. We have some GPUs in the case of enterprises, it's in their data center, in the case of startups, it's in some cloud. And you put these together and you get production inference.

And I know for a fact that this is not true. I wish it was true, but I know for a fact that this is not. That there's a lot more that goes into making inference, especially mission-critical inference, work well inside of your company.

So what are those? These are the dragons. These are the dragons. So

one, at the performance layer, you know, we talked about, you know, situations where things are very latency sensitive. The way that you optimize models for latency is actually quite involved, both at the model level and at the infrastructure level.

You have to do it. You have to attack it at both levels. So as an example, you know, for at the model level, it's like, "Hey, do you use speculative decoding? And if so, which route do you go?

Do you go with a good draft model? Do you go with Medusa heads, with Eagle 3? Do you go with MTP?" There's a lot. And new techniques are coming out all the time. Like, the Eagle 3 paper came out like six months ago, and it's like running in production and actually being very meaningful.

And so as an enterprise, can you hire theright folks to be able to be on top of the research? Because you don't, these are not just, you know, switches that you flip in SGLang or VLM and get the result.

Some of these optimizations bleed out of the model level into the infrastructure level. So as an example, if you want to be able to do prefix caching really well, if you want to be able to desegregate a serving really well, because that starts really mattering, especially in agentic use cases where, like, the prompts are massive, but the prompts are somewhat similar from one to the other, it ends up mattering a lot.

And you hitting your, you know, time to first token and you hitting your p99 of that in a reliable way. Another thing on the infrastructure is, especially if it's a mission-critical inference, which more and more I see that that's the case, how do you guarantee four nines?

And with this, this formula does not guarantee you, you know, more than two nines. And I saw this firsthand. So how do you make sure that when the hardware fails underneath it, you actually recover? How do you make sure that, you know, when VLM crashes, which it happens often, I saw this firsthand with, like, when Triton crashes often, your tail latencies go through the roof while you wait for these things to come back.

And during that time, like, your users are feeling that. How do you build against those and make sure that, you know, you can still guarantee four nines and not be super over-provisioned and mess up all the unit economics that we talked about?

When a big burst of traffic comes in, how do you make sure that you scale up fast? How do you make sure that, you know, I was talking to this massive enterprise soft drink example where they're like, "Yeah, it takes us eight minutes."

When you want to bring up a new replica of the same model, it takes eight minutes. And I believe that because if you add up all the different things that it goes into doing that, that is how long it takes.

But that's not okay. Again, your tail latencies go through the roof as soon as there's a big spike of traffic. How do you account for that? And then there are other things around, again, like, making sure that your engineers move fast, the tooling, lifecycle management, the observability, massive

iceberg, which is like, it's like, "Oh, yeah, I just put some logs and metrics," and you realize there's a lot more to do underneath it. We're just, previous talk, Michael talks about. And then lots of stuff around controls and audits and things that enterprises actually care about.

So these are the dragons. These are the dragons. And these are the times that then enterprises have a decision to make. Like, once they get to this level, either, like, I tell them or, and they have to believe me, or they don't and they go and build it.

And then they see some of these things, they have a decision, which is build or buy. And that's my job to then try to convince them that I think they should buy this layer of infrastructure and platform as opposed to build it.

And that's sometimes harder than it seems. So I'm happy to talk more about these things. So I'll be at our booth, like, two topics that I'd love to talk about if you're interested. One, self-servingly, if you're an enterprise and those problems resonate, I'd love to chat with you about.

And two, less self-servingly, if you're a startup and you're trying to sell to enterprises, I'm happy to chat about all theright decisions that we made, the wrong decisions that we made along the way to build something that then, when it comes to selling to enterprises, when it comes to deploying it into their own clouds, that is actually possible and not a massive other set of dragons over there.

And then last thing, we have a happy hour. We'd love to see you there. Thank you.