Substrate Launch: the API for modular AI

Yeah, it's really good to be here. Um, this is a particularly exciting talk for us because we've been working with private clients for about a year now, but this is the first time we've really talked about it in public, um, since our launch last week.

I'm incredibly proud of the work we've done so far, and I'm excited to take a few minutes to tell you about it.

So if you look at the products out there that have really successfully leveraged this generation of AI, I think one thing is true about nearly all of them is that they're using more than one inference run, often many different types of models in tandem to accomplish a specific kind of task really well.

And I think people have really quickly realized that the foundation model is not enough, and even very simple tasks like summarizing a document to much, much more complex tasks like solving coding problems end to end, I think the best productsright now are all using systems of inference runs in a logical structure.

So I think at Substrait, we believe that building with modular intelligence is always going to be more effective than building with a monolithic intelligence. Um, these systems are inherently more legible, which means you can understand them structurally, which means that they're debuggable and they're extensible.

And evals become a lot easier because the decision trees are explicit, and you can sort of verify at every step what's going on and what's going wrong. Um, so

Substrait, I think, is a sort of new way, uh, new approach to this. Um, I think our model is sort of fast in ways that other paradigms can't be. It's sort of flexible enough to build any AI product out there, and it works at scale by default.

So what is it? Um, I think at its core, Substrait is a coupling of two things. First, I think it's a really elegant developer SDK that lets you describe a computation graph over any number of nodes. Um, and the abstractions here are, are really general, and so we have, we have a bunch of intelligence nodes across all the modalities that you might care about, which is like generating images, transcribing speech, generating text, JSON, embeddings, executing code.

Um, but second, Substrait is also an inference engine specifically built to run these computation graphs as efficiently as possible. Um, so these graph representations here, um, are it's a representation of many tasks and their relationships. And since we run a very coordinated compute cluster, um, we can statically and dynamically optimize things like batching, caching, sort of networking, concurrency, physical placement, um, which really makes a big difference.

Uh, and if you look at most frameworks out there, um, they're typically involving dispatching a bunch of API calls separately. And if you look at what happens mechanically when you do that, it's every step means you've got to resolve DNS, you've got to go through proxies, you've got to do authentication, like balance checks, um, and all of that sort of adds hundreds of milliseconds of latency on every single step.

And if you contrast that with Substrait, we, we transfer data from node to node, process to process, on the order of microseconds, whereas it's some 10,000 times faster, meaning that it's actually feasible now to run online applications that involve dozens of nodes.

Um, we've also noticed that JSON decoding is, is one of the most useful patterns for multi-inference runs. And I think we've invested a lot into offering a, a best-in-class, um, JSON mode, both in terms of reliability and speed.

And if you look at all of this together, I think what it means is that Substrait is, is, is really a way that, way to enable higher quality outcomes with AI, letting you work in a system that's more flexible, it's more legible, it's more verifiable than any of the current paradigms that sort of exist now.

Um, I think there's a lot more to say. There's all the time I really have today. It's only five minutes. Um, but if you're curious, um, please come out and say hi on the expo floor. Um, you can scan this QR code we and get some credits, um, and go to the website substrait.run, um, or give me an email at, uh, rob@substrait.run.