# Substrate Launch: the API for modular AI AI Engineer · 2025-02-06 Rob Cheung, co-founder of Substrate, presents the platform as a new approach to building AI applications using modular intelligence rather than monolithic models. Substrate combines a developer SDK for describing computation graphs over multiple AI nodes with an inference engine optimized for running these graphs efficiently, reducing latency between nodes to microseconds—10,000 times faster than typical API dispatching. Cheung argues that modular systems are more legible, debuggable, and extensible, enabling higher quality outcomes through explicit decision trees and easier evaluation. The platform offers best-in-class JSON mode for multi-inference runs and supports modalities like image generation, speech transcription, text, embeddings, and code execution. Substrate is designed for developers to run 10-20 AI modules on a single user request at scale, solving the infrastructure problem that currently limits multi-step AI workflows. ## Questions this episode answers ### How does Substrate make it possible to run 10-20 AI modules on a single user request? According to Rob Cheung, the main bottleneck is infrastructure overhead: each separate API call adds hundreds of milliseconds for DNS resolution, proxies, and authentication. Substrate solves this by running a coordinated compute cluster where data moves between processes in microseconds—roughly 10,000 times faster. This eliminates per-step latency and enables online applications that chain dozens of inference runs without slowdown. [0:03](https://aie.addtry.com/4cad63a9-0536-43e8-ade8-1407b7e6eceb?t=3420) ### Why does Rob Cheung say that modular AI is better than using one large model? Rob Cheung argues that building with modular intelligence yields systems that are more legible, debuggable, and extensible. Because the decision trees are explicit, evaluations become easier at every step. He contends that the best AI products already use multiple models in tandem, and that a modular approach will always outperform a monolithic one due to this structural clarity and verifiability. [0:01](https://aie.addtry.com/4cad63a9-0536-43e8-ade8-1407b7e6eceb?t=1180) ### What exactly is the Substrate SDK? Rob Cheung describes the Substrate SDK as an elegant way to describe a computation graph over any number of nodes. It provides general abstractions for intelligence nodes across modalities—generating images, transcribing speech, generating text, JSON, embeddings, and executing code. This SDK lets developers define how these tasks relate, forming the logical structure that Substrate’s inference engine then executes efficiently. [0:02](https://aie.addtry.com/4cad63a9-0536-43e8-ade8-1407b7e6eceb?t=2070) ## Key moments - **[0:00] Intro** - **[0:37] Modular AI** - [0:37] The best AI products use systems of multiple inference runs in a logical structure, not single foundation models, says Rob Cheung. - **[1:18] Legible Systems** - **[1:52] Substrate Approach** - [1:52] Substrate couples a developer SDK for describing computation graphs with an inference engine to run them efficiently. - **[2:07] Developer SDK** - **[2:41] Inference Engine** - **[3:15] Latency Advantage** - [3:15] Substrate transfers data between processes in microseconds, 10,000x faster than typical multi-step API calls. - **[3:58] JSON Mode** - [3:58] Substrate offers a best-in-class JSON mode for reliability and speed in multi-inference AI runs. - **[4:15] Better AI** - **[4:36] Outro** ## Speakers - **Rob Cheung** (guest) ## Topics Modular AI Orchestration ## Mentioned Fin.com (company), Substack (company), Substrate (product) ## Transcript ### Intro **Rob Cheung** [0:13] Yeah, it's really good to be here. Um, this is a particularly exciting talk for us because we've been working with private clients for about a year now, but this is the first time we've really talked about it in public, um, since our launch last week. I'm incredibly proud of the work we've done so far, and I'm excited to take a few minutes to tell you about it. So if you look at the products out there that have really successfully leveraged this generation of AI, I think one thing is true about nearly all of them is that they're using more than one inference run, often many different types of models in tandem to accomplish a specific kind of task really well. ### Modular AI **Rob Cheung** [0:55] And I think people have really quickly realized that the foundation model is not enough, and even very simple tasks like summarizing a document to much, much more complex tasks like solving coding problems end to end, I think the best productsright now are all using systems of inference runs in a logical structure. So I think at Substrait, we believe that building with modular intelligence is always going to be more effective than building with a monolithic intelligence. Um, these systems are inherently more legible, which means you can understand them structurally, which means that they're debuggable and they're extensible. ### Legible Systems **Rob Cheung** [1:38] And evals become a lot easier because the decision trees are explicit, and you can sort of verify at every step what's going on and what's going wrong. Um, so ### Substrate Approach **Rob Cheung** [1:52] Substrait, I think, is a sort of new way, uh, new approach to this. Um, I think our model is sort of fast in ways that other paradigms can't be. It's sort of flexible enough to build any AI product out there, and it works at scale by default. ### Developer SDK **Rob Cheung** [2:07] So what is it? Um, I think at its core, Substrait is a coupling of two things. First, I think it's a really elegant developer SDK that lets you describe a computation graph over any number of nodes. Um, and the abstractions here are, are really general, and so we have, we have a bunch of intelligence nodes across all the modalities that you might care about, which is like generating images, transcribing speech, generating text, JSON, embeddings, executing code. ### Inference Engine **Rob Cheung** [2:41] Um, but second, Substrait is also an inference engine specifically built to run these computation graphs as efficiently as possible. Um, so these graph representations here, um, are it's a representation of many tasks and their relationships. And since we run a very coordinated compute cluster, um, we can statically and dynamically optimize things like batching, caching, sort of networking, concurrency, physical placement, um, which really makes a big difference. Uh, and if you look at most frameworks out there, um, they're typically involving dispatching a bunch of API calls separately. And if you look at what happens mechanically when you do that, it's every step means you've got to resolve DNS, you've got to go through proxies, you've got to do authentication, like balance checks, um, and all of that sort of adds hundreds of milliseconds of latency on every single step. ### Latency Advantage **Rob Cheung** [3:42] And if you contrast that with Substrait, we, we transfer data from node to node, process to process, on the order of microseconds, whereas it's some 10,000 times faster, meaning that it's actually feasible now to run online applications that involve dozens of nodes. ### JSON Mode **Rob Cheung** [3:58] Um, we've also noticed that JSON decoding is, is one of the most useful patterns for multi-inference runs. And I think we've invested a lot into offering a, a best-in-class, um, JSON mode, both in terms of reliability and speed. ### Better AI **Rob Cheung** [4:15] And if you look at all of this together, I think what it means is that Substrait is, is, is really a way that, way to enable higher quality outcomes with AI, letting you work in a system that's more flexible, it's more legible, it's more verifiable than any of the current paradigms that sort of exist now. ### Outro **Rob Cheung** [4:36] Um, I think there's a lot more to say. There's all the time I really have today. It's only five minutes. Um, but if you're curious, um, please come out and say hi on the expo floor. Um, you can scan this QR code we and get some credits, um, and go to the website substrait.run, um, or give me an email at, uh, rob@substrait.run. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.