# [Workshop] AI Engineering 201: Inference AI Engineer · 2023-11-07 Charles Frye, instructor of the Full Stack LLM Bootcamp, leads a workshop on AI engineering inference, focusing on the build-versus-buy decision between proprietary and open models. He argues that proprietary models like OpenAI's GPT-4 and Anthropic's Claude are currently more capable but expensive, while open models like LLaMA 2 are less capable but offer hackability, though they may catch up if capabilities requirements saturate. Frye covers inference on end-user devices, noting that running models locally avoids network latency but faces tight memory and power constraints—e.g., a 7B parameter model requires 14 GB, too large for phone RAM. He explains inference-as-a-service (e.g., OpenAI, Replicate) versus self-serving on cloud GPUs or serverless platforms like Modal, highlighting that memory bandwidth is the bottleneck: GPUs have 1.5 TB/s memory bandwidth vs 312 TFLOPS compute, so batching is crucial for throughput. Frye also discusses inference arithmetic, custom silicon like TPUs which offer ~30% better efficiency but not drastic gains, and containerization challenges for GPU workloads. ## Questions this episode answers ### Will open-source LLMs eventually outperform proprietary ones like GPT-4? Charles Frye argues that open models trail proprietary ones in raw capability now but are rapidly closing the gap on leaderboards. He suggests the outcome hinges on whether capability requirements saturate: if users stop demanding ever-smarter models, open models could catch up and dominate, much like Linux did in operating systems. But if the appetite for intelligence remains unbounded, proprietary models will likely keep their lead due to the huge costs of pushing capability boundaries. [30:40](https://aie.addtry.com/f536d5de-abbe-456b-b87f-a73ce2f89a07?t=1840000) ### Why does serverless GPU inference for LLMs have long cold-start times? Charles Frye explains that serverless GPU inference struggles with cold-start latency because when a function scales from zero, the model weights must be loaded from disk into GPU memory. Moving up to half a terabyte of data can cause cold boots of 30 seconds to 3 minutes. This delay is a major obstacle for achieving low-latency responses expected from serverless platforms. [1:15:00](https://aie.addtry.com/f536d5de-abbe-456b-b87f-a73ce2f89a07?t=4500000) ### Why is batching critical for GPU inference of large language models? Charles Frye details that GPU inference is memory-bandwidth constrained: an A100 GPU can perform 312 teraflops per second of math but only reads memory at 1.5 terabytes per second. Since each weight is used once per input, you need to process many inputs together—a batch size around 200 for the A100—to amortize the slow memory reads and fully utilize the compute. Batching greatly increases throughput, and the optimal batch size depends on the GPU's memory-to-compute ratio. [1:21:03](https://aie.addtry.com/f536d5de-abbe-456b-b87f-a73ce2f89a07?t=4863000) ## Key moments - **[0:00] Intro** - **[3:52] Inference Defined** - [4:06] Generative model inference is a data-to-data function mapping human-interpretable inputs to human-interpretable outputs. - [5:34] The neural network's tensor-to-tensor mapping is the compute bottleneck of inference, not tokenization or sampling. - [8:56] Foundation model teams generate weights; AI engineers can now skip training and focus on application. - **[10:16] Model Options** - [10:16] Top 5 models on the LMSIS leaderboard are all proprietary from OpenAI or Anthropic. - [13:52] Anthropic's Claude instance costs $1 per million tokens; Claude 2 costs $10 per million tokens. - [14:47] Proprietary models are disincentivized from giving log probabilities to prevent reverse-engineering. - [16:53] Enterprise tier offers a white-glove service with built-in security, SOC 2 compliance, and SLAs. - [18:13] Enterprise tier provides an SLA, higher rate limits, and fine-tuning controls not in the public API. - [19:34] Fine-tuned models cost more to inference because the provider cannot amortize the cost over many users. - **[30:40] Future of Models** - [30:40] Like operating systems, open and proprietary models will coexist; openness wins if capability requirements saturate. - [34:33] If AI capability requirements ever saturate, open models will dominate, similar to Linux in data centers. - [36:06] Q: What language support do these models have? A: Quality scales with available internet text, giving English an edge. - [38:15] ChatGPT's safety guardrails can be bypassed by prompting in low-resource languages. - [40:10] Q: Should AI engineers still learn deep learning? A: Yes, organizations need someone who understands underlying models. - [42:43] Vertical-specific foundation models risk being outperformed by broader generic models and narrower domain-specific models. - **[44:12] On-Device** - [44:31] Running inference on-device eliminates network latency and compute costs but sacrifices control and telemetry. - [45:08] 'I've run LLaMA 2 13B on this laptop without it catching on fire.' - [45:55] A network roundtrip from California to Europe takes 150ms, consuming a third of the 400ms Doherty threshold for interactivity. - [47:35] If end-users run inference, your compute costs become $0. - [48:28] Google's Pixel 8 Pro now includes on-device large language models for summarization and image editing. - [49:43] Phone RAM is a few gigabytes, while a 7B parameter model needs 14GB, making on-device inference challenging. - **[1:04:32] Inference as a Service** - [1:04:45] Running AI in a data center trades simplicity and control for latency, but is where most applications start. - [1:07:49] Proprietary models are only accessible as inference-as-a-service, while open models can be served via Replicate or self-hosted. - [1:08:47] Inference services for open models struggle to offer user control while amortizing costs across many users. - **[1:10:00] Self-Serve Inference** - [1:10:21] Specialist cloud GPU providers like Lambda Labs often offer cheaper instances than AWS, GCP, or Azure. - [1:12:00] Serverless GPU providers offer scale-to-zero pricing, eliminating costs during idle periods but with cold-start latencies. - [1:15:48] Serverless GPU cold starts can take minutes due to loading multi-terabyte model weights into accelerator memory. - [1:18:07] Running inference on owned hardware in co-location can be cheaper long-term but requires large upfront capital and rare skills. - **[1:20:12] Hardware Accelerators** - [1:20:43] GPUs are ideal for inference because they perform many math operations per memory read, amortizing weight loading costs. - [1:22:47] A100 GPU provides 312 teraflops of FP16 compute but only 1.5 TB/s memory bandwidth, creating a bottleneck. - [1:25:07] Achieving high inference throughput requires batching multiple requests to amortize memory reads across many input tokens. - [1:26:01] Low-latency APIs favor balancing memory and compute; batch jobs maximize batch size since latency is unconstrained. - [1:27:07] TPUs provide about 30% better performance per watt than GPUs for neural network workloads, not 10x. - [1:28:27] TPUs' main advantage is operational and power efficiency, not raw computational performance. - **[1:36:11] Inference Deployment** - [1:36:33] Containerizing GPU inference is problematic due to multi-terabyte weight images and CUDA driver dependencies leaking from the host. - [1:40:13] Self-serving inference requires MLOps to be a core competency, often built on Kubernetes and open-source tools like Ray or Seldon. - [1:41:01] Managed MLOps options include SageMaker, Vertex AI, Bento Cloud, and Anyscale, offering varying levels of abstraction. ## Speakers - **Charles Frye** (guest) ## Topics Inference Optimization ## Mentioned Anthropic (company), Google (company), Hugging Face (company), Meta (company), Modal Labs (company), NVIDIA (company), OpenAI (company), Replicate (company), A100 (product), ChatGPT (product), Claude (product), Docker (product), GPT-4 (product), H100 (product), Jetson Nano (product), Kubernetes (product), Llama (product), Mistral (product), PaLM (product), Ray (product) ## Transcript ### Intro **Charles Frye** [0:18] The key lesson in the workshop today was everything that comes after once you've, you know, written your thin wrapper around the OpenAI API to make your ChatGPT-powered app. You've acquired $100 million in venture funding at a $4 billion valuation, and now you're like, uh, what—what am I supposed to do next? Um, so we're going to—I'm going to split this into two parts so we can take kind of a break. It's like a 3-hour-long workshop. It's a long time. Uh, so in the first half, we're going to talk about, uh, inference. So about the what exactly this workload is that we have given over to the, uh, the Inference-as-a-Service Provider. What, you know, what's the shape of it, um, why do we need these expensive accelerators, what are the other options available. Um, and we're going to spend half of our time on that, because that's the place where we can actually talk, you know, in engineering terms about constraints, uh, about, uh, service-level objectives and service-level agreements, uh, the kinds of things that—that lead to robust systems. Um, and then we'll spend 90 minutes on the rest of the owl, uh, the rest of what it takes to make a successful AI-powered app, just because there's so, uh, very little to say just yet about how to engineer these robustly. But we'll talk about what the—what the emerging consensus on what to do is, uh, so far, and what tools are out there and available to start accelerating that process. Um, so, yeah, so for inference, what, uh, what are we talking about when we're doing inference workloads? How do we decide between using Open and Proprietary Models, uh, to do that inference? Um, where do those models live? Do they live on a device? Do they live in a cloud server? Um, uh, and then we'll, uh, spend some time talking about what it takes to self-serve inference. Um, for the rest of the owl, we'll talk about architectures and patterns. So what are the emerging kind of, uh, patterns for usage of—of large language models in AI applications? And we'll talk about monitoring, evaluation, and observability, which is how we try and actually improve applications that fit those patterns over time. Um, yeah, any high-level questions? Um, my mic's off. The Zoom is not hooked up. No questions? Probably not. Still at a very high level. Um, no? Just stretching. Great. Yeah, maybe we should all stretch. Um, yeah, so, uh, who am I? And before we dive in, like, who am I and why should you listen to me tell you about any of these things? So, uh, my name's Charles. I like to teach people about AI. I've been doing it for a while now. Uh, I went to Berkeley, studied neural networks back in the 2010s, um, taught people about, uh, how to use them and Bayesian networks, RIP, um, for data science. Um, and then I worked, uh, in developer relations and education for weights and biases, um, a previous generation MLOps tool. Um, generation times being shorter than more doubling times, I guess, at this point. Um, and then for the last 2 years I've been working with Full Stack Deep Learning, uh, teaching not just things like the math of machine learning or how to do monitoring of an ML application, but how to build an application that uses ML from soup to nuts, uh, from the GPUs up to the user experience. ### Inference Defined **Charles Frye** [3:52] Allright. So now, let's dive into the first half here on inference, and specifically what is actually going on, why does it cost, um, so much to, uh, to paying OpenAI. Um, and so generative model inference, the kind of inference that's done, uh, by these—the generative AIs, is, uh, you see when you use it as a via an API, you see it as a kind of data-to-data function, where both sides of that data are—are human-interpretable data. So things like text, images, sounds, and out come back new generated text, image, and sounds. So for example, this is from the Palm ePaper from Google. You might show a picture of a restaurant and the question, if a ro-robot wanted to be useful here, what step should it take? And the output of this, uh, uh, language modeling generative API would be, you know, clean the table, pick up trash, pick up chairs, wipe chairs, put chairs down. Um, which could then be sent to a cleaning robot, and now you've got something, you know, that's a—that's a pretty useful system there. So this is what you see from the outside. Um, but as you start, like, digging in a little bit, you'll start hearing about things like tokens, um, and, and log probes and temperatures. And, um, to understand what's going on and, and weights and networks, and to understand what's going on, you have to realize that this has been broken down into kind of three pieces. One part that goes from what humans understand, like text and images and sound, to what neural networks understand. Uh, and then one part that goes back, uh, and in between the operations of a neural network that operates on arrays and returns arrays. So tokenizers take in text that a human can read and turn it into an array of numbers, a tensor. Um, just due to the kind of physics-y background of people in the ML community, they ca what people would call arrays I guess, in other software engineering communities they get called tensors. They also because they have, um, uh, derivatives attached to them a lot of the time. It's like there's a connection there, but really an array of numbers. Um, those tensors get turned back to things the humans care about. Um, and in between, neural networks map tensors to tensors. This step is the bottleneck. This step is the hard part. There's interesting stuff going on in the sampling process. There's interesting stuff going on in tokenization. There's cursed stuff going on in tokenization. Um, but this step is the bottleneck. This is where the vast majority of the engineering time is spent. This is where the vast majority of the compute time, the memory, are spent. And this is the interesting part, and this is the part where the engineering focus, uh, needs to be. Um, or this is the part that you farm out to somebody else. Um, so diving in, double-clicking on that, uh, tensor-to-tensor arrow at the bottom, that, uh, a neural network is kind of a fancy term for a composition of a bunch of tensor-to-tensor functions. If you have a function that takes in an A and returns an A, you can just stack those one after another. Um, and that's what gives neural networks the kind of Lego-y flavor. You can grab, uh, bricks from one set and bricks from another set and, uh, attach them to each other. Um, so this neural network, this is an ancient neural network, uh, the Inception v3 model from Google that was state of the art in computer vision for a few months in the 2010s. Um, and each of those little blocks there takes in a tensor and returns a tensor. Uh, and so it starts with a tensor that looks like an image. So it's got a red, green, and blue channel and comes out something, uh, that 8x8x2048 example there. That's probably somewhere in the middle of the network. It's not the final classification output. Anyway, big block of numbers. They get passed into each other. Uh, and then each one of those is itself parameterized by a tensor. So it's not just, like, uh, a map that you could kind of, like, write down by hand that's like add one to every entry or something like that. Um, it's, uh, it's defined by, like, another big pile of, uh, of numbers, the weights and biases of the neural network. So this is, uh, the weights from the first layer of a computer vision network, the AlexNet that kicked off this, uh, whole deep learning revolution. Um, and it's these—these are—these are little—this is a visualization of a little block of red, green, blue, uh, uh, like, tensor with three color channels in it. So humans can actually look at it and interpret it, unlike the rest of them, and see what's going on, that it's got little things for detecting edges and textures and color differences and things. And so, uh, we want to run a big tensor-to-tensor map, and we are going to parameterize that with a big pile of tensors. And before it's time to actually serve users, those tensors need to be generated by the training process. And back, uh, a couple of years ago, um, or even as recently as 18 months ago, this would be the part where we would stop, talk about gradient descent, optimization, statistical learning theory, and, uh, you know, GPU acceleration and all the things that are needed to turn something into to, like, get those numbers, to get things like that involving, like, grabbing, you know, hoovering up a bunch of information from the internet, uh, without consent, um, and then crystallizing it into those piles of numbers. But nowadays, you don't have to do that anymore. Specialized foundation modeling teams, uh, generate these weights, and then they either put them behind proprietary service or they share them with everybody else to use. Um, and so we can skip past all that stuff and jump into the actual application. Um, uh, yeah. So any questions before we start talking about, you know, where those—you know, where those weights come from, uh, or, or rather, what the various ways to get a hold of such a set of weights or to be able to use such a set of weights are? Um, any questions at the level of what we're trying to do with inference? Probably pretty clear. That's, uh, maybe a reminder from the 1:01 stuff. Allright. So now let's start diving a little bit deeper. ### Model Options **Charles Frye** [10:16] So you want to run an AI. Uh, your fir one of the first choices that you need to make, as is, you know, pretty common with, um, with software, is a build-versus-buy question. Um, are you going to, like, are you going to make this, uh, you know, out of existing open components, or are you going to, uh, are you going to kick it off to a service? So, um, there's in the proprietary corner, there are a couple of players. In the open corner, there are a couple of players. Let's walk through what those are and, um, what the sort of dividing lines are and why to choose one or the other. So there's, uh, a number of proprietary modeling services, uh, like Anthropics, uh, from whom we just heard. Uh, and the good thing about these proprietary models is that they are the most capable models out there. Uh, so this is from the LMSIS, uh, leaderboard, maybe the, uh, the Hugging Face-hosted version of that leaderboard. If you look at the, the top five models on that are all proprietary models, and they're all from OpenAI or Anthropic. Um, so there are a couple of other players out there, and in the future, the, you know, they could release really high-quality models. Um, but for now, this is the, the, the state of things. So if you need the absolute highest level of intelligence in your application, uh, then you want to roll with one of these. Um, it's also often common to start with some of the most highly capable models and then kind of, uh, prove out your application there and then move to doing it, uh, like, with a less capable model that's cheaper, easier to run, uh, the sort of, like, rewriting it in Rust or something. Uh, so the usual concern with using a proprietary service, there's a number of them, including things like vendor lock-in. Um, but one of the one of the ones that comes up immediately is, like, how much is it going to cost me to use a proprietary service, and can't I save money by doing it myself? So the fact that the capabilities are higher with the proprietary models is one reason to say, well, you're not going to get exactly the same thingright now, um, using an open model. Um, but then the other kind of kicker here is that the proprietary models are priced very affordably. Um, so this is something that's that has been the case. Uh, thisright quote here is from a blog post I wrote back in, like, January, um, when the only open large model was GLM-130B from Tsinghua. Um, and at that time, just, like, trying to get the thing running, uh, in a day, I got to within an order of magnitude of the cost of OpenAI, but, you know, on the up on the upper end. Um, and then more for a more recent, uh, and more serious example, the folks at Honeycomb, uh, made a natural language to SQL, uh, kind of transformation, maybe not SQL, but query language, uh, transformat transformation, uh, AI product. And, like, their opinion was that OpenAI was very inexpensive to run for their task. Uh, and so you'll find it seems that, like, nobody's attempting to, uh, like, extract rents from monopoly pricing the way that you can get with some proprietary services, uh, where they know that you have no choice but to pay them $10 million for a MATLAB license every year or whatever. Um. **Guest** [13:44] Not to pick on anything. **Charles Frye** [13:45] Not to pick on a sorry. For a license for an array programming language. Um. Uh, yeah. So and the costs here, you know, a, a dollar for a million tokens for Claude instance, $10 for a million tokens for Claude 2, uh, relative to how much we're used to paying, like, humans for text. You know, that's, that's, like, a pretty decent deal. Anthropic has been slightly more expensive than OpenAI without, like, a clear win on, um, capabilities. Uh, but that's the current state of things. They do have longer context windows, which is kind of nice, but, um, uh, but yeah, most people for the pricing reason and the capability reason, uh, choose OpenAI at this time. Um, so the bad thing about proprietary models, besides, like, you know, uh, hearing Richard Stallman screaming in the back of your mind all the time, uh, is that proprietary models cannot offer you full control by dint of their, like, very nature as proprietary models. Um, so Alex Gravely created, uh, Copilot, part of the team that created Copilot at GitHub, um, was celebrating that GPT-3.5 Turbo Instruct, a recent release from OpenAI, had brought back log probabilities. So you can see what not just, like, what text did the model generate, but what probabilities the model gave each token along the way. And back when the back when it was just the GPT-3 API and the playground, you could see that information. And that's where a lot of the, like, early work on sort of, like, intuition about prompt engineering came from being able to see those numbers. And there's all kinds of cool techniques that you can get up to if you have those log probs, um, and if you can manipulate those log probs. Uh, so yeah, you can read out confidence information. You can use it, like, during your development process to sort of, like, more gather more rich information about the system. Um, and there's really, you know, you are you're interacting with a probabilistic model. If you can't see the model's probabilities, you're, like, fundamentally hamstrung. Um, and so that was mid-September. And then, like, a couple of days ago, they turned that off. Um, and the reason why is because if you give somebody that amount of information, they can start to reverse engineer your model pretty quickly. Um, and then you are stuck in the situation of IBM creating a personal computer, and then a bunch of people with soldering irons irons and oscilloscopes turn around and make clones of your machine, like, within a year. Uh, so proprietary models are, like, fundamentally disincentivized from giving you that level of control, despite the fact that it's very critical for, um, for, like, actually effectively operating the system. So there needs to be that capabilities edge, um, that, like, raw capabilities edge in order to, uh, make up for this fact. Um, then lastly, maybe some people work in an enterprise, uh, in this room. Don't you don't have to out yourself, but maybe some. Um, and if you're operating in this sort of situation, you can't just ship a ping to an external API out there, uh, into your business. You people want to know about governance. People want to know about GDPR compliance. Um, and the one of the nice things about OpenAI's offering, probably true about Anthropic by this point and definitely true about Google AI soon, is there's a nice white glove enterprise tier, uh, uh, around this that gives you, like,right underneath, like, you know, launch an artificial intelligence application and achieve your childhood sci-fi dreams is built in security and compliance. Um, we will spend $20 billion on cybersecurity so that you don't have to. Um, so this, like, if you are in a situation where you need to, like, you know, assuage concerns about data privacy, um, this sort of enterprise tier, um, you know, SOC 2 compliance, et cetera, is, uh, can be really critical for making your life easier. Uh, any questions about proprietary models and such? **Guest** [17:48] How much more expensive is it to run consuming the API versus the cloud rights? **Charles Frye** [17:54] Hmm. With, uh, with Azure? **Guest** [17:56] Yeah. **Charles Frye** [17:56] I want to say that Azure was cheaper at the start, um, but maybe it, uh, I haven't ha-had any reason to use it. Um, so I'm, I'm not sure. **Guest** [18:06] And, uh, just a follow-up question. Like, are there, like, any, um, trade-offs in general with regards to performance? I don't know. **Charles Frye** [18:13] Hmm. **Guest** [18:13] Tokens per minute consuming the API or? **Charles Frye** [18:16] Ah. So for you mean for the enterprise tier versus yeah. So the enterprise tier also offers, like, an actual SLA, which the OpenAI API, like, doesn't, um, and is, like, geared that's, that's, you know, that's maybe another very critical feature besides security and compliance. Like, they will promise that you will get a response and not, like, a 500. Um, and they have much more generous, uh, rate limits and things like that. And I think they also offer, like, a little bit more control over stuff so you might be able to do some fine-tuning that you can't do via the generic OpenAI API. **Guest** [18:52] Oh, the fine-tuning for, for Azure, uh, limited to, like, three models. **Charles Frye** [18:57] Okay. Yeah. But there's also limits on the public API for fine-tuning,right? Yeah. Um, I personally did not find it particularly useful to, like, fine-tuning API, both, like, hard to use and not clear benefits. Um, I think the example from Gradient did show that, like, if you want to achieve a style and you don't want to spend money on, like, context to set up that style, uh, then maybe you can win with fine-tunes. But it's, like, it's not really a successful way to inject new information, so you aren't saving on the tokens that you would retrieve. Um, and you have to pay more to inference a fine-tuned model. And that just goes down to the fundamental. Like, you're asking them to do more work for you, um, and they can amortize that cost over fewer users. And so it's just always going to be more expensive. Um, and so, uh, yeah, so that limits the utility of those fine-tuning APIs. Yeah. Please definitely, like, ask questions, um, uh, customize to what people are interested in. And also, if I don't know about something, like, please do, uh, interrupt. Great. **Guest** [20:03] Google has Vertex. **Charles Frye** [20:05] Yeah. Yeah. So. **Guest** [20:06] What about that model? **Charles Frye** [20:07] Uh, well, so Vertex is a little bit different from this, I think. Um, I think of Vertex, which I was going to talk about later, as sort of, like, a something that I can launch my own services into, as opposed to, like, uh, oh, here's a private version of the PaLM Bison API. But maybe, maybe is that part of Vertex? **Guest** [20:27] Which version? **Charles Frye** [20:28] Okay. **Guest** [20:28] Yeah. Talk to them. **Charles Frye** [20:29] Great. Yeah. Um, so yeah, that's, um so they're already available for Google AI. Does anybody know if Anthropic's AWS, like, uh, pri like, enterprise offering is up yet? **Guest** [20:42] Bedrock? **Charles Frye** [20:43] Yes. Yes. **Guest** [20:44] It is, yes. **Charles Frye** [20:45] Great. Yeah. I, like, refuse to make slides about this stuff more than, like, 48 hours in advance, and I still find myself getting cut. Like, yeah. **Guest** [20:57] Yeah. So in proprietary models and open-source models, which ones do you are you betting on? Which ones goes to? **Charles Frye** [21:06] So the question was, uh, between proprietary and open models, which ones are you betting on? Um, gambling is illegal in the state of California, so, yeah. Um, uh, we'll get we'll get there. So, um, let's talk about the open models, and then we can answer or, uh, open up that discussion. Um, so open models are less capable, but catching up. Um, and their hackability is very powerful. Um, so going back to that leaderboard that I showed, if you look at the next five out of 10, um, four of them are LLaMA 2 models. Uh, actually, so this column here, to be clear, is the license. Uh, so the top five are proprietary. Like, there's no, uh there is no license for those weights. Um, for the bottom five, they have, uh, special licenses. Um, so these are fine-tunes of Meta's LLaMA model series. And this model series has, like, kind of captured mindshare in the free and open-source software world. So a lot of the people who are, like, hacking independently, um, and, uh, you know, making public Git commits, um, and, you know, funded by the Linux Foundation and things like that, these people are working in the main on, uh, adjustments to or improvements to the LLaMA model series. Um, and this is really critical because the secret to, like, the success of open-source software in general is the ability to do this kind of, like, highly parallelized development where lots and lots of people are adding tiny little features and, like, you know, going out into the last mile and, and adding those tiny little things that they need, um, and sort of, like, making use of all that, uh, work by others. So insofar as you're able to do that, you are able to provide useful open-source software that can compete with software that's made by, you know, highly remunerated teams, um, you know, in, in Northern California. Um, so the important question, uh, is this actually, uh, open-source? So you'll notice these licenses here do not have friendly, beloved names like LGPL or MIT or Apache. They they have, uh, a special unique name. Um, and that's because Meta's license for the LLaMA 2 weights, the LLaMA 1 weights for at least under a research-only license, and were only sent to certain people, and then were immediately, like, torrented, and the license was violated. So they gave up on that, uh, on, like, fully controlling it. But they did say you cannot use the, uh, data or output to improve other large language models. You can only use it to improve LLaMA models, um, and also release under the same license, which is pretty typical with open-source, um, which, uh, is partly an attempt to sort of, like, capture this, like, as people are doing parallel development, they should only be contributing to the development of this, um, this branch. Um, and then also, if your product's monthly active users in June of 2023 was 700 million users or above, um, you're not allowed to use it. Or sorry, you have to pay for a special license. Um, and that, uh so apologies to anybody, you know, who's, um, you know, if you're running an app with more than 700 million users, um, uh, you'll have to go elsewhere, I guess. But the, um, the key thing is that these are violations of the, like, sort of agreed terms of what makes something an open-source license according to the Open Source Initiative who, you know, has some, uh, claim to controlling how that term is used. I didn't I don't think they ever ended up getting a trademark. Um, but they, uh, you know, they are got the community aligned around a small number of licenses that and around a key set of principles that include, like, you can't tell you can't say who's allowed to use this software, for example, which is included in the LLaMA license. So there's, like, uh, uh there they have opened up a multi-stakeholder process to define open-source AI. This is occurring at a time in which op like, the meaning of open-source is also being contested in sort of, like, software as a service. So things are a little things are a little tense there. Um, but hopefully, we'll come to, uh, an agreement as a community on what that means. Um, so this is a fast-moving space still. So just because you get mindshare early on, if, like, things change rapidly, that doesn't mean, like, LLaMA's locked in forever. Um, so Mistral, for example, dropped a model, like, two weeks ago, um, that at only 7 billion parameters was outperforming, um, like, larger models in the 13 to 30 billion parameter range. And those models were outperforming the previous models, um, at their size. So there's, like, except insofar as those things, like, continue to get updated, um, you know, they, uh yeah, they can be outcompeted. Um, the other thing to watch out for is that there are a lot of people who are very excited about, like, taking on the death star of OpenAI or whatever and, um, get very excited about these open models. There's also some political things about the politics of how ChatGPT likes to respond to questions versus the politics of how people who meet other people in Discords like to respond to questions. Um, and that can that sort of, like, enthusiasm can lead to, like, pretty big errors. So, for example, there's a lot of excitement about these models that were take take an open LLaMA model, uh, the, the weights from that, and then grab, like, 10,000 requests from the OpenAI API or scrape, like, R/ChatGPT or whatever, and then just fine-tune. Like, now there's a data set. Use it to fine-tune the model. And those are the ones that were, like, up there on the arena, uh, on, on that, uh, like, ELO ranking, um, from the leaderboard that I showed before. Um, and there was a claim that these, uh, had, like, 90% of ChatGPT's quality. They're only 7 billion parameters. Like, you know, people were, like, very enthusiastic about this back in April when I was talking about this. A lot of people were like, "Why are you even talking about OpenAI anymore? We like, you know, Vicuna has done it." And, like, yeah. Uh, so, um, this is a fake screenshot, uh, that's, uh, from a paper or that I modified from, from a paper about, uh, about this, this topic of how well these models work. Um, so this is one output from a language model anonymized, um, on answer, uh, how does actor-critic improve over reinforce? Um, so this is an algorithm from reinforcement learning. So one language model says, uh, actor-critic algorithms are an extension of reinforce that combine both policy-based and value-based methods. Um, it's got a critic network. It's got advantage estimation. It's got function approximation. So that's one answer. That's A. Answer B, uh, actor-critic algorithms are a type of reinforcement lear-learning algorithm that improves the reinforcement algorithm by combining actor, policy, and critic value components. Actor-critic algorithms use a single critic. Reinforce has a separate critic for each action. Actor-critic algorithms learn the policy and actor simultaneously, but reinforce learns them separately. Um, so, you know, those might seem fairly similar. So does anybody have a strong preference for, uh, answer A here? Anybody have a weak preference for answer A over answer B here? Some people raising, like, a like, a, a soft hand maybe. Yeah. Um, anybody have a strong preference for answer B? Maybe one, maybe two. And a weak preference for answer B? This one's got, like, a they both got these nice numbered lists, you know, which looks very authoritative. Reminds me of a Medium article, which is likely to be true, of course. Um, uh, so the, uh so, uh, answer B comes from, uh I want to say this was GPT-4. Yeah. Um, answer B comes from GPT-4. And oh, wait. Wait. Sorry. Answer A comes from GPT-4 and has the advantage of being correct. Um, answer B comes from one of the, uh, fine-tuned models, um, and is, like, gibberish, basically. Um, and so this just, you know, like, just having humans rate the outputs of, of language models and the way that a lot of those, uh, leaderboards were constructed, um, did not, like it didn't have any grounding in the actual utility of the answers. It was just a lot of people going, like, "Looks good to me." Like, "Nice. Yes. Merge." Um, and, um, without, like, knowing whether it was actuallyright or not. Um, and so, uh, there's a nice paper from some folks at, uh, Berkeley about, um, sort of walking through, like, what's going on. Basically, the models are picking up style from a fine-tune, which is things like that delightful little split into bullet points and, like, you know, like, a very authoritative and friendly educational style, um, but not, like, actual knowledge, not, like, reasoning capabilities. And, like, a lot of people in the open modeling communities, like, sort of missed this or, like, willfully ignored it. Um, some of the sharpest people were definitely up on this. Like, the Guanaco paper, for example, mentions that there's, uh, some issues with evaluation came out before this paper, um, but definitely a lot of people missed it. ### Future of Models **Charles Frye** [30:40] Um, so the, the immediate question that comes up is, like, between these open models and these, uh, proprietary models, who's going to win long-term? Like, who should I bet on? Um, and in some ways, I think that's a bit of a misguided question. Um, so consider operating systems. Uh, like, the first operating system, roughly, was System 360 from IBM on mainframes, extremely closed. Um, in the '80s, there was a rash of operating systems, most of them closed. The original Xerox pilot on the Xerox star, uh, was extremely closed. DOS and Wind DOS was closed. Uh, macOS at the time was, like, completely closed. Um, and there were Unix operating systems that were kind of, like, mixed. Um, then over time, the, like, closed versions of Unix lost out to more, like, friendly licensed ones and in particular to GNU/Linux. Um, and there's been a bit of a trend towards open, uh, open operating systems kind of taking more mind and market share over time. Like, data centers have more Linux in them now than they did in 2005 and then than they did in 1995. But, uh, from what I can tell, it's still, like, 70% plus Windows, um, for, uh, for operating web servers. In mobile phones, we also have an open operating system and a closed operating system. And these things have been able to coexist and serve different needs for different organizations throughout, like, the history of operating systems. Um, and the same is true of databases. Uh, so back in the '70s and '80s, it was Oracle and IBM's dbt2, um, which is still around, I found out. Um, like, you lived too long in, in the in San Francisco and you forget that there are people who use IBM dbt2. Um, in the '90s, there was some, like, consolidation around more open implementations of, of SQL. Uh, and in the 2000s to 2010s, there was the NoSQL movement, but that was still, like, mostly open-source databases. Uh, so there's been a lot of, like, movement in the direction of open databases. Um, with streaming databases, we have both proprietary and open options. If you look at the top 10, uh, databases as ranked by dbengins.com, which is you could quibble with the ranking thing, but the, the key point is that there are, like it's like half-and-half split between, um, between proprietary databases and, um, and open-source databases. Uh, and that has been, like, relatively stable over time with, like, a soft maybe a soft trend in the direction of open, uh, databases. Uh, so with these, like, very, like language models, foundation models are this very, like, low-level, uh, component of a software stack, more like an o like an operating system or a database, I think, than, like, um, you know, than a SaaS app or a or a UI. Uh, and because of that, they're likely to be subject to some of these same forces that say there's some people who want to work one way. There's some people who want it to work in another. And for some of them, that openness, that hackability is going to be critical for others. The, like, reliability, the existence of a white-glove enterprise version is going to be really critical. Um, and those will allow these two things to coexist. Um, and, uh, yeah. And the CEO of Hugging Face liked my tweet when I said that. So it's probably true. Um, but I think a lot of people are like, "Well, no. But, like, who's like, who's going to win? Like, who should I bet on?" Um, and I think the closest thing to an answer that I have is that if capabilities requirements saturate, if people no longer want the absolute smartest model out there, they just want a model smart enough for XYZ, then open models will probably catch up and then, like, start to dominate. Um, the thing that keeps open models behind proprietary models is the extreme expense of maintaining a large resource team and, like, you know, continually constructing new data centers at an increased scale, um, to the tune of, like, $500 million in order to hit that next capability level before everybody else. But, you know, uh, uh, at a certain point, processors got fast enough that people were not, like, clamoring for the next upgrade as soon as possible. Um, and at that point, we're starting to see, like, a little bit more opening up in the sort of, like, in the chip space with, like, RISC-V. Um, and so, like, at like, with in any number of other technological domains, you've seen that when requirements start to saturate, um, then, like, open, uh, like, open versions can catch up. Um, if they are unbounded and it's like, you know, uh what's a good example? Like, RAM. Like, nobody has enough RAM. Everybody wants more RAM. I don't think there are any, like, open, like, attempts to make, like, an open RAM architecture or something. Um, and that's because and one reason why I think is capabilities requirements there are re remain unbounded. Um, and so, um, if that's the case for, uh, cognition and AI models, then proprietary models should be able to maintain that edge in, in capabilities, which would sort of tilt the balance in favor more people would say, "Oh, no. I need this, this proprietary thing." Um, so that's the closest to an answer that I have. Um, yeah. Any questions on that front before we dive into, uh, um, where we actually run these models? **Guest** [36:06] I was curious, uh, what's the language support for these language for these models? Like. **Charles Frye** [36:12] Yeah. **Guest** [36:12] Can you use a use, like, a language other than English for these models? **Charles Frye** [36:17] Yeah. Um, so the question was, what kind of language support do these models have? Um, and because it's only an API call away, you can, of course, use Python or Node or whatever you want. Uh, no. So the question was about, like, these are language modeling, like, machines. What languages do they model? Um, and the basic answer is that the more text in that language that is available on the open internet, the better the language models will be on, on that language. So they are, like I want to say GPT-4 is smarter in maybe smarter in Malayalam than it is in Mandarin. Um, I forget. There's, like, some interesting inversions of, like, number of people who speak the language versus how, uh, how intelligent the language models are. Um, so I think a lot of them release benchmarks that say, like, how multilingual is this language model and for which languages. Um, there is you run into the fundamental token constraint of, like, you need, uh you need existing lang like, you need examples of that language that you can get a hold of in order to train the model in them. Um, and there just are more English tokens. Um, but for a given capacity, you can probably achieve, like, higher quality in a specific model by looking for, um by looking for a model trained in that language. So there's definitely some, like, good old nationalist European endeavors to make, like, a French language model that insults you if you ask it for stuff in English, um, which it, of course, picks up just from reading French. Um, um, but yeah. But the but the core models, like, they support English really well. The instruction fine-tuning and the ROHF is actually mostly applied to them in English since the annotators who, uh, enforce that policy, uh, through their examples are mostly writing in English. Um, so fun fact, you can get ChatGPT to tell and Claude probably to tell you how to build a bomb if you ask in theright low-resource language. Um, uh, just fun facts about language models. Um, yeah. So that, that does that, that is a problem, and it does sort of, like, ingen uh, it has a multiplying effect on English the English language's kind of, like, cultural hegemony, um, which is a bit unfortunate. Yeah. **Guest** [38:41] So for the for languages that are less represented. **Charles Frye** [38:44] Mm-hmm. **Guest** [38:45] Is, um, is reasoning capability lower or is understanding lower? And also, are there arbitrage opportunities in translating first? Uh, in these prompts, for example, translating first, um, and then doing the reasoning and testing back? **Charles Frye** [39:01] Yeah. I'm unaware of any aca like, you know, any benchmarking work on this. My gut tells me that translating to English first, doing chain of thought, and then translating back to the original language would work better. Um, you'd you're kind of, like, wondering whether the lost in translation effect is bigger than the, like, boost of chain of thought in English. A lot of the, like, circuits in language models are very token-specific. Um, and then yeah. So there's, like, the like, just one example. If you ask it who Tom Cruise's mother is, then it answers better than if you ask it that woman's name's son. Uh, I don't know her name at all, um, so I can't really do this example effectively. Um, ChatGPT wins again. Um, but the, uh there so that's, like, an example of a very of a token-specific circuit. It's, like, related to Tom Cruise. Uh, and so you can see, like, it's not reasoning the way that a person would or that or that you would guess from, like, you know, how, how you would think about a knowledge graph or something like that. Um, and so that's where you get these unintuitive things. Um, but yeah. **Guest** [40:10] So I saw you, um, used to or maybe still do a deep learning bootcamp. **Charles Frye** [40:15] Oh, yeah. **Guest** [40:16] That you offer. And I was wondering, like, one of the pieces of the whole event is, okay, there's these reasoning-as-a-service APIs now,right, where you can do a lot more things without having your own ML tool. Um, in your opinion, for the typical AI engineer who wants to build things, does it still make sense to learn some amount of deep learning, some amount of ML, you know? And is it, like, just watch some videos, read some blog posts? Is it, like, actually do, like, bootcamp? Is it, like, actually read some papers? Like, what would you recommend? **Charles Frye** [40:51] Yeah. That's a great question. Um, for individuals, I think it's a matter of your personal interest in understanding the modeling. Like, I guess the analogy I would immediately jump to is as an individual developer, you can get away with not knowing anything about databases. Like, I have done that. I couldn't write a B-treeright now. I don't want to ever learn how to do that, like, and to think about page sizes. And yeah. It makes me ill to think about that. And whereas I get excited if I wake up in the morning and I can think about Bayesian inference in language models. And so as an individual, I think you can, like, kind of be guided by your like, what you find most exciting. As a team and as an organization, though, if you have nobody who understands databases in your organization, you're probably going to be in trouble. Um, just, like, it ends up like, most applications require, like, pretty decent knowledge of databases and when they go down or when they need to be configured. Even if you are using Redshift or, you know, you're using some managed service, being able to, like, understand some stuff about them is actually critical for debugging and being able to know when you need to switch managed services or, like yeah, or how to reconfigure them. So I think, like, the direction that we're going to go is to evolve there. And it's a question of whether you want to be, uh, uh, a site reliability engineer focused on LLM reliability or, um, you know, uh, a, a modeling engineer or whether you want to be, like, more at the, like, application layer. Um, yeah. **Guest** [42:30] So just an interesting one. What's your personal opinion on this? There's a handful of companies that have gotten recent funding to build, like, vertical-oriented commercial models, finance, healthcare, etc. **Charles Frye** [42:42] Yeah. **Guest** [42:42] What's your opinion on those? **Charles Frye** [42:43] Yeah. So the question was, what about these models that are foundational but, like, less broad? So it's like a foundational model for law, a foundational model for, um. **Guest** [42:55] Healthcare. **Charles Frye** [42:56] For healthcare. Yeah. I, um my experience has been that if you bet that some capability is not going to be available in a language model, um, or in a foundation model, like, you will get you will lose that bet. Um, uh, so just as an example, in the deep learning bootcamp, uh, we spent a long time trying to make an optical character recognition system. Uh, and it's like you know, it's the, like, pinnacle of a class when you can finally, like, deploy a web service that does optical character recognition. And that's, like, an accidental side feature of GPT-4V, and it's, like, better at it than the thing that we built. Um, and a lot of ML teams have experienced something like that. Um, so I worry I would worry if, if somebody were, like, offering me that as a job opportunity, for example. I would worry that it's going to get, like, scooped on either side by a hyper-specific model that's, like, 10 times more efficient and isn't, like, a generic healthcare model but is, like, a, um, uh, ultrasound for the heart model. This is one I worked on before. Um, yeah. Uh, or just send it to the ChatGPT API or to the GPT API. Great questions. Um, so, uh, maybe another reason to think that there might be, like, a little more alpha in, in actually learning more about the models is, um, inference doesn't have to be executed over a network. It doesn't have to be executed, like, in some central server. ### On-Device **Charles Frye** [44:31] There are lots of reasons why you might want to execute your, uh, uh, your inference on an end-user device. Um, so we'll talk about the different types of end-user devices and the different constraints that they put on inference and the implications, um, like, engineering and strategic, and then also, uh, talk about what the options are for doing things over a network. Um, so running stuff for end-users is, like, uh, like, where the sorry, where the end-user actually executes it themselves is not quite there yet, but it's, like, uh, it's getting there and maybe a little bit faster than I personally expected. Um, like, I've run LLaMA 2, uh, 13B on this very laptop, um, without it catching on fire. Um, so there's, uh, there's some hope, uh, that there will that, that will continue to get better. Um, and so this, uh, this is critical for latency-sensitive applications. So, like, being able to actually execute the inference at the same place that the user where at the same place where the user is. Um, and the reason why it goes back to this, like, this famous set of numbers every engineer should know from, uh, Peter Norveg and Jeff Dean at Google, um, which is that the time it takes to send a packet, just one packet so probably this probably isn't even a whole HTTP request. I'd have to check again. But let's just say you send information back and forth from, like, here in California to Europe and back. It's 150 milliseconds. Um, and there's, like, a number of kind of made-up numbers in the UX world about, like, how fast you need to be for something to feel interactive. Um, so one of them going back to, like, the '70s or '80s is the Dougherty threshold, which says the user and the computer can interact with each other in under 400 milliseconds. Then the, like, human won't feel like they're waiting on the computer. And as you're programming things, you won't, like, end up blocked on human input, um, but you'll and you'll still have plenty of time for doing stuff in, in side threads and things like that. Uh, so if you are like, if you have to, like, do a network call every single time, you're using up, like, a third of your budget just on, like, waiting for information to come back. And now you're going to spend a ton of engineering effort on, like, trying to find ways things that you can do asynchronously during that, like, pa like, that network call. And, like, you can you can work around it, but it is punishing. Um, and that's, like there are even tighter, like, reaction time things. Like, if you have a self-driving car, um, you can't wait 150 milliseconds to, uh, to find out that you need to brake. Um, so, uh, yeah. So and the nice thing about this, uh, the other benefit to it, beside it being necessary in some places, is that if end-users run the computation, then you don't need to pay for it. Um, so your inferencing costs can be $0, which would be which would be great. Um, so the cost that you pay, um, is control. So you have less control over the execution environment. Um, your ability to do telemetry and see what is going on is limited. Uh, people don't like it when you, like, carefully observe their activity using software on their machine. But if they you put the same software at a URL, you can spy on them as much as you want, and they don't get mad. Um, so you lose out on telemetry. You, uh, have to worry about compatibility with different execution environments, and you have to actually support past versions, unlike, uh, if you're running it as a service. Um, or rather, if you, like, you know, con-control the execution environment. Um, so the things that are unlocked by this are some of the best applications here. Uh, like, some of the most exciting ones, especially to me. So, uh, use on smart used in smartphones, used in robots, used in wearables. Um, so Google just in the past couple days announced that the Pixel Pro 8, um, is going to have, uh, large language models directly on device. Um, they mostly showed off stuff that looked like kind of, like, summarization and some, like, light image editing. So not, like, full-on, like, you know, like, "Hey, Siri, um, why did the Ottoman Empire fall?" Like, I don't know what you talk about with ChatGPT. Um, but, uh, like, it's not quite that level, but it's a move in that direction and a trend we can expect to kind of continue getting that inference onto the device. Um, and, uh, there was also a recent hardware hack, um, on, like, using you know, getting this inference on mobile robotics platforms. Um, and so there's there was a ton of cool applications there, like, um, yeah, some stuff with, like, 3D, like, point cloud rendering for, for your inside your house, like a Roomba you can control with your voice. Very cool stuff. Um, the, the constraints that appear here, uh, that you'll have to engineer around are, like, very tight hardware constraints. So, um, there-there's memory limits, both disk and, like, VRAM and RAM are, like, are extremely tight. And, like, current language models, you can always trade more up to points where you're spending, like, $100,000 on a machine. You can trade more money for smarter models. Um, and phones are down at, like, gigabytes, uh, low gigabytes of RAM. Uh, like, yeah. Um, was running a language model on a single board computer, and that had, like, 2 gigabytes of shared RAM between the CPU and GPU. Not a lot of space. Um, and the, like, real, uh, deep limit is power. Um, it's, uh, uh or the sorry, there's a limit on power, which is, like, an A100, uh, which you might use for inference, draws 300 watts of power. And something like the single board computer that was using the Jetson Nano that's 10 watts of power. So a factor of 30. Not going to make that up anytime soon. Um, and underneath both of these is the problem of heat dissipation. Um, there's that's, like, a really, like, tough thing to deal with when you are in these, like, small environments, um, and, like, prevents them from just being like, "Oh, I'll just, like, make a chip where you can actually move, like, 9 petabytes a second, um, like, across an inch." And it's like, uh like, you just do some, like, back-the-envelope math, and it's like that's going to, like, egress so much heat, the thing's going to catch on fire. Um, so, um, yeah. This is we're talking about some hardcore engineering stuff here. Um, allright. So the, like, mobile environments, uh, maybe, like, further out in the future to get, uh, large capabilities onto them, but, um, not impossible. What about, um what about other consumer hardware? Desktops, um, which are a place where you could have video games with actual artificial intelligence in them, uh, operating system level assistance, na-native apps with these, like, kinds of features that we're starting to see in, um, in browser apps. Uh, so you still run like, you run into even more heterogeneous hardware, and that's going to give you different constraints depending on the system that you're on. And that is going to require, like, really heterogeneous software to meet those constraints. Like, you probably can't assume that everybody has an NVIDIA 30 series or later GPU, even though it would make your life a lot easier. And you probably can't assume that you can use up all of the RAM on that, uh, uh, uh you know, on that chip, even if it would make your life easier. Um, I think the long term, we might be able to expect ecosystems to adjust around the requirements of these workloads a bit. So, like, kind of, uh, like, make it to, like, make cleaner interfaces for using these things so you don't have to write 15 different versions, um, or write a make file that looks like Llama.cpps. Don't look at it. Um, uh, very scary. Uh, there's kind of a sweet spot, actually, in what little, like, next-generation video game consoles because you have you have total authority to just use up as much of the system as you want. People pay lots of money for them. They often build custom silicon based on what developers want. So that could be if you're thinking about what you want to be doing in, like, five years, seven years in this field, consider that as a possibility. Lots of people would love to have, um, a real, like, human-like intelligence in the, um, uh in the things they're shooting in their first-person shooter, you know? Like, that, that I think that would make a lot of money. Yeah. **Guest** [53:09] Uh, when it comes to building, especially for mobile hardware with those constraints you're talking about. **Charles Frye** [53:13] Yeah. **Guest** [53:13] Can you say a little bit about quantization and how that, uh, helps build for that? **Charles Frye** [53:18] Yeah. So the question was, for mobile hardware, like, what are solutions and specifically quantization? So, um, when one of the key constraints is memory, like, just trying to make the size of the model smaller and the size of the computation smaller is helpful. So the, like people are pushing to try and take the parameters of language models down from being 2 bytes to 1 byte to half a byte to, like, a single bit. Um, and I think people are kind of stalling out at the, like, half-byte level. Um, and often to actually recognize those gains, you need to, like, write assembler and stuff. It's like it can get pretty gnarly. Um, so that's often only, like, highly resourced teams working for a long time that they can actually see those benefits. Um, the other thing that people talk about a lot, uh, for, like, making models work on smaller devices is sparsity. Um, and so sparsity means, like, oh, there's this giant weight matrix. Maybe most of them are close to zero, and maybe we can just, like, get rid of those. Like, if we were going to go to 1 bit, there's 0 or 1. Like, why not? There's zeros. And then 0 is, like, a very easy number to work with. Like, you the number that comes out is multiply at 0, add. You just keep the number. So it's you don't need, like, a full logic circuit to handle it. Um, so there are some things that make use of sparsity. The problem is that the type of sparsity that neural networks need is called unstructured sparsity. You have just had zeros kind of, like, scattered around your matrix multiply. And the all the, like, existing easy-to-use has, like, you know, Python API, uh, stuff is, um, uh is in structured sparsity. And so you get gains there. And it's like you might need, yeah, a lot like, yeah, hand-tuned CUDA kernels or, yeah, to, like, actually take use make use of unstructured sparsity. So that's something you know, if there's a ton of pressure, we could see those developments in five years or so. But, um, we haven't seen people have been thinking about that for almost for, like, seven years, and it's, like, not made a ton of progress. But, yeah, helps definitely has made it easier. And, like, Google has been able to fit a decent amount of language modeling capabilities on a, uh on a mobile device, um, using distillation and quantization and probably more secrets that I won't share. **Guest** [55:43] I had a dumb question. What's the size of these models in terms of memory? Like, if you have. **Charles Frye** [55:49] Yeah. **Guest** [55:49] Suppose you have to keep the whole model in memory. **Charles Frye** [55:52] Mm-hmm. Yeah. Um, so the if somebody tells you a number like 50B, you know, like Llama s Llama Anthropic 52B, Llama 70B, that's billions of parameters. And then the question is, like, how what, what how big is a parameter? Like, how many bytes? And the, like they're trained or where the, the, like the way they come out of the factory is 2 bytes per parameter. So take the number that somebody gives you, multiply it by 2, and then the B is giga. So, like, a small Llama model is like 14 gigabytes. 7B times 2, 14 gigabytes. So not going to fit that in phone RAM. Uh, and that does make, yeah, doing this, uh, a lot harder. Um, you can do thing you can, like, try and do stuff with paging, like, put stuff on the disk, bring it bring it into RAM, then, like, execute with it. Um, but that, like, slows things down a ton. Um, so in general, yeah. **Guest** [56:52] 4 bytes per parameter? **Charles Frye** [56:54] Hmm? **Guest** [56:54] 4 bytes per parameter? **Charles Frye** [56:56] So you people used to train in Float32 and release models in Float32. Maybe I thought the Llama models were released in Float16. No. **Guest** [57:04] So only Float16 usually? **Charles Frye** [57:06] Yeah. A lot of like, a lot of people train in this new, like, Google Brain float, uh, thing. And then they they're doing that because they want to be able to use Float16 and so have 2 bytes per parameter. But, like, definitely the so the, like, the default before that was 4 bytes per, per parameter. And before that, when people were doing scientific computing with graphics with graphics cards, like, um, people at the national labs, the default was, like, 4 bytes per parameter. Like, um or sorry, 8 bytes, 64 bits, um, because they really needed that, like, high precision and high range. Um, and yeah. But now the trend has been to push them lower, and many model releases are now, like, already 2 bytes, 16 bits. Yeah. Um, and now Georgy Gergonov is, like, immediately converting them down to 4 bits, uh, and 3 bits, um, which is, like, wild. Like, what does that even mean? Um, like, a non-power of 2. It's scary. Unsettling. Um, okay. So, uh, with there's another execution target that gets you a lot of the benefits of desktops, which is, like, you have a beefy machine to run on, and it's not yours, so you don't have to pay for it. Um, but then you get a more homogeneous execution environment, which is the browser. Um, so this is not, like, a web app where they, like, talk to a model running in a service, but, like, there is a model inside of the browser that runs inside the browser's, like, runtime. Uh, and that's like, the homog homogeneization environments would be very huge. Um,right now, this is kind of, like, awaiting some technical improvements in the world of browsers. So there is a target in WebAssembly that you could compile your programs down to, um, and in principle run them. Uh, the support for, uh, GPUs is very gross. Um, there's, uh, a working draft from the WWW consortium for WebGPU, which would make it cleaner and easier to use. Um, so that would help the, like, ecosys the, like, stack and ecosystem around this for other kinds of web applications is developing. We'll ma like, maybe lead developments in using this for, uh, delivering inference. Um, you have a new constraint distinct from the other ones, which is you now, at least as it standsright now, you would need to deliver weights over the network. And so now it's like you're, you're you have kind of the model size constraints that you might associate with mobile hardware, um, but only during the, like, first load. Um, so, um, there are probably clever ways to get around that, like progressively delivering them, um, or, uh, like, browser, uh, companies sort of agreeing to incorporate some foundation models into the actual browser runtime itself. Um, so, like, inside of, uh, like a, uh like, v9, uh, an update to v8 with a foundation model already built into the runtime. That would make your life a lot easier. Um, so this, like, would, uh yeah, browser assistance, maybe sort of, like, general, uh, like, executing apps inside of a browser that feel more like native apps. Um, that's the potential applications here. But, um, still a little, um, at the edge. That was a pun, I guess. At the edge. Um, okay. So because oh, yeah. Question. **Guest** [1:00:38] Uh, how many gigabytes is a small and a large modelright now? **Charles Frye** [1:00:43] A small and a large model. So when I hear small, large language model, first I cringe internally, and then I accept GPS system, ATM machine, whatever. Um, so a small, large language model, in my mind, is something that has, like, kind of limited ability to, like, speak and interact with. And that's what you see at, like, the, like, 13 billion to 30 billion parameter range. The, like, the medium size is, like, the 70 billion parameter range, which is, like, the largest open models. And then, like, a true large language model, the ones that, like, make people scared about losing their jobs, are generally, like, mixtures of 70 to 100 billion parameter models, or maybe they are themselves 200 billion, 280 billion parameter models. Yeah. So then for all of those, take that and multiply it by we'll call it 2, um, to get the number of gigabytes. So, like, half a terabyte for the, um for, like, a, you know, palm well, a whole terabyte for Palm 540B, um, which is one of the larger ones ever trained. **Guest** [1:01:51] I guess the context was, like, if they are preloaded in the browser. **Charles Frye** [1:01:54] Mm-hmm. **Guest** [1:01:55] It would have to be a smaller one,right? Which is still in the tens of gigabytes. If I understand correctly. **Charles Frye** [1:02:00] Um, yeah. I it's been a long while since I downloaded a browser to my computer, but I want to say that the package that you download to install a browser is in the, like, couple of gigabytes range,right? **Guest** [1:02:10] No, I think it's, like, hundreds of megs. **Charles Frye** [1:02:12] Hundreds of megs? **Guest** [1:02:13] Yeah. **Charles Frye** [1:02:13] Really? **Guest** [1:02:13] Yeah. Did I actually download the whole thing in the installer? **Charles Frye** [1:02:17] Uh, yeah. Oh. Uh, yeah. Wait. So you download an installer, and then you have to download anyway. Yeah. So if, if people want in to install stuff that's only a few hundred megabytes, then that's a non-starter. Um, I guess I expect a Linux distro image to be in the, like, couple of gigabytes, like, if I'm playing around with containers. So that's, um, uh that's, like, another anchor point. Um, and also, I for those things, we're probably talking, like, 2, 4, 8 years before that kind of, like, standardization effort agreement, like, happens. And we can hope that internet speeds will increase in that time to match the increasing needs, uh, uh, of the internet. Um, but yeah. The that's it's a pretty tight constraint and, like, probably looks a lot more like mobile stuff for a very long time. Um, yeah. **Guest** [1:03:09] Uh, programming languages like Mojo or Moddeler could change any of this math? **Charles Frye** [1:03:15] No. I think, like,right now, I've been kind of assuming that you're doing stuff relatively efficiently. And to be honest, like, PyTorch is, like, pretty good at this already. Like, um, the fact that the application layer is written in Python isn't the problem. Um, but yeah, good question. **Guest** [1:03:33] Another question. I was curious, what's the cost of inference? Like, uh. **Charles Frye** [1:03:37] Yeah. **Guest** [1:03:38] Inference on CPU versus inference on GPU. How is it, like, just the latency? **Charles Frye** [1:03:44] Yeah. Um, I think we'll come to that in, uh, once like, wanted to talk about, um after we talk about running AI over a network, talk about the actual inference workloads. Um, so we'll definitely get to talking about that. I don't think I'm not going to have a price number to give to you. Um, but you have to take whatever tokens per second you can get and then, like, however much you're spending on GPUs, um, and then convert that into a dollars per token. Um, and that's going to give you something you can compare to the, like, model providers. Um, and until you put some decent optimization into it, you aren't going to match them. Um, you did ask about CPU inference. That is rapidly evolving. I think there are cases where you can kind of, like, compete in price there. But, um, yeah. We'll cover that more later. I it seems like let's put a pin in those two things, and we'll come back to them after we talk about the, like really, the thing that almost everybody's going to do, like, immediately after they leave is going to be run AI somewhere in a data center. ### Inference as a Service **Charles Frye** [1:04:45] Um, but those are important questions long term. Okay. So, uh, like, uh, the running stuff on end-user devices has a lot of reasons why it's not so greatright now. So what, what, what do you get when you run AI in a in a AI workloads in a data center? Um, the biggest win is simplicity. Uh, the biggest pain point is latency, as we've already discussed. Um, so simplicity, like, you just you control the whole environment. It makes your life a lot easier. Yeah. **Guest** [1:05:13] You said that latency is the biggest pain point. Is that really a thing for LLMs compared to, like, vision models? Because, like, anyway, the tokens that you can infer per second are quite a bit slower than any of the network latencies that we talk about. **Charles Frye** [1:05:29] Yeah. So I would say that you can let's see. So the question is whether latency is actually a problem. So, um, if you need to do, like, back and forth, like, you need to get something back from the OpenAI API, then possibly, like, call it again with some added context or, like, run some if statements and then send it back, now you're looking at, like, multiple network calls. And you could avoid all of that overhead if you were running things locally. So that's an example of the case where people would run into a latency problem. **Guest** [1:06:00] But locally, you're going to get way lower tokens per second anyway. So you're completely dominated by the time it takes to generate a token. **Charles Frye** [1:06:10] So tokens per second is a throughput number, not a latency number,right? So it doesn't matter if your tokens per second is half. **Guest** [1:06:16] No, it doesn't. So it doesn't. **Charles Frye** [1:06:17] If your tokens per second is half that of what OpenAI is getting, but you only need to generate 30 tokens,right, that then it like, the, the latency number is going to be the larger one, even though your, like, throughput is lower. Like, this is definitely something that people have run like, run into when you have, like, highly interactive things. Um, like, there definitely so there to be clear, there are tons of applications in which you don't feel this pain. Um, and, like, ChatGPT, for example, is at this point, like, the latency of the response from the machine is not really the problem. Um, so yeah. It's not guaranteed to be a pain point, I would say, as well. Um, yeah. Um, so yeah. And I guess I'm also kind of maybe imagining situations that are closer to the computer vision case in which you need cognition well, in the computer vision case, you need rapid responses because it's, like, in the motor loop of a system, for example. And if we want to use language models as the cognitive component of a of a moving system, then they would need latencies like that, um, like, in the tens of milliseconds or something. Yeah. Um, and, uh, you are never going to be able to achieve that over a network. But yeah. Great question. Um, allright. So inference as a service providers, this, uh, makes it really super easy to get started. It's what, you know, when you're using OpenAI, they are inference as a service provider. Um, so all the proprietary models basically live here. There's not some, like, way that they would ship you the model, and you could run it, and it's proprietary license. Um, that doesn't exist yet. Um, open models are also available. So, like, if you want if you want to bet on the, the, like, open ecosystem, uh, you can use a service like Replicate, um, that will, like they'll run, uh, open models for you. It's, uh, generally, like, easy to get started. It's not that much more expensive than running it yourself in a lot of cases. Um, but you have limited control of the model. Um, for proprietary models, we already talked about how you would have less control kind of inherently. Um, for even for people who are providing open models, since they don't have IP they want to protect, in order for them to, like, serve it cheaply to you, they need to have, like and to have, like, an economic win that they can, like, pass on to you, um, and, like, keep a little bit for themselves, they need to do something like amortize costs across many users of models, many more than, than you have. And that requires some amount of homogeneity of usage. Um, and it's like,right now, it's proven to be, like, pretty hard to give people control while also giving them homogeneity of usage. Um, for something like AWS, they came up with really smart ways to cache pieces of containers so that the fact that everybody's using kind of the same software allows them to amortize while also giving customization. Um, but people have not figured out a similar trick for, um, for language models or image generation models yet. Um, so you don't have as much control as you would have yourself. Um, so new constraints arise. So there are things like API rate limits. Um, and now this is sort of like a cost management game. You look at this as, like, um, rather than, like, paying upfront, uh, for compute that you have, and then you think about maximizing the use of that compute, you think the other direction, you try to minimize your use of compute while fitting the rest of your constraints. So it's a very different feeling if, you know, if you ever switch between having your own compute and switching to cloud. It's the same idea. Um, yeah. Uh, soright. I so you could do that inference yourself. So rather than having somebody else do it for you. Um, and this works pretty well and is getting easier every day. So cloud, like, this is like, running stuff on a public cloud, um, is, like, one of the most popular choices for how to run ML workloads. ### Self-Serve Inference **Charles Frye** [1:10:21] Um, and for, like, SaaS in general, um, there's some specialist cloud providers in this space, um, like Lambda Labs that can be, like, very competitive. They're, like, often cheaper than the, um, than the, like, big three. Um, and it's a nice balance of control with, like, complexity and ha like, which things you actually care about having to deal with versus not. Um, it can get expensive over time. It's definitely, like, you know, more expensive than, um, uh, like, over a long period of time than if you bought the stuff yourself. Uh, GPUs sometimes feel like second-class citizens, and especially a lot of the, like, big public clouds. Um, uh, Google Cloud's a bit of a distinction there in that you can just a-add GPUs to any instance, which is kind of nice, but in other, other, uh, public clouds, that's not really the case. Uh, and again, this is, like, uh, a cost management problem. And, um, the one of the popular ways to solve the cloud costs is to just agree to a large deal upfront. And now you're starting to get, uh, some of the illiquidity associated with actually build buying hardware. Um, and you start to get some of the, like, vendor lock-in, um, that you would also associate with that. Um, so the you have the opportunity to kind of, like, trade those things off, but, um, they are your constraints to work with. Um, did want to call out that there are some serverless approaches, which gives you some of the, like, usage-based like, really tightly usage-based pricing associated with inference as a service providers, um, but also the, like, control associated with, like, you know, ha uh, renting servers in the cloud. And by this by serverless, I mean anything with, like, scale to zero semantics and pricing, um, uh, that doesn't involve you having to, like, literally manage servers. So, like, thinking about the operating system, uh, for example, um, and that offers high availability. Um, this is an like, a relatively new category in software in general and especially in machine learning. There's a couple of players here. Um, Modal, uh, Labs is one that I like quite a bit because it doesn't just do the ML stuff, though it is very good at it. Um, Replicate, which also does inference as a service, will do this. Uh, Hugging Face Spaces recently changed their endpoints to scale to zero. Um, and there's also, yeah, Banana.dev and, and others. Um, the good thing is that it's, like, easier to get started, especially if you're not, like, a cloud ops person, um, and very inexpensive at low traffic. Like, you only have to pay when you have traffic. And if you're, like, running a small if you're running a demo that only needs to be up when you're showing it to, um, investors or if you are working on a tiny feature at a large organization, then you might have very low traffic patterns. Oh, my. It's, uh, it's Fleet Week, I think. So that might be the Blue Angels. Um, yeah. Um. **Guest** [1:13:16] Scale to zero? **Charles Frye** [1:13:17] Scale to zero means that you, um, when there are no requests, you are not when your requests go to zero, the amount of resources that you are using and being charged for also goes to zero. Yes. Um, so yeah. For a while, Hugging Face, uh, and Spaces endpoints, they changed the there's inference endpoints and yeah. They for a while, it was like it could scale down to one, and it would auto-scale. So there's, like, having a cloud server with auto-scaling built in, and then there's that thing, but then it also scales to zero, and you don't have to think about server management. And that, that, like, is the combination. That it's the original, like, AWS definition of serverless that has kind of fallen. Not everyone goes by the old ways. Um, yeah. So inexpensive at low traffic when nobody's calling your API. Um, you don't pay for anything. If you, like, come up with a feature, it doesn't work, then it doesn't matter. Um, bad news is that you kind of generally lose, like, tight control over auto-scaling behavior that you could have if you were, like you know, if you have a, you know, Kubernetes team to work with, they can very tightly set it up so that the auto-scaling is delivers exactly the throughput and latencies p99s that you promised. Um, and you kind of give all over some of that control to these serverless providers who are themselves probably running Kubernetes, but for a lot of people. Um, and then the real the thing that has kind of prevented this from being as successful as maybe serverless architectures in many other places is latency. Um, so when you're it shows up as, uh, kind of p99 latency, so the 99th percentile of requests that hit a point when you need to do auto-scaling. Um, the you need to get the weights of the model you're using into, uh, not just off of disk and into RAM, but then from there into the RAM of the accelerator. And that takes, uh, like that can take a very long amount of time. And so you're looking at, like, 30-second, 1-minute, 3-minute cold boots in some cases because you are moving, uh, half a terabyte of data around. Um, and so that's a place where people could maybe come up with these, like, clever ways to, to cache and share. Um, but yeah. It's the memory constraint that you hit in other domains showing up, like, in disguise as latency. Um, and, uh, yeah. So the like, you still are probably going to be thinking of this in terms of, like, cost management and, and cost reduction as opposed to, like, resource maximization. Um, yeah. **Guest** [1:16:08] Yeah. So, like, that's not necessarily always true, like, when you're cold storing, um. **Charles Frye** [1:16:14] Yeah. **Guest** [1:16:14] Do you think, like, software workers, um, in software development, they would, like, I think they're, they're trying to, uh, kind of potentially use cloud as a solution. **Charles Frye** [1:16:25] Yeah. So, uh, the point was about Cloudflare workers. So I did see the Cloudf I didn't include them on the slide, but Cloudflare actually recently released these, like, GPU workers, which is their entry into this. And I haven't had time to play with it, so I don't know that much about it. Um, I think if I need to go from not consuming any of your resources to having a terabyte of my own personal bytes, like, in the VRAM of a GPU, I find it hard to believe that they don't have a latency problem there. Like, so that but so I'd be cur I'm curious what this what you know about the solution. Yeah. **Guest** [1:17:01] And worth it? **Charles Frye** [1:17:02] Yeah, yeah, yeah. **Guest** [1:17:03] Um, what, what's the. **Charles Frye** [1:17:05] Yeah. That's interesting. I'd yeah. I'd love to hear about that. That's been my experience with the other serverless GPU providers. So I'd love to hear more about the Cloudflare workers. And, um, yeah. If that goes away, then serverless becomes a much more competitive way of delivering inference. So yeah. Um, so, uh, I maintain a page for Full Stack Deep Learning that has information about, uh, like, cloud GPUs and serverless providers, pricing, and, um, uh, and what compute they provide. Um, so you can check that out from the slides later if you're interested. Um, allright. And then last, uh, let's talk, um, like, actually, what if you actually physically owned the computers that the inference ran on? Like, uh, you can do that. Uh, and rather than having to, like, actually, you know, construct a building, which only which maybe the largest, uh, enterprises could go, uh, go about, uh, using a co-location facility isn't so bad. Um, and there's more reason to do this than for other kinds of workloads. And in particular, there's actually room to beat a lot of the major public clouds, which is why there's competitive clouds, like, alternatives in this space. Uh, a lot of data centers that have been, like, around or which who that were designed before 2021 or so are configured for, like, disk and network-heavy workloads rather than power-heavy workloads. So even if you can, uh, get a hold of, like, uh, 30,000 A100s, you can't just necessarily put them in the same US East data center that used to run, uh you that was designed for, like, uh, running databases. Um, it's so it's capital-intensive but ends up being cheaper in the long run. You have total control if you if you need it, uh, which is awesome. Um, but it's very hard, very rare skill set, um, because it, like, kind of crosses this, like, the ML stuff and the, the, the hardware stuff. Um, and all of these people can go and work for OpenAI for, like, a million and a half a year. So, uh, uh, good luck holding onto them. Um, uh, and the biggest constraint that shows up is illiquidity. So you're going to make a big bet on what this looks like. For example, that inference is not going to move onto CPU or not going to move onto custom silicon that behaves very differently from graphics cards. Um, there's a great talk from Mitesh Agrawal of Lambda Labs about this that goes into kind of detail. I think it's, it's only, like, a year old if I yeah, if I remember this talkright. Um, and of course, he makes it sound very hard because he wants you to use their cloud, um, or to pay them to, like, help you build, uh, your co-location, uh, um, uh, yeah, help you actually build it. Um, but it is a detailed explanation of everything involved. And, uh, you know, there's not very many of those out there. Um, allright. We're at halftime. Uh, so I plan to take a break when I finished part one, which goes through the rest of self-serve inference, which will I'll say is another 15 minutes. So let's do that, and we'll leave, uh, an hour for part two after a little break. ### Hardware Accelerators **Charles Frye** [1:20:20] Um, allright. So let's, uh we actually haven't talked in great detail about, uh, like, you know, why why are we using GPUs in the first place? Like, what, what is actually going on here? When we run this, uh, this, like, Tensor2Tensor map with neural networks, um, like, what, what, what actually does that workload turn into? We have two tasks. We need to load numbers from memory, and then we need to do math on those numbers. Those are our, our, like, two basic tasks. Um, and that is the reason why we have why we end up using graphics processing units, um, because memory is slow and math is fast. And in most in the in the transformer architecture in particular, but in many, like, sort of most neural network architectures you might write down, um, you only need a given number from the weights w like, one time per input. So that means you need to do a memory read, like, uh, of this of, like, a couple of bytes for a particular parameter to use it in a single floating-point operation. And the memory read is going to be very slow, and the floating-point operation is going to be basically instant. Um, so in order to do this economically, you need to do a lot of math for each read from memory. Uh, you need to, like, load, uh, you know, load the weight, um, out. And in particular, we're talking here about, like, getting out of the VRAM and into the place where the you know, into the, like, um, the what is it called? Well, yeah. It's basically like an L1 cache, like, closer to the actual compu uh, uh, computation. Um, and so you want to do that and then, like, use it multiple times, you know. And, uh, that means you want to run on multiple inputs at once. And that is memory-intensive, single, uh, instruction, multiple data, parallel linear algebra, the same thing you need for graphics workloads. So the, uh, like, graphics cards have turned out to be, like, pretty good at solving this problem. Um, and the, the numbers there in the corner are, um, uh, show like, demonstrate this, uh, like, general fact of, like, memory is slow, logic or math is fast. The you can do 312 teraflops per second, uh, in for 2-byte numbers, uh, 2-byte floating-point numbers in a Tensor Core in an A100. Um, and you only get 1.5 terabytes a second of memory bandwidth. Um, and when you're using, like, you know, optimized existing CUDA kernels, these two things are, like, multiplexed. So, like, you load a weight, you start doing math on it, and then the next weight gets loaded, like, you know, uh, concurrently. Um, but you do still have this, like, mismatch in the bandwidths. That means that you want to be able to, like, amortize the memory load across as many computations as possible. And, like, in principle, this could be flipped around, and we would have, like, very you know, things would look very different. Um, uh, so you can get, like, very, very large throughput gains by amortizing memory reads where basically, if you are operating on a very small number of tokens, uh, then you'll see that as you add like, if you're running this, this workload yourself, as you add more tokens, you would think, like, you should expect, like, a s a slight increase in the amount of time that it takes. And you'll see, like, basically a flat curve for a very long time. Um, and then once you hit the point where the, um yeah. So if you look batch size at which you'll see that flip. So for an A100, it's about 200, uh, uh, elements in the batch. Um, and you'll see for an H100, you'll see that that ratio these numbers both go up, but the ratio becomes more extreme. Um, so you, uh there's a great blog post from Carol Chen on, like, inference arithmetic that both goes through in greater detail and then, like, matches that onto some, like, actual experimental results. And it's able to, like, track where did each, you know, microsecond basically of, uh, of inference time come from. Um, so the, like, key takeaway from this is that if you want to get large throughput in, like, an inference system that you're running yourself, you're going to need batching. You're going to need to, like, collect up multiple inputs, uh, from multiple users and operate on them at the same time. Uh, so this is, uh it's challenging to achieve the same thing in an end-user device. Um, you are like, if you're only working for one person, then they might not make 10 requests, uh, you know, quickly enough for you to fill up a batch. Um, and that actually might kind of tilt things in the direction of, uh, of CPUs or of, uh, you know, uh, different, uh, different architecture with different, um, you know, memory bandwidth versus Tensor Core uh, versus, uh, math bandwidth, um, uh, trade-off. Um, the one of the useful things that came out of this I guess this is a restatement of kind of what I just said. When you have a, like, low-load API, um, you will end up with smaller batch sizes because maybe you have you have to deliver with a certain latency, so you can't just wait forever. Um, and so you will make different decisions about compute memory trade-offs. For example, like, caching past computations of, uh, keys and values is very common when you're doing batch inference, but it's actually not necessarily theright choice when you are already memory-bound. Trading off memory for compute, um, isn't a good idea. Um, if you're, like, at for larger batch sizes, um, you if you're doing something that's an API where, like, serving requests directly, then you would want to, like, roughly balance being, uh, flop-bound and memory-bound so that you can get quick latency of responses even though your throughput is less than it would be otherwise. Um, whereas if you're doing, like, a batch job, like, an overnight-type job, um, or that is, like, typical of data science or, or big data workloads, then you would just go for the largest batch that you can. And that's what people do during training. They go for the absolute largest batch that they can because there's no latency requirement. Uh, there's only throughput. **Guest** [1:27:01] I'm assuming all this still applies to, like, other accelerators like TPUs or, like, different GPUs or these kind of chips? **Charles Frye** [1:27:07] Uh, yeah, yeah. So TPUs end up I, I, like, looked around, and they're I-I've never personally used them for anything serious, so I wasn't able to, like, directly map on, like, pull out the equivalent of a Tensor Core, um, like, flop bandwidth and the and v and VRAM to L1 latency. Um, but the results and benchmarks that I've seen is that they're, like, 30% better. Um, like, they, they have a di they do have a slightly different choice of trade-off. Um, and they and, and it's better for neural network workloads than graphics workloads. And that gets, what from what I have seen, like, 30% improvement, but not, like, a 10 to 50x improvement, which is what you would really like to see if you're making as drastic of a, a decision as, like, going over to a completely different accelerator with a completely different software stack. Um, yeah. I guess yeah. I had I got into a discussion with, uh, one of the people on the TPU team about, like, the hardware lottery, and he his he was like, "The GPU is already, like, is just an excellent machine. Like, it does this workload really, really well." Um, and I think that is ba worn out in the numbers. Yeah. **Guest 2** [1:28:18] From what I heard, the main advantage of TPUs is, uh, operational efficiency and power efficiency, not, like. **Charles Frye** [1:28:26] Mm-hmm. **Guest 2** [1:28:27] Input or performance. **Charles Frye** [1:28:28] Yeah. So, um, for the folks listening online, repeating the point, uh, the one of the benefits of TPUs is power. I totally buy that. Um, yeah. Uh, I think yeah. The numbers that tend to get reported in things like Google's Pathways paper and Palm Papers or Open and, like, what is known about OpenAI, it's like it's all about, like, flop utilization and, and total flops and stuff like that and not things like power that do matter. Um, I'd say, like, people have not gone rushing to try and get a hold of them, which feels, uh, uh, like a strong signal. Um, similar with, like, other types of custom silicon, like Cerebras's, um, uh, chip. Uh, but, you know, who knows? I think this is getting back to a question that got asked earlier. Um, I believe the, like, having custom chips works really well when workloads stay very fixed in, like, kind of precise detail, not just this, like, "Oh, we need to load from memory, and then we need to do " but, like, "No, we need to do this shaped thing, like, with this many " like, that works super well. And you can see that in blockchain mining where there is, like for a for many chains for a long time, ASICs were the, like, only profitable way to mine, um, or, like, the most profitable way, way to mine. And that's the workload is, like, unchangeable except by, like, distributed consensus. Um, and that allows you to, like, very tightly target a specific workload, whereas, like, neural network architectures actually kind of change reasonably often. And the, like, you know, even the difference the difference between a 7-billion-parameter model, a 13-billion-parameter model, 170-billion-parameter model, that's, like, a bigger difference than you would see between workloads for, um, like, the same blockchain over time. Um, and so that's that, like, difference in tech like, that technical difference is, I think, a big reason why we haven't seen, uh, much uptake of custom, custom. Yeah. I think that, like, they are one of the, um they're one of the providers for, um like, they're one of the startups working on this. Pe they come up most frequently. Um, uh, I don't it's still, like, uh, fairly experimental, I think. Like, it's not it's not the, um it's not, like, generally available in a public cloud or something. So, um, yeah. I don't have much to say about it, I guess. Yeah. **Guest 2** [1:30:57] I think the main thing you can see there is their the, the memory that they have. **Guest** [1:31:02] Yeah. **Guest 2** [1:31:02] Is 40 gigabytes. Compare that with an A100, which has 80. **Guest** [1:31:07] Yeah. **Guest 2** [1:31:07] And for most of LLMs, memory is your biggest problem. So. **Guest** [1:31:10] Only the LLM they trained is, like, JSON, Arabic thing,right? But. **Charles Frye** [1:31:14] Mm-hmm. **Guest** [1:31:15] They, they trained this Arabic LLM on this one, but you haven't what we know. **Charles Frye** [1:31:19] Yeah, yeah. So Cerebras has done some large model training, so it's definitely, like, possible. Um, they, um the, the they have this very fast memory bandwidth, I think, is, like, maybe the, the big one, 20 petabytes memory bandwidth. Now, like, the, the point that was raised about the bottleneck being the memory capacity is very is, uh, is well taken. Um, you can network multiples, multiple chips together, and then you have more storage. But, um, yeah. And so, like, bal better balancing of, of memory bandwidth and math bandwidth would maybe get you more effe like, more efficacy at lower batch sizes. Um, but you can't exactly put that chip in a phone. Um, so that yeah. Um, but definitely, uh, a like, people are really converging on transformer architectures, so that could mean that this, um, custom silicon works better, um, in five years. Um, yeah. I think I yeah. The way that you like, the way that you solve this problem is if you have multiple GPUs, these the, like, memory per second and flops per seconds, like, scale at the same amount,right? You have now you have two GPUs loading. You have two GPUs doing operations. They both scale linearly. If you're really clever, you can, like, actually get that linear scaling in practice. Um, the GPU memory goes up as well. Um, and so you can serve larger batches and eventually hit this ratio is staying the same, so you can eventually hit that point. Um, and that's what and then, like, once you hit the point where you're actually, uh, you know, flops-bound, um, you can just, like, crank it you know, just crank it up, uh, just get really big, uh, uh, really big batches, speed up, uh, speed up the computation. And that is what people do during are, are, like, used to do used to doing during training. Um, and so you'll very commonly see people talking about this way of making inference efficient. Um, so if you are running a, um, like, a web service, this is going to give you your sort of, like, uh, pod size roughly, like, um, like, a pod in Kubernetes is, like, a bunch of services that end up on a single physical machine. So this is your, like, unit, uh, um, uh, is determined by, like, how many GPUs do you need to hit the, like, batch size, um, for efficiency implied by the, like, uh, this flops memory per second. And you're checking to make sure this is also true for your architecture. Um, yeah. Um, all that's, like, kind of theoretical stuff or, or sort of, like, trying to be relatively first principles. Uh, when you when it comes time to actually check whether you're doing things correctly, make sure to use, like, a profiler or a tracer and actually, like, look at these things. Um, even, like, people who are pretty good at this will, like, miss stuff that shows up on a on a tracer. Um, like, the VLLM, uh, implementation was launching a CUDA kernel for, like, every individual element of a batch. And, like, a s a tiny change ch made it switched to, like, one CUDA kernel launch, and that saved them, like, uh, uh, increased their throughput by 30, 33%. Um, and it's something that shows up as, like, a t like, way more CUDA kernel launches, which would show up as, like, I guess yeah. If I had the interactive version of this, it would show up as this, like, huge thing of red lines. Um, and also just the, like, general utilization shows up as these big, uh, like, patches of gray here, which shows you where you're, like, not actually using your GPU at all. Um, and, uh, I found, like, looking directly at these, um, at these traces, not just at statistical profiles, helps, like, catch those, uh, those kinds of, um, easy-to-fix, 80/20 kind of bugs. Um, you can also profile and trace memory, uh, and that's going to be, like, given that memory is this key constraint, that's likely to be at least as useful. Um, and these two are examples from PyTorch. Um, these, uh, this the con compute trace and the, um, the memory trace both from PyTorch. You might not be using PyTorch, um, so you might have to fall back on generic, like, GPU profiling or CPU profiling or memory profiling and tracing tools, um, if you're using, like, a custom inference solution. Um, yeah. Speaking of which, there's a bunch of specialized LLM inference libraries out there. There's a nice blog post from Hamal Hussein about using these, and this is specifically for, like, batch size one. Um, and so the, the VLLM is people are, like, pretty that seems to be getting, like, the most community excitement and community contribution. Um, but Hamal suggests some reasons why you might be into MLC and CTranslate too instead, um, for doing your, uh, LLaMA inference. Um, so actually so all that was about, like, just thinking in terms of, like, an individual workload, um, setting up an entire inference service, um, which is now something that maybe s like, somebody in the company sets up an inference service or a inference platform, and then, uh, people can, you know, submit workloads to it. ### Inference Deployment **Charles Frye** [1:36:33] That can be fairly challenging. Um, containerization for GPU-accelerated workloads is less painful now than it used to be. Um, like, NVIDIA Docker actually works in a way that it did not, um, like, early on. Um, but, uh, containerization is, like, fundamentally, uh, you know, more dubious for these, uh, these workloads than a lot of other ones. So one is that the, like, application layer of the container is probably where the weights are going to live. Like, it doesn't live in the operating system yet. Um, it's not built into the Docker engine. So it's like that's up there, and that's, like, maybe half a terabyte in, in, in bad situations. Um, so now your container images are really large, and that's, um that can be pretty unpleasant. Um, you could try and do the things that people do with databases and, like, move it into remote storage, but, um, yeah, then the container is not as, like, u unitary. Um, and then the, like, the worst problem maybe is that the CUDA driver changes the choices that you need to make kind of at the, like, application level of, like, we were just talking about actually going one level down. The, like, which GPU it's going to run on changes the point at which you switch from being memory-bound to flops-bound because that has a different, uh, like, memory bandwidth to math bandwidth ratio. Um, so things at the level of the NVIDIA of the GPU are entangled with things at the application layer, um, like choices of, like, batching strategy. Um, so that's and a similar thing can be said for the CUDA driver. I was, like, naive about containers, and I was like, "Why is the, like, CUDA driver showing up as 12.0, which is what I have installed in my host operating system, when I specifically downloaded the CUDA 11.2 container?" And it's like, containers can only virtu only virtualize so much. Um, and so that also changes, like, which kernels are available by default. Um, uh, I guess, well, kernels would be a layer above. So there are features of CUDA of CUDA drivers that change the, like, you know, how Hugging Face transformers or PyTorch will work. Um, and so that will and that will also change decisions that are made at the application layer. So when you're in these, like, super pro like, this is clearly a performance-limited regime that we're talking about here. Like, the inference is, like, extremely expensive. Um, and we are, like, trying to maximize performance really aggressively, and that, like, s starts to reveal the limitations of virtualization and containerization. Um, so that doesn't mean it's impossible. It just means it's hard. Um, and yeah. I don't know. If you've ever w-worked with, like, you know, trying to set up a heterogeneous compute Kubernetes cluster where there's, like, some have GPUs and some have ARM and some have, uh, like, x86, you like, this dealing with this requires a, a better, um, uh, ops engineer than if you can ignore that. Um, speaking of which, this, like, application serving can be and is, in fact, done with the, like, industry standard for container orchestration in Kubernetes. Um, I would note that now the, like, NVIDIA stuff is showing up here again and, like, all the problem like, the entangling th and now Kubernetes is in between them. So now there's, like, a lot of opportunity for cross-talk, a lot of opportunity for tiers. Um, and, uh, so this is a hard mode version of that problem, which is not notorious for being easy. Um, so you might choose and need to do it, uh, by hand. There was a comment on r/mlop that was like, "Oh, yeah. We look " uh, you know, I was looking around to see what the opinions are on these tools. And, uh, uh, opinion I saw in a couple of places was, like, if you care about like, you've chosen to do serving of inference yourself and not do, like, just API calls, then, um, maybe you should make MLOps a core competency of your company, and you're going to want to do something more like building it out of the, like, Kubernetes af uh, affiliated ecosystem of open-source tools or other open-source things, um, like kubray to do, um to run Ray on Kubernetes for serving or, uh, Selden Core to run that on, uh again, on Kubernetes as your container orchestration layer. Um, but given how, like, painful that is and the fact that there are so many people who are trying to run a relatively similar workload, um, you might consider, um, like, either of two kind of tiers of managed services, either the, like, white-glove end-to-end kind of cloud provider approach, the oldest one, Amazon SageMaker, um, more recently, Vertex AI from Google. I'm sure there's an Azure version that I'm forgetting. Um, there are also from some startups, a sort of more toolboxy approach, a less end-to-end approach, in part because it's not, like, integrated all the way at the layer of the, like, actual hardware like the cloud providers are. Um, so, uh, Bento Cloud, uh, Selden, and AnyScale. AnyScale, um, being a managed version of Ray. Selden offers a managed version of Selden Core. Bento Cloud, managed version of Bento ML. Um, so, uh, depending on where you want to, maybe you have a really great sweet deal with GCP, and so Vertex is theright choice. Um, but yeah. I think from I have had limited experience with these things because I've tried to keep my life, uh, simple and happy. Um, but if you're, uh if you end up in this space, I've, I've seen some nice things and played around with RayServe, um, and AnyScale. Um, Ray is the tool that a lot of people use for cluster management for training. So a lot of the teams that train the foundation models use Ray. Um, and so there's kind of, like, a natural competency for them, uh, both in the open-source library and AnyScale as a as a, um, layer around that. So maybe it'd be a good choice for, um, for inference as well. Allright. Um, so rather than take questions, I think I'm going to do a five-minute break in which you can ask me questions up here while we all get a little stretch, maybe a little air, um, and we'll come back for the rest of the hour in five minutes. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.