# Why TTS Models Now Look Like LLMs — Samuel Humeau, Mistral AI Engineer · 2026-05-09 Mistral AI scientist Samuel Humeau explains why text-to-speech models now resemble LLMs, using an autoregressive transformer with neural audio codecs to compress 200 kbps audio into ~500 tokens per second. He demos Mistral's open-weight TTS model, cloning a voice from a few seconds of reference audio and generating speech with 17 ms first-token latency via frame-by-frame streaming. The talk covers the codec-backbone-decoder pipeline, the model's 4B-parameter backbone generating 37 tokens per 80 ms frame via a diffusion decoder, and a live voice agent answering conference schedule questions. Humeau notes the next latency win is handling streaming text input from an LLM, with no settled architecture yet, while keeping the encoder for voice cloning proprietary. ## Questions this episode answers ### Why are modern text-to-speech architectures converging to look like large language models? Samuel Humeau from Mistral explains that the dominant pattern is an autoregressive transformer generating audio one frame at a time, inspired by LLMs' success at modeling token sequences. To avoid generating sample by sample, a neural audio codec compresses 80-millisecond frames into tokens (around 37 per frame), reducing the information density from 200 kbps of audio to about 500 tokens per second, making it manageable for a transformer backbone. [8:55](https://aie.addtry.com/6f076164-c6ee-4732-8ac5-bcf23d225688?t=535000) ### How does streaming the first audio packet reduce perceived latency in text-to-speech systems? Humeau demonstrated that by generating and playing the initial audio segment as soon as it's ready, rather than waiting for the full audio to be computed, the perceived latency drops significantly. In his voice agent example, the first audio packet arrived quickly, allowing the agent to start speaking while the rest of the utterance was still being generated. This trick is critical for natural-feeling voice interactions, where the user hears a response almost immediately even if the full audio takes longer. [3:20](https://aie.addtry.com/6f076164-c6ee-4732-8ac5-bcf23d225688?t=200000) ### What voice cloning capabilities does Mistral's open-source TTS model offer? Humeau showcased that the model needs only a few seconds of reference audio to clone a speaker's voice with high fidelity, including preserving a strong accent when speaking other languages (e.g., a French speaker cloning into English). He demonstrated cloning his own voice and having a conversation with himself. The open-source release includes pre-set voices, but the voice encoder for cloning arbitrary voices is currently only available through Mistral's proprietary service to prevent misuse. [6:00](https://aie.addtry.com/6f076164-c6ee-4732-8ac5-bcf23d225688?t=360000) ## Key moments - **[0:00] Intro** - [0:15] Mistral AI released its first open-source text-to-speech model last week, says Samuel Humeau. - [1:49] The king use case for text-to-speech is its usage within AI agents, says Samuel Humeau. - **[2:06] Latency** - **[3:33] Voice Cloning** - [3:51] Mistral's TTS model clones 'Paul's voice from a few seconds of reference audio in a live demo. - [4:19] Streaming the first audio packet before full computation completes greatly reduces perceived latency in TTS. - [5:04] Mistral's voice agent answers real conference schedule questions using the cloned voice of 'Paul'. - **[6:00] Multilingual Voices** - [6:31] Mistral's TTS generates English with a strong French accent, preserving the original speaker's accent across languages. - [7:00] Samuel Humeau clones his own voice with Mistral's TTS, demonstrating easy voice impersonation. - [7:21] Vocal identity will become as important as brand identity for companies, predicts Samuel Humeau. - **[8:01] Audio History** - **[8:55] Transformer Architecture** - [8:55] Most text-to-speech labs now converge on an autoregressive decoder backbone, generating audio one piece at a time. - [9:21] Modern TTS generates patches of audio (80ms frames) rather than individual samples. - **[10:00] Information Density** - [10:01] Audio requires a much larger bit rate than text; one token from a 1k-word vocabulary carries only 10 bits of information. - [10:35] Text captioning reduces acoustic information to just 15 bits per second, a massive reduction from audio's 200kbps. - **[11:39] Neural Codecs** - [11:39] Neural audio codecs compress audio to about 500 tokens per second, reducing information density for transformers. - **[13:10] Frame Generation** - [13:29] To handle 500 tokens per second, TTS models use one backbone step per frame and a smaller decoder to generate frame tokens. - [14:22] Mistral's TTS model generates all 37 tokens per frame at once using a flow matching diffusion model, diverging from the common pattern. - [14:54] TTS conditioning approaches split into providing all text upfront versus streaming text input during generation. - **[14:56] Text Conditioning** - [15:50] Mistral's TTS uses a few seconds of reference audio and the target text as context for voice cloning. - [16:06] Mistral's TTS model achieves 17ms latency from text input to first audio packet on a single GPU. - **[16:22] Streaming Text** - [16:22] Handling streaming text input from LLMs is the next step for latency wins, but no clear architecture winner exists yet. - **[17:35] Q&A** - [17:36] Q: In Mistral's voice agent demo, were text and audio generated simultaneously? A: No, a fast LLM produced all text first. - [18:24] Q: Is Mistral's voice cloning encoder open-source? A: No, it's proprietary; only pre-selected voices are available. - [19:23] Q: What is Samuel Humeau's take on native voice-to-voice vs cascading architectures? A: Cascading with streaming text tokens can go very far. - [20:44] Q: What does interleaved audio-text streaming enable? A: Lower latency by voicing LLM text tokens as they arrive. ## Speakers - **Samuel Humeau** (guest) ## Topics Text-to-Speech, Voice Agents ## Mentioned 11 Labs (company), Google (company), Mistral (company), OpenAI (company) ## Transcript ### Intro **Samuel Humeau** [0:15] So I'm from Mistral AI, and we're going to talk about speech generation and text-to-speech. There's an occasion: we released last week our first text-to-speech model, and it's open-source, so I really encourage you to check it out. It's an extremely strong text-to-speech model; we're very proud of it. And for this occasion, I thought we could review some of the recent trends in text-to-speech architecture, since there's a dominant trend emerging these days, although this can change very quickly. And so this talk is slightly academic and addressed to people who want to know a bit more about how you do text-to-speech. This being said, we have a few years before the machines do all the science for us, so we might enjoy it today. I'm Sam, yeah, I work at Mistral as an AI scientist. Before I was at Facebook Fair when it was called Facebook. And Mistral, a few words about the company: we're a frontier lab, we've been founded a couple of years ago. We produce frontier models, but we're also a B2B business. We help organizations in their AI transformation, which is kind of a buzzword, but literally every company is transforming with AI. We help them by providing them tools, products, and dedicated people to help them in their custom needs. Back to the text-to-speech: there are a few offline use cases of speech generation, like the famous "listen to the blog" or "listen to the article." But nowadays the king use case for text-to-speech is its usage within agents. And in particular, it's used to interface with a chat agent, typically in a pipe like this, where you have a central chat agent that does text-to-text, but does it extremely well. ### Latency **Samuel Humeau** [2:19] And you want to talk to it, so you add a speech-to-text, and you want it to speak to you, so you add a text-to-speech. As everybody in this conf will tell you, the latency is key here. So you can reduce the latency on the left by having the speech-to-text done in real time, so that when you detect the end of turn, you already have the transcript, it's already done. And we're going to focus a bit on theright side today. It's also very important that as soon as you have the first audio packets, you start to voice them out. This way the perceived latency is lower. In fact, since your LLM can stream some text to you, actually what you ultimately want is something like this if you're going to interface a chat assistant, which is a real-time text input, text-to-speech, where as soon as you have the first token of the LLM, the machine starts to speak. We're going to talk about it in the end of the talk. I want to focus a bit at the beginning at the output side and what it means to stream audio. So to illustrate this, I have this app that I vibe-coded for the occasion. ### Voice Cloning **Samuel Humeau** [3:38] And so we're going to use this text-to-speech model that we released, that I mentioned, and we're going to hear Paul. So Paul is an actual human being that sounds like this. **Speaker 2** [3:51] The persistent anxiety that fills the rest of my life is calmed for as long as I have the flavor of something good in my mouth. **Samuel Humeau** [4:01] So this is like some actual recording on some actual person named Paul, and we're copying his voice. **Speaker 2** [4:09] And so with the sunshine and the great bursts of leaves growing on the trees, just as things grow in fast movies, I had that familiar conviction that life was beginning over again with the summer. **Samuel Humeau** [4:19] So let's focus first on what's happening here. As you can see. **Speaker 2** [4:24] And so with the sunshine and the great. **Samuel Humeau** [4:25] We're copying the voice, and the first audio packet happens first, and we can start to emit audio, which greatly reduced the perceived latency, even though the full computation of the audio happens a few seconds later. **Speaker 2** [4:39] And the great bursts of. **Samuel Humeau** [4:40] So if you use it in an agent, so here I crafted a small agent using a speech-to-text, one of our LLMs, and this very text-to-speech. So we can speak to Paul. And hey Paul, can you tell me what's the title of the session at 12:20, please? **Speaker 2** [5:04] The session at 12:20 PM is titled "Reachy Mini: Giving a Body to AI" by Andrés Marafioti. **Samuel Humeau** [5:11] And what was the session at 11:15, please? **Speaker 2** [5:20] The session at 11:15 AM is "Beyond Transcription: Building Voice AI That Actually Understands Conversations" by Hervé Brédain. **Samuel Humeau** [5:30] Did you enjoy it as much as I did? **Speaker 2** [5:34] I don't have personal experiences or emotions, but I'm glad you enjoyed it. **Samuel Humeau** [5:38] Right. That's all I can do. So the important thing here is that since the audio packet arrives first, you still have a decent latency, and you can enjoy the conversation with the agent despite the fact that the audio is still not generated fully. And so we're going to dig, oh yeah, sorry, I want to make one digression. So I mentioned the voice cloning here. This model can only need a few seconds to clone the voice of someone. So again, this is how Paul sounded, and this is how we generate text. ### Multilingual Voices **Speaker 2** [6:15] And so with the sunshine and the great bursts of leaves growing on the trees. **Samuel Humeau** [6:18] It really sounds alike. It's also very good at inferring how a person would speak in another language. So for example, this is a voice, a French voice, maybe. **Speaker 3** [6:32] Ryan Coogler, réalisateur et scénariste des deux Black Panthers. **Samuel Humeau** [6:36] And if we generate. **Speaker 3** [6:37] And so with the sunshine and the great bursts of leaves growing on the trees, just as things grow in fast movies. **Samuel Humeau** [6:43] Yeah. So we can clearly recognize her, and we can clearly recognize a very strong French accent, which as a French myself, I do enjoy. I can even clone my own voice. So this is how I sounded during the recording. **Speaker 4** [6:59] Hi, this is Sam. **Samuel Humeau** [7:00] And this is how I sounded. **Speaker 4** [7:03] And so with the sunshine and the great bursts of leaves growing on the trees. **Samuel Humeau** [7:07] It works pretty well. So this way, in my time of delusion and at the peak of my ego, I can discuss with myself on a complicated problem, which is nice. And so it's becoming so easy to impersonate a voice that it's becoming very easy to configure. So it's a small digression, but currently, actually, a lot of large companies, they do have a concept of vocal identity, and they do care in their branding about how they sound in their advertisement in particular. But I think this concept will becoming more mainstream. And just as a lot of companies define how their website appears as their brand identity, it will be the same for the voice identity. Oops, sorry. Back to how we do it in general. Soright, this is an audio. Physically, it's the pressure of the microphone that we measure from time to time, like several thousand of times per second. So it looks like this. And historically, to generate the audio, there have been a lot of attempts, a lot of systems. ### Audio History **Samuel Humeau** [8:20] Like in the prehistoric time, you have stitching of words that were spoken, like in the French train system, SNCF, for those who know. And then a neural generation arrived. At some point, the trend was to generate each sample one after the other. Then another era was generating the whole audio at once. But as we saw, it's very interesting to have the beginning of the audio generated first so that we can start to play it out. So it seems that most labs have converged to some common patterns. ### Transformer Architecture **Samuel Humeau** [8:55] And obviously, the first one is inspired by large language models. We're trying to transform the problem as a language modeling problem because humanity is extremely good at modeling sequences of tokens. So pretty much everybody is using an autoregressive decoder backbone and generates audio one piece after the other. Now, as I said, we really don't want to generate one sample after the other. So what we want to do is generate patches of audio one after the other. So the expected system looks like this, where you have an encoder that transforms a frame of audio, something like 80 milliseconds, into something that ideally is a token, because humanity is very good at modeling sequences of tokens. And then you have a decoder that does the opposite. Now, for text, it's pretty easy because transforming the text into tokens, well, it's easy,right? You can take words as tokens, and it works pretty well, even though we do much better. ### Information Density **Samuel Humeau** [10:01] For audio, it's much harder because one token doesn't have a lot of information. Like one token of a vocabulary of a thousand is 10 bits of information, and the audio requires much, much more, like a much larger bit rate. For example, a standard quality MP3, that's 200 kilobits per second. And so in order to transform this into a sequence of tokens and not have thousands and thousands of tokens, we need to somehow compress it, reduce the size of it, maybe drop what's not needed. An interesting point of comparison is text captioning, because if you drop all the acoustic information and you just focus on the text, like with a subtitle track, you actually drop most of the information. It's a massive reduction, and you only have a few bits per second remaining. So here in this demo, I use our real-time speech text to measure my bit rate in terms of tokens per second of text. And I'm a very competitive person. I'm very good at speaking, yet I'm barely 15 bits per second of actual information. You can try to beat that, but in the grand scheme of things, compared to 200,000 bits of information per second, that's not a lot. Obviously, we want to use something that allows us to recover acoustic features, like the voice and other aspects, and not just the semantic information as in the text. So the codecs that are used typically reduce the audio to about a few thousand bits per second. ### Neural Codecs **Samuel Humeau** [11:50] In our case, for instance, we treat the problem with, sorry, we cut the audio with pieces of 80 milliseconds, so 12 frames per second, and we transform each frame into several tokens, like 37 in our case. So we reduce the problem to about 500 tokens per second. I'm not going to dig too much on how we do train these codecs, but obviously, first we train them. We train them by reconstructing a very large set of audio and using a bottleneck here. The training procedure constrains the reconstruction to go through a step where each frame is decomposed into several tokens. To do this, typically it's guided so that the model drops information that is useless and only retains the one that is useful. And so we guide it via some losses: reconstruction losses, adversarial losses, and particularly for some of the tokens, we try to make sure that they contain the text information so that you can reconstruct the text from it. Still, 500 tokens per second is a lot of tokens. And you could put them one by one aligned as a sequence like this, but it would make a lot of steps of the main transformer that is at the core of the system, which is ### Frame Generation **Samuel Humeau** [13:29] huge. In our case, it's 4 billion parameters, sorry, which is still a lot, even though it's not extremely big now. What most people do then is having one step of the backbone per frame and a smaller model here, typically a dev transformer that recomputes all the tokens of one frame at each step. So this way, you still have a lot of tokens. You still carry a lot of information, but the computation is much faster. So this is the main pattern that we see. Even though for that last bit, the model we released does not follow this pattern, we actually differ on that part. I'm not going to dig too much on it, but just so you know, each frame, which is represented by 37 tokens in our case, we do generate these 37 tokens at once using a diffusion model. So it's slightly different from the vanilla text-to-speech nowadays. I encourage you, by the way, to read our technical report, which contains all this information. Also, it's a pretty cool use case of flow matching models, which is similar to diffusion models. Now, the main part is conditioning, actually, because so far we're just generating audio, but we are not conditioning it on text. So it's not really a text-to-speech. It's just speech. And ### Text Conditioning **Samuel Humeau** [15:08] for conditioning, there is way more variance across labs and papers and implementations. You have typically two categories. There are the people who focus on producing the audio once you have the text, and some of them who focus on having a stream of text. Typically, the first category will tend to provide all the context at the beginning and then produce the audio as we saw. And typically, the second category will also add some context as the audio is produced. The model we released is in the first category. So what we do is we provide the audio of the voice we want to clone, so a few seconds, then the text to pronounce, and that's our context in our case. Yeah, regarding the latency, so it's pretty fast if you remove the network. And with a single GPU, you have 17 milliseconds between the moment where you input your text and the moment where you have the first audio you can play. ### Streaming Text **Samuel Humeau** [16:23] Regarding real-time text input stream, which is our next step for us, there is no clear winner. First, it's still possible to generate independently the text and stitch them out, but obviously, you will have a lot of continuity problems. And there are several patterns. The two main ones are people who interleave audio and text. So as soon as there is a new text, they put the text in the same layer. And some others who have a dual stream architecture where you have a stream of audio and a stream of text, and you kind of blend them together during the inference. How am I doing on time? There's two minutes remaining. Thank you. The takeaway is check our open-source model, please. Read the technical paper, and I hope you learn a few things today. We do have two minutes. How you spend them is up to you. Yeah. ### Q&A **Guest** [17:36] You said that your model first takes all of the text and then produces the audio, but on the example that you showed of the voice agent, it seemed like it was generating the text and the audio at the same time. **Samuel Humeau** [17:47] No. So the question was like, A, on the demo, it looks like we are generating the text and the audio at the same time. No. For the voice agent, hello, Paul, it's me again. Can you say anything like, I don't know, a poem? **Speaker 2** [18:06] I'm afraid I can't recite poems. **Samuel Humeau** [18:08] So the text is produced in one go. It's just that I'm using a small LLM that is very fast. So it's nearly immediate, and then the audio is produced later. Yeah. Yeah. **Guest** [18:24] So thanks for a great model, by the way. I know that the weights are open. Is the voice cloning encoder also open? **Samuel Humeau** [18:31] Yeah, there is a small asterisk here. We didn't release this part, like the encoder part, which means it's the only thing that is missing for you to clone your own voice. It's a feature that we only serve in a proprietary fashion for now. So what you can do is use the text-to-speech model, use one of the open voices that we provide. We may provide more in the future. Yeah, so far we just didn't want to give everybody the ability to clone any voice. Yeah. **Guest** [19:23] So big labs like Google and OpenAI are working on native voice-to-voice models. **Samuel Humeau** [19:31] Yeah. **Guest** [19:32] And 11 labs are doing more this cascading architecture. What's your take on both, like paralleling, and how do you see things evolving? **Samuel Humeau** [19:43] What's my take? So on the consumer side, you will always have the impression to speak to a single system that hears what you say and outputs something,right? So it's purely an architectural model here. My take on this is that we can go very, very far by just using speech as an interface, especially because these central LLMs, they are extremely capable, but they also do a lot of things. So just for the sake of being able to use any agent that has been released with the same interface, it has an advantage to interface. So you can go very far with just interface, especially if you are doing this kind of thing where you stream the text tokens that are output by the LLM. Yeah, last question, because I'm out of time. Sorry. **Guest** [20:44] Yeah, so the next steps on the interleaved function, which accepts both audio and text, sounds really interesting. So if you do that real-time, what do you see as the possibilities with that feature? **Samuel Humeau** [21:01] So I didn't say that our next step would be this,right? I just said that there are several patterns to handle a stream of text as input, as opposed to a finite amount of text. We actually don't know which one we'll choose, whether it's interleaved or another solution like delayed sequence modeling, for instance. So it's unclear which architecture is best, at least to us, at least to me. What it allows is lower latency, because as soon as you have the first bit of text that are produced by the LLM, you can start voicing them out. So in this agent, that was not clear because the utterances were very short. But imagine I ask Paul to generate a full page of text. It would be nice if I don't have to wait the end of the text generation to voice it out. **Guest** [21:58] Yeah. Yeah. So it also increases kind of the UX side of things using this type of system. **Samuel Humeau** [22:04] Yeah, absolutely. Yeah. **Guest** [22:05] Very nice. **Samuel Humeau** [22:07] Thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.