Text-to-Speech Data Preparation and Fine-tuning Workshop - Ronan McGovern

Welcome to this workshop on text-to-speech model fine-tuning. I'm Ronan of the Trellis Research YouTube channel, and today I'll be walking you through data preparation and fine-tuning of Sesame's CSM-1B model. You should be able to follow along; all of the materials are available on the Trellis Research GitHub if you go to aiworldsfair-2025.

You'll be able to find the collab notebook and also the slides that I'm going to go through for this workshop. Now, by the end of this workshop, you should be able to train a text-to-speech model so that it sounds like a specific voice.

You should be able to understand how token-based text-to-speech models work. The one we're going to train today is Sesame CSM-1B, but Orpheus from Canopy Labs is also a token-based text-to-speech model. We'll go through how to create a voice dataset.

You can record one yourself, but today we're just going to pull audio from a YouTube video. You can pick which video and use that as a basis for the fine-tuning. We'll cover then fine-tuning using Unsloth, which is a library based around transformers.

Many of you might be familiar with it. Last of all, we'll evaluate performance both before the fine-tuning and after the fine-tuning. Now, before I start to get into the collab notebook, it's just going to be one collab notebook that covers all of these steps.

I want to describe a little bit how these models work, very briefly. If you do want to understand in much more detail, I'll point you to a much longer video about an hour and 30 minutes where I talk about token-based text-to-speech models.

I also talk a little bit about Moshi. I'll put a link to this in the GitHub repo as well. It helps to start off with reviewing how a token-based text model works, such as, say, the ChatGPT models from OpenAI, like GPT-4.0 or LLaMA series of models.

Those models, which you'll be familiar with, most likely involve taking in a string of text and predicting the next word or token, and then passing that around so that you recursively predict the next word. Now, when we talk about text-to-speech, what we want to do is take in a series of text tokens and predict the next audio token, if you can bear with me as to what an audio token might be.

And in fact, we'd like to be able to pass in a history of text and also audio from the earlier conversation, and then decode recursively we'd like to produce the next audio token so we have a string of audio tokens that represent the output speech that we want.

So conceptually, this is what we're trying to do. We need a transformer model that, instead of taking in only text and outputting text tokens, can input, ideally, text and audio and output a string of audio tokens. So the first question then is, how do you make tokens for audio?

How do you take a sound wave like this and represent it in terms of discrete tokens rather than continuous frequencies or sound waves? And the answer is, you can represent audio, or a little piece of audio, as a choice from a codebook.

A codebook is like a dictionary, and inside the dictionary, there are a series of vectors where a vector represents a given sound. Now, these codebooks could be fairly large in order to represent a wide variety of sounds, and that could be the acoustics and also the semantics.

But it is possible to take sound and represent it in a discrete way, much like text represents meaning in a discrete way. The way that these codebooks are trained, I won't get into too much detail, but broadly speaking, you need to train an encoder-decoder where you take a sound wave, you then have some kind of transformer that converts it into tokens, and then you have another transformer that will decode it back into a wave.

And the question you asked during training is, what are the output here after encoding it to tokens and decoding it back? You ask the question of whether it matches the input. And to the extent it doesn't match, you will use that difference or the loss to backpropagate and update these weights here.

So it is possible, giving sufficient data on sound waves, to then train a codebook so you can represent that sound in a discrete form. Now, it turns out, if you use that approach, you can represent sound using a codebook, but it turns out insufficient to just have one token for each time step.

Actually, what works out better is to have a hierarchical representation, or at least have multiple tokens allowed at the same kind of time stamp or same window so that you can have a more detailed representation. So this is the idea that's used in most token-based audio models.

Instead of just having one token at each window, you will have multiple tokens, potentially set up in a hierarchy so you represent higher-level kind of meaning or acoustics, and then you have more granular, and so you have these multiple layers of representation.

Now, there are a few ways to set things up to have different tokens represent sound at a given window. The one we're going to focus on is the Sesame model, where there are 32 tokens at every window that will represent the sound.

There is a zeroth token that will be predicted by the main transformer here, but then there's a secondary transformer that's being used to decode out the other 31 tokens of meaning. So whereas here, we basically have Sesame decoding out with a main transformer autoregressively these first zeroth tokens.

The kind of stacked tokens are being decoded by a second transformer. So actually, Sesame has two models. It's the main 1 billion parameter model, and then a much smaller model that will decode out kind of these hierarchical tokens here.

So this is a quick overview of how the model is going to work. We will start off with a pre-trained model that is capable of taking in text and audio and outputting a stream of audio here, and we'll see how it performs just when we give it some text.

We'll see it generate, or we'll hear it generate some sound. But then we're going to take a dataset, and in my case, it's going to be of one of my YouTube videos with my voice, and we're going to see if we can make the model sound a little bit more like me by fine-tuning it on that data.

So with that, we will move over to the collab notebook. As I said, you can find it on Trellis Research if you look for the aiworldsfair-2025 repo. And if you open that up in Colab, you should be able to run it for free using a T5.

So when you open up the notebook, make sure that you are connected to a GPU. This should be available for free. If you're not connected to a GPU, you can say, "Change runtime type," and then you'll want to select the TPU here.

So I am connected, and there are two sections in this notebook. If you're following along, you can just open it up in your browser now. There are two sections we'll go through. The first is data generation, where we're going to pick a YouTube video.

We're going to use Whisper in the collab notebook to transcribe the YouTube video, and then we're going to convert it into a dataset. The dataset will have audio snippets of up to 30 seconds, and they will have text, which will be the transcription using Whisper.

Given that dataset of snippets, I think I will have maybe around, let's see, 41 rows of data, which is actually enough to have an effect on the quality. I have previously done trainings with maybe shorter snippets of only 10 seconds and 20 of those, and I don't think that's enough data.

So I would say, if you've got 30-second snippets, probably having roughly 50 of them is enough to start to have an effect. And given that dataset, we'll then fine-tune the Unsloth model, or sorry, we'll fine-tune the CSM model, the Sesame model, using Unsloth, and we will measure performance at the end and see how that fares.

So let's start off with the data generation, and you can pick your own YouTube video if you like. I recommend picking a video where you just have one speaker. You could pick a video with two speakers, but you need to do more data processing.

You need to do diarization to split that audio up into multiple speakers, which is not supported in this basic notebook here. So I recommend taking a video where you just have one speaker. This is a Trellis video here.

You can then pick the model size, the Whisper model size that you want to use for transcription. I recommend using Turbo, which is almost as good as large. I think it's better than definitely small, base, and tiny, but it's actually a lot faster than large.

So we'll go with Turbo. Run the installs. So you'll need YouTube DLP. Note that this works well in Colab. If you try and run this notebook outside of Colab, you can run into authentication issues where you'll get blocked from downloading from YouTube.

So I recommend running this in Colab. You also need to install Whisper, which is from OpenAI. And with all of that installed, you should be ready to then load the Whisper model here and run the transcription. And we're going to use the GPU, so we'll use CUDA here.

And it should then save your transcript to a file, to a JSON file, which will be saved here locally. So what you should find is a JSON file that's saved, and you should be ready to then use that file as a basis for the training.

Now, when you run the transcription, I'm not going to run it here because it's going to take a little bit of time. And when you have that JSON file, you can take a look at that file, which I'll show you now.

Okay, so I've opened up the JSON file, and it has a bunch of text and also details around the exact tokens. It also will have the segments of the transcription. And what I recommend doing is just take a quick read through it.

I know you won't be able to see because of the font size here, but take a look through for any words that you think might be misspelled. And what you can do is a quick find-replace on some of those variables.

So, for example, often I'll see the word Trellis with two Ls because that's how the correct English word is spelled, but the word Trellis for the name of Trellis Research is one L. And so I'll often just do a find-replace for Trellis and put in T-R-E-L-I-S, and then I'll replace that throughout.

Now, I've already done that, so it's completed. But if you want to make those fixes, what you can do is do the find-replace and then re-upload the file to Google Colab. Just re-upload it here so that you can read it in and use that corrected transcript then as the basis for your fine-tuning dataset.

So at this point, what we've done is we've grabbed the audio from YouTube, we have transcribed it, we've corrected it then manually, and we're ready now to split what's a long transcript consisting of short segments from Whisper. And what we want to do is combine some of those segments so that we have longer chunks up to about 30 seconds in length.

We're going to use a simple algorithm. We're simply going to stack them until they reach more than 30 seconds, and then we'll move to creating a new row of data. So what we ultimately want is rows of data that are max 30 seconds in length.

So that's what we're going to do here. I'm going to just log into Hugging Face so that I can push a dataset up to Hugging Face. Now, I've done this earlier, so I'm not going to log inright now.

I'm going to define the organization. So if you've created an account, you can create an organization minus Content Trellis. I'll create a repo name, and I'll create the maximum length I want for the rows of data. Then I will grab the source audio, which should be saved from running the YouTube extraction earlier, and I'll also specify the path to my JSON file, which could be an updated JSON file with those corrections.

Then I'll load that JSON file, and I'm now going to run through basically taking the segments here and adding them together until we get to the maximum duration. And when we're at the maximum duration, we'll call that a row of data, and we'll move then to the next dataset.

So with that done, as you can see here, I've generated 41 clips. I've pushed it up to YouTube, or sorry, you pushed it up to Hugging Face. If you don't want to push it up, just set this to false.

And yeah, my dataset should now be available. So if you're following along, it will take a bit of time. You might need to pause the video because you need to run the transcription. It should take a few minutes at most, even if you have like half an hour or an hour worth of video.

And then you should find your dataset up here. And you can see my snippets of text, and then you can see the sound. So I could, for example, play some of these sounds and test them out. Now, this is kind of a basic implementation, what I'm showing, but you can see already an opportunity for improvement.

Here,right now, I'm adding segments of text, but I'm not being careful as to whether the segments are ending. It would be better if I ended every row of data here on a full stop. That could be done by using a library such as NLTK to detect sentence boundaries, or you could even use just a regular expression.

So there is room for improvement of how these datasets are being compiled. It shouldn't make a huge difference, but generally, you want to have kind of paragraphs that are complete within a row of data here. Allright, so at this point, hopefully, you have your dataset pushed up, and you've noted what your org name and your dataset name is.

So we're ready to move on to fine-tuning. Now, the first thing for fine-tuning is to install Unsloth. So I've actually run this, and I have run it just earlier this morning, so we're going to be able to follow on with these cells.

We're installing Unsloth here, which includes transformers and a number of packages for loading and fine-tuning models. Next, with Unsloth installed, we're going to load the base model that we want to train, which is CSM-1B. This is the Sesame model.

We're going to load it with a maximum sequence length. This is the maximum text length, which is probably quite a bit longer than 30 seconds, which it should be. We will leave the data type as none. Yeah, the data type if I'm on a T4 will be float16, so it'll be float

torch.float16 on T4, and it'll be torch.bfloat16 on anything more recent than that, so on Hopper or Blackwell GPUs. Also Ampere as well. The auto model is CSM for conditional generation, so Sesame models are now supported by transformers, and that's what Unsloth is leveraging here.

You could quantize the base model, but it's not necessary because CSM-1B is just a few gigabytes in size, and the T4 is 15 gigabytes, so it will fit easily within this here. So the model is going to download, and it will then be loaded into the GPU.

When the shards are loading, you should see that the GPU should start to get full with the shards. Now, this here looks like my GPU usage is kind of high. It might suggest I've loaded the model twice, so I may have to restart my runtime and reload it again, but we'll see.

It's lowering down now, so maybe that's okay. And what you can do then is print out the model architecture just to take a look. And yeah, there's quite a lot in the CSM model. There's the backbone model. This is the main transformer model here.

And then there's the depth decoder. So this is where you're decoding out those 31 other tokens. And then there's also the model that allows you to convert from the waveform into tokens and tokens back to the waveform. So you've got the codec model, and there's also the decoder down here as well.

So this is the architecture. It's relevant to print the architecture because actually, we're not going to train all of these parameters. We're just going to train a subset. We're going to focus on the linear layers. And actually, we're not even going to train all of the parameters in linear layers.

We're going to train adapters that plug on. So that's going to save memory and also make the training faster. So you'll just take note of some of these modules here. These are the attention modules that we're going to train.

These are the linear layers. We're going to train these as well. Okay, so that's the model printed out. And we're ready now to apply those adapters I just mentioned. So apply little adapters to train and apply them to these modules here.

I'm going to come back though before I apply this because I'm going to skip Rodder because I want to see how the base model is performing without any fine-tuning. So we're going to skip the data preparation just for now.

We're going to skip the training, and we're going to run straight to inference just to hear what some of these sounds are like. So

yeah, let's start off and just pass in a piece of text here saying we just finished fine-tuning a text-to-speech model. We didn't finish because we're just starting, but let's run this here and see what it sounds like. There is an alternative way of running here.

Basically, here we're using a chat template to apply to text that we're passing in, whereas here we're preparing the tensors in a more raw format just by passing in the speaker ID and text here. So essentially, we're bypassing the chat template, whereas here we're using the chat template.

Just two different ways to call the model. So this is just a basic called zero-shot inference. You're passing in text, you get out audio. You're going to get a random speaker's voice here because the temperature is non-zero. So you could get female, you could get male, you could get deep, you could get high-pitched.

You could really get any speaker. And one way you can avoid that is by doing what's called voice cloning. So with voice cloning, which is different to fine-tuning, with voice cloning, we will pass in a sample, and we will then get the model to speak.

So basically, we pass in audio, and then we get the model to generate audio, and the model will tend to generate audio that sounds more like the audio we passed in. This is called voice cloning, and we're just going to use some of the dataset that we already created for this purpose.

And yeah, that dataset will get passed in, and then we will ask the model to speak this text here. And this should be quite a bit closer to my voice than the first example because we've passed in a sample, but it still won't be as close as we get after fine-tuning.

Now, to be able to run this, I do need to have loaded the dataset, so I need to load the raw dataset. So for that, I'm actually going to go up here to the fine-tuning. I've got Unsloth, data prep.

I'm not going to, yeah, and then I do need to run the loading here. So I will just run the data preparation script here so that I load in my raw dataset from my YouTube TTS. Just a note that when we load the dataset, we do need to have a dataset with an audio column and a text column.

Optionally, you can have a source column, and the source will refer to the speaker number, which would start at zero. So if you have two speakers, it could be zero, and then it could be one for a different speaker.

If you don't have a speaker column, what this code here is doing is simply assigning speaker zero. So we'll just, if there's no speaker found, it will use the default of source, which is zero. So it will apply a column effectively of source being equal to zero.

Okay, so I'm just loading the dataset here because we need to have some data if we're going to clone. And yeah, the message is showing here that the speaker is being set to zero for each of the rows, which is good because it's a video where I just have one speaker.

And now it's just checking the maximum audio length, which we're going to need when we set up the trainer later. So we configured a trainer to also have that maximum audio length. But while that's running, we can just go down so that we're ready to run the cloning script.

And this is just going to run inference with cloning. And I do have an issue here. I need to comment that out. And the cloning is now underway. So we're basically going to pass in that, well, a sample of my voice, and then we're going to try and generate text to say thisright here.

So in the meantime, I've already generated a sample without the cloning. So let's just play that.

We just finished fine-tuning a text-to-speech model.

Okay, so yeah, you can hear it's a woman's voice. And as I said, it can be unpredictable exactly what voice you get based on temperature. And here's another sample.

Sesame is a super cool TTS model which can be fine-tuned with Unsloth.

So yeah, you can see that there's a very wide variance in the type of speaker that you're going to get if you just do zero-shot inference. I have got a sound file now of the cloned audio.

Sesame is a super cool TTS.

And here's that audio.

Model which can be fine-tuned with.

And here's that audio.

Sesame is a super cool TTS model which can be fine-tuned with Unsloth.

So it's not too bad. My voice has quite a bit of drag on it.

Sesame is a super cool TTS model which can be fine-tuned with Unsloth. Sesame is a super.

So it's definitely closer to my voice, but not as good as we can get with fine-tuning. So what we're going to do now is go back up to the start, and we'll continue on after having loaded the model to apply LoRA adapters and then run through the training, and we'll see if the fine-tuned model performs better than just using cloning on its own.

So I'm going to scroll back up here to the point where we loaded the model. The model is still loaded. I'll just rerun this to kind of make it more compact. And now we're going to apply those adapters.

So we're going to target the linear layers, the QKVO, and then also the MLP linear layers here, get up and down. We'll use a LoRA alpha of 16. It's a pretty small model. I usually recommend 16 if it's about 1 billion in size.

And we will use rescale LoRA. This means that the learning rate for the LoRA adapters will be scaled according to the size of the adapters, because if you have larger adapters, you need to train them more slowly and vice versa.

So we'll go ahead with this. We'll use a rank of 32. That means the rank, kind of like the width or the height of the adapter matrices. You can make that larger if you want more granularity, but 32 should be fine.

And with that, we will apply the adapters, and we should get a printout of the number of trainable parameters. So yeah, I think we've already applied everything there. It's not actually showing us the trainable parameters, but we could probably do that by just running print trainable params.

And yeah, so print trainable parameters. You can see that we've got about just under 2% of the parameters are trainable. Technically, you could train more parameters, say by training the embeddings. If you want to do that, you would add in here modules to save.

And the ones that you typically would want to save, or that you would typically want to train, are LM head and LM embed, or embed tokens. But because we're not changing the embeddings, we're not changing the tokens here, there's not really any reason to train them.

So just applying adapters to the linear layers should be plenty. Okay, so we have the adapters applied, and we're ready to move to training. We've already loaded the dataset, the raw dataset. We've set the source to zero because we've just got one speaker.

There's a little more formatting we need to do, though. And for that formatting, basically preparing the data for the trainer, we do need to know the maximum audio length. We've already got that printed here. It's about 700,000 audio steps, audio tokens in length.

Sorry, not tokens, but it's the length of the audio. Then we need to measure the length of the text. So I'll just run that cell here, and the max is 587. And we're going to pass that max text length and the max audio length into the trainer down here.

So you can see when we're preparing the inputs, we need to prepare text keyword arguments, and we need to prepare audio keyword arguments. We need to set a sampling rate, and we do need to set the text length and the audio lengthright here.

So I've got this set, and Unsloth is doing the work to prepare the input IDs, the attention mass, the labels, the input values, and the cutoffs. So all this is set up nicely for us per the original Unsloth notebook.

And we just need to apply this mapping now to our pre-processed examples to get a processed dataset. So I'll go ahead and do that now. And then I'll be able to print out the dataset, which will show the number of rows, the input IDs, and these other columns here.

Next, we're ready to train the model. So we can pass that processed dataset into the trainer. You can see that's being passed in here. The model is being passed in here. We've already applied the adapters. We will use a virtual batch size of eight.

You probably can go a bit bigger within the T4. I usually would like to have this of 32 in size, so the product being 32, but we don't have that much data, so we won't go with that large of a virtual size.

We'll go with eight. With 41 rows, there'll be five different steps for every epoch of training. We'll just train for one epoch. The warm-up steps of three, this means that we're going to increase the learning rate slowly. To be honest, three is a bit much if there's only five in terms of total steps, so you could just reduce that to one maybe.

The learning rate two e minus four is probably good for a 1 billion parameter model. This will automatically decide whether to use float 16 or brain float 16 depending on the TPU that's being used. We'll use AdamW 8-bit optimizer to reduce memory requirements a little bit.

Weight decay prevents overfitting, probably not essential. We will train at a constant learning rate, and we will save our outputs here to the output directory. So with that, I think we can set up training, check our current memory stats.

So we've got almost 15 gigabytes in the GPU, and nearly seven of them are used up. And we can start our training, and it's going to take about 10 minutes if you run the training. And what you'll find is during that training, the loss will fall from roughly 6.34 down to somewhere around 3.7.

It looks like the loss is still falling, so probably there's potential for improvement here in terms of the quality of the outputs. Maybe it'll help a little that I reduced the number of warm-up steps because that means we'll have a bit higher of a learning rate.

So I'd expect if I run this now, I probably should do a little bit better. Let's get the training going. And when the training is done, we'll print out the final training stats. You can see with that batch size that I specified, the virtual size of eight or the actual batch of two, there is only half the memory being used.

So you probably could increase up to a batch size of four without a problem. So I would run this. And then what we'll do is run the inference again, but this time on the fine-tuned model and see how the performance compares.

And what we'll do, what will happen automatically is we will get saving

of some of these sound files. So we'll be able to just play back the zero-shot example here. We've got the zero-shot audio and the cloned audio on the base model, but now we're going to generate voice samples using the fine-tuned model.

So I'll just run this here, run all of these cells so that when the training is done, and then run the cloning test for when the training is done. Training should be underway here now. If I scroll up, you can see we're at two out of 60 steps.

Note that there's no eval dataset. Ideally, you would split your training set into train and eval, and you would inspect the eval loss and make sure that that's falling. If it stops falling, that means you probably need to stop training.

Either you've gone too far in epochs, or maybe your learning rate is too high. You also typically want to monitor the grad norm. Make sure it's around one or less than one. You would do that typically by adding in here a TensorBoard.

So you would report to TensorBoard. You also need to set a log directory. So I think it's logging deer that you need to set here. Deer. Typically, you would set it to something like logs. But I'm going to leave this because we're just looking at a basic setup for now.

But if you do that, you can pip install TensorBoard, and then you can look up how to monitor the losses. You can see that in the longer video I already published. I'll probably publish a more detailed video after this too on the Trellis Research YouTube channel.

So training is going on. It looks like our loss is falling, which is good. And while that's happening, I'll just explain once the training and the inference is done, you have the option to save the model and also push it to HUB.

If you just run this cell here, it's going to save a copy of the model locally. So it'll be in your temporary storage on Colab. If you want to push it to HUB, you need to set one of these to true.

Both of them to true because you need the model and processor. This will just push the LoRA adapters, which is lightweight. It'll be a smaller repo. But if you want to push the full model, you need to merge it first.

And that's where this comes into play here. You can save the model in merged form. Well, this is just saving again as LoRA adapters. But if you want to merge it to 16-bit format, you can set these two to true, and you can save the model and also push it to HUB.

And I believe these will actually save both the model and the processor as well. So if you want to push to HUB, yeah, you just set this one here true. It will merge the model and then push it up to HUB.

By the way, if you do want to reload a model that you have previously created, you would do thatright back up here where we loaded the model. Instead of specifying the base model name, you would specify the fine-tuned model name, which in my case is going to be Trellis My YouTube TTS.

Okay, so training is done. We've gotten down to 3.72 as loss and used just under half of the memory available. Next, we'll run inference using this fine-tuned model. So actually, this is run. We should have a sample here.

We should have a second sample here, which we listen to now. And then we have a cloned example, which we expect should be the best. So let's give a listen to the fine-tuned model here. We just finished fine-tuning.

Let's have a listen.

We just finished fine-tuning a text-to-speech model. It's pretty good.

So it sounds a little bit like me, but yeah, definitely it's a male voice. So that randomness of it being any kind of voice has been removed. And it does sound a little bit Irish. You can see the pacing is not quiteright.

That perhaps would be improved by better filtering the original dataset so that I don't have segments from my YouTube video where there's just a lot of pause. Now, there isn't a lot of pause anyway in my YouTube videos typically because I cut those out.

But you could better do some filtering on the data if you want to improve that. Let's see a second sample though, because there is some randomness.

Sesame is a super cool TTS model which can be fine-tuned on slot.

So it's a little bit of an Irish accent. Probably indicates more data is needed to improve the quality here. The T in tuned is still a little bit American, although it's kind of slightly Irish. So some room for improvement here.

But what we see now is probably the best chance we have, which is using the cloning plus the fine-tuning. So I'll play this sample now.

Sesame is a super cool TTS model which can be fine-tuned with Unsloth.

Okay, so that was great. It even has the Irish tuned. It even kind of makes this little bit of an error here, which actually sounds natural. Let's give it one more listen.

Sesame is a super cool TTS model which can be fine-tuned with Unsloth.

Yeah. Now maybe the super cool, that's not something I would say. So maybe it hasn't got my intonation correct. But broadly speaking, this sounds pretty great. So at this point, we've pretty much finished the workshop. You should have a sound, ideally with the cloning, that sounds something more like what you do or what the person in the video you chose sounds like.

If you wish, you could create more data, aim for maybe 500 rows of 30 seconds, and I think you would see better performance, particularly without the voice cloning. But you can see here that if you combine fine-tuning with cloning, you're able to get pretty good performance even with a relatively small amount of data.

Just say, I think a video of 30 minutes was enough in this case here. So that rounds up the workshop. As I said at the start, you can get all of the resources on GitHub. That's Trellis Research forward slash AI Worlds Fair 2025.

I will probably make, I will make future videos on voice that you can check out on the Trellis YouTube channel. Probably some more detailed videos covering in depth, a little more depth, the data preparation and some of the hyperparameters around the fine-tuning itself.

In the meantime, let me know if you have any questions by dropping some of your questions or comments below in the YouTube comments. Cheers, folks.