Training an LLM from Scratch, Locally — Angelos Perivolaropoulos, ElevenLabs

Thank you very much for joining for this workshop. This is going to be a hands-on workshop, so hopefully you can get your hands dirty doing some very interesting, uh, very interesting project. To start with, a little bit about myself: my name is Angelos, I lead the speech-to-text team at ElevenLabs.

I'm a research engineer, so I spend most of my time training new models, working on inference, and also I'm on the product side, so I'm responsible for, unfortunately, talking to clients sometimes, which I don't fully enjoy. I'm the kind of person that really likes to go straight into research and do training models and make things that are set of the art and very powerful.

Currently I'm working on training real-time models for transcription, specifically for agents. And if you have, if you don't know already, we have this Scribe V2 model that my team trained. It's currently the best transcription model in the market in terms of, like, popular public benchmarks.

So if you ever need any transcription use cases, feel free to use it; I think it's quite good. Now, as part of the workshop itself, today we're going to be training an LLM from scratch. So no pre-trained weights, no, nothing that you can just grab online from, like, a transformer's library.

We're going to work purely on Torch and some, like, very basic libraries. We could go one level below and not use Torch, but I don't want to, to torture you that much. So I think Torch is, like, a good level.

And this will be, like, a good indication of how, like, actual, like, research engineers, like in big labs, like, design their models. And whatever is further than that is more, like, optimizations and making the scale be better and bigger and improving for specific use cases.

So this, what you're going to be doing today is, like, pretty much like 80% of the way there to, to create, like, a model from scratch. If you can go on this QR code, you're going to find this GitHub repo.

I'm going to switch to that now. We'll leave it for a little bit longer. There's, you have two options. One is to either train the model locally on your laptop. If you have, like, 16 gigabytes of memory, you should be able to do it.

It's a small, very tiny model that you can train fast. If not, and I'm guessing because not many people have outlets, maybe Google Colab would be another option. Google Colab gives you free GPUs that you can use for, for training such a small model.

So either it will be your choice. While

I'm going to leave this on for a little bit longer,

and then we can go in the actual repo itself.

So to give you some idea of the inspiration of this, of this project, my first exposure to transformers and LLMs in general was through this video from, from Andre Karpathy, who was one of the co-founders of OpenAI, which was called NanoGPT.

And for me that was, like, a very inspirational project. It's essentially like what inspires, like, this, this workshop as well. It's a bit lower level than what we're going to be doing now, so it goes a bit deeper, like, into how you're using NumPy for, like, a lot of calculations.

But I think it's a good blueprint that we can follow for, for us to be able to create a model from scratch.

Let me put this to the screen.

But yeah, if you, if you don't, if you, if you don't know this project, I think it's a great introduction. If you do know this project and you've trained LLMs like that before, this might be, like, sim- a bit simple for you.

But I think that there are ways we can expand this workshop towards doing something that's a little bit more of a competition, which I would see, I would like to see what you can come up with. For this specific, for this specific workshop, we're going to work with a very small model that's based on the GPT-2 architecture.

It's a bit of an older architecture, but the, the, the base and the, the fundamental parts of it are basically haven't changed too much. And we're going to go over, over those in, in a bit. The, the four building blocks you need to, to train a model is, first one is going to be the tokenizer.

And depending on what your use case is, you'd want to use a different tokenizer for that specific use case. If you want to train, for example, a very big model that can, that wants, you want to generate text for, like, multiple languages, you'll need a huge tokenizer, which means you're going to need a huge amount of data as well to train it.

But for a, for a smaller model, a smaller tokenizer with, with, with smaller embeddings is what would work best and train the fastest when your data, when your data is limited, which is what we areright now. Next would be a model architecture.

To be honest, like, most models, like, have at least at, at the, at the period that we're going to be working in, were very similar. Like, in, in, there were just decoder-only, causal decoder-only models that had a very similar way of, of using causal self-attention and the same, like, MLP layers and the same, like, layer norms and all kinds of stuff, which we'll get, we'll get to it.

So if you know how to do this, this, like, small models, it's very easy to go and do the same process for, like, bigger, like, newer models. But of course the newer models are going to be way more specialized for longer contexts and essentially, essentially being able to scale this train.

They're architected in a way to be able to scale training to as many tokens as possible, which we won't need in this case. And lastly, it's a, it's a training loop, which this is generally the most important part when you're training a new model.

Like, if you check the difference between, like, GPT-4, GPT-4.0 and GPT-5, or even, even if you go before that, what you'd see mostly is that usually the pre-training is very similar. It's the fine-tuning and post-training and essentially what you use with a, with a same base model or, like, very similar base model.

And the way you train it actually makes the big difference in performances. And now we see, for example, Gemini 3 comes out and then has this, this, this many, this good benchmarks. And then 3.1 comes out that has, like, doubled the performance in some benchmarks, which is crazy.

Like, obviously it's very sim- the models are very similar, but actually during training, they train, they train the new model in smarter ways to improve performance, like, very substantially. Now, as for the, and, and lastly, of course, the inference part, like, that's going to be very easy for us, like, because we're just, it's going to be a small model that can run everywhere.

So, like, that's going to be a very simple part of the, the building blocks. Then the prerequisites: any laptop would do, that has at least 16 gigabytes of RAM. You can work with, with, with smaller laptops as well.

It'll just be a bit slower. Bigger laptops, you can crank up the batch sizes higher, so it will train faster. Python 3.12, and I expect that, I expect that most of you have, like, some idea of how to write Python.

Like, if you don't, I think it's still quite, you can just copy-paste things until something works or, like, ask for some help. This training can uses Apple Silicon, so the, the NPS architecture, CUDA or CPU, so it can basically support everything.

And as for getting started, this just to make things easier, like, I'm using UV for this project. So if you have your laptop out, you can install UV on your machine if you haven't. It's, it's quite straightforward. And the reason UV is quite simple, you can just run UV sync and it creates the virtual LLM for you and makes your life easier.

And we're going to write the code in a ScratchPad. Or if you're using Google Colab, which depends if your internet is overly good, maybe that's a better idea. If you just go to,

if you just go to and create a new, let me actually create one, on a separate one so we can, we can follow along.

Yeah. So if you open Google Colab and you create a new Colab project, you can run this command which just installs the, what we need, which is just Torch, NumPy, TQDM, and TikToken. TikToken is mostly for, for testing things out.

And yeah, we're going to move on to the first part of this,

of this workshop, which is going to be the tokenizer. As I mentioned, like, this is generally the first thing you think about when you create a new, a new, any new transform model is what tokenizer you're using. I come from the voice world, so this is where that, that's the, one of the most important things.

Like, we're thinking, okay, we need to train this new TTS model and we're going to spend maybe six months thinking about the tokenizer and then we're going to spend two months on the architecture. So that's, that's like, generally one of the most important things of deciding how to create a new transformer.

For those who don't know what a tokenizer is, like, LLMs don't see text. They, they work with embeddings or, like, vectors. So we need, like, some kind of representation of those vectors for the model to be able to process.

What we're going to be using here is a character-level alignment, a character-level tokenization, sorry. Just because it's, it has the lowest number of possible tokens. In that, in our case, for, on our data set, it's going to be only 65 embeddings essentially, because it's 65 different characters that will appear in the, in our training data.

And the way it works, like, this is going to be using the Shakespeare data set, just a few works of Shakespeare. It's part of the, it's part of the repo itself, but you can, if you're using,

if you're using Colab, there's going to be a link on how to download it on Colab a little bit later. And essentially we're going to use this, this story library, which basically just converts,

will be converting strings to integers. And that, these integers will then be turned to embeddings through the embedding layer when we train the, the LLM. And it's a very, very simple, straightforward tokenizer. It uses this enumerate function from, from, from Python and just then selects that specific item that has been selected with, with this dictionary.

So yeah, as I said before, like, we use, we use character-level because it's much easier to train. Because we have 65, 65 tokens only, it means that we, the bigram combinations would be like 65 times 65, so 4,225 possible bigrams.

Bigrams are essentially like when you have one token and then you predict the next token after it. And this concept of, of, of bigram is very, is a very important concept when, when you're training transformers, because you want your model to see as many possible bigrams as possible.

So if you have a model with, let's say, 200,000 tokens, you need 200,000 tokens squared at least data to be able to, to train, like, from scratch in a, in a, in a very good way. So the, or at least, like, this is the magnitude that you're looking for.

In our case, 4,000 bigrams is, like, very doable. Like, this, this data set should include all bigrams, like, multiple times, very likely.

If we did try to train using a full tokenizer, this will never converge. We can just be training it for hours and hours and that our model will never be able to, to get good results. Now, the problem with, with character-level tokenizers is that they don't really scale very well.

Because the models, they, they, the way they work is that they need to understand correlation between different tokens. So you can very easily have a correlation saying the sky is blue. Like, these tokens combined together, like, make a lot of sense.

But sky and then is and then blue, that it's a bit harder for them, for the model to be able to make, like, good, to attend to these tokens in, in, in a good way. So this would, this would work quite well for our example.

But if you want, like, a very, very good model, one, it will be expensive to train because, of course, like, you have to create, like, a ton of tokens when during inference and during training. But also it will just never converge to something good because the model, the, the tokens combined don't make too much sense.

So that's our trade-off, but this is a trade-off that we're willing to take because, of course, we're running a, a small model. But in the future, if you want to expand this to something better, if you want to train, like, a proper LLM and you're happy to train for, like, a week using bigger GPUs, using a proper tokenizer, that's a, using, like, byte pair encoding is, like, the most common way of, like, doing tokenizers these days, which essentially takes all, you can, the way you would train a BPE tokenizer is that you take all your training data and you'd find all common patterns and combine those common, those common, like, character patterns into specific tokens that then you can reuse and the model can understand the relationships of.

So the, the way that this connects to the model itself is we have this embedding table, which is size, vocab size, and then it's what I said before, it takes

a vector of integers and then it returns, sorry, a list of integers and then it returns a list of vectors, which is going to be our embeddings.

And like, like I said before, like, that's, if we did use such a big, such a big tokenizer, that would also be, like, more than twice the size of our models. Because if you just multiply this, our embedding size is 384 for this model that we're going to be training.

So that's already, like, 25,000 parameters. And in that case, like, if we used GPT-2's vocab, which is 50,000, that would be 19 million parameters, which is, like, more than three times the model, which wouldn't make sense in our case.

So now moving on to the next part of this workshop, which is going to be the transformer itself. And as, as I mentioned before, like, the transformers, they have been kind of commoditized now. The, the way transformers work, like, there are different labs that find different optimizations, but the optimizations, like, in principle is more about we have this, like, base, like, idea that works really well.

How can we make it train faster and, like, have bigger context and, and people find ways that are more optimizations than necessarily, than necessarily reinventing the wheel. For this at least, there are some, of course, hybrid approaches that are more complicated.

So I'm not going to go too deep on how transformers work. And also to prove a point that you don't necessarily need to know in a very deep level how transformers work to, to be able to train something like this.

When I first, like, did this project myself, I didn't, I had no clue how transformers worked and I still didn't that much at the end of the pro, the works, the previous project I worked on. But it, once the more you work on it and the more you, like, you have motivation to continue pushing through, you can understand all the different concepts together and how they, they, they stack together and the reasons why they ended up the way they are.

To go back to the big picture, transformers are, are, are using, like, these four different building blocks. One is multi-head self-attention. Attention is, like, what the difference between that makes transformers different than other neural networks is that they can actually attend to previous tokens and understand the relationship between tokens that I was, I was mentioning before.

And that's where attention comes in. And of course, the bigger your attention is, the more the model understands those relationships. And going back to what I said, that's what, like, big labs like Gemini, they're trying to do one million contexts and they're finding ways.

Because if you just try to use a one million context for, for a model like this, it will just break very easily. Like, the, the math wouldn't work. So then that's when, like, the engineers from, the researchers from Gemini found ways to, to make it work.

And that's what that, that makes a difference. But again, fundamentally, it's like the same, the same architecture. Next one is the, is the MLP or like the feedforward network, which essentially takes these different

relationships between those tokens and while taking them, while the, the, the relationship themselves are arranged all over the relationship between the tokens themselves, it, it combines them together to, to be able to generate the logits that will then be.

So it's essentially kind of a, it takes the context and then organizes it in ways that the model can then generate the logits and then generate the tokens. I hope that was a decent explanation. And then you have the residual connections, which are basically there for the model to, to not have to reinvent itself after every layer.

Residual basically means that every layer that, that passes through the, because as, as, as we'll talk about later, transformer is, is, is, is built on multiple different layers that we pass through the activations one after the other. And residuals are there so that each activation doesn't completely restart things from scratch.

It just changes them slightly. So it just takes the previous input and makes a small, it adds a small difference. The next layer does the same and next layer does the same and this will continue on. And this way the model doesn't, each layer doesn't make a huge change on the inputs themselves and the model can be more stable during training.

And lastly, the layer normalization has a very similar, has a very similar role in

being able to, the, the layer normalization is able, is, is allowing you to scale down those activations in ways that, that allows those activations to not be, to not be exploding into very big values. So if one layer, like, multiplies your activation by 10x, let's say, the layer norm is going to push it back to, like, normal values.

So it doesn't go 10x, 10x, 10x, and then you kind of, like, millions of values that start at, like, 0.5 and then end at 10 million. That's what the layer norm is for. But again, these are just the building blocks.

You don't necessarily need to know, like, why they're there and what the, what the purpose of them is. Like, you, you learn this as you start working on these models more and understand how those, why, why these decisions were made.

Because all of them, as, like, as I'm talking to you, of course, were done by, we have a certain idea, all this didn't work. So let's add this to make it work. Sorry, go ahead.

Are you showing some slides that we aren't seeing?

Oh, oops. Yeah, I was showing. No worries. Yeah, apologies for that.

Yes. So I was, I was going through the tokenization. That was the previous, what, what my, my previous point. And that's, if you, if you can go back and you'll have to go back through and forth through the slides when you're working on your, the model yourself, because you can, you have to, like, copy-paste the parts or figure them out yourself.

But it's, it was explaining why we're going with character-level tokenization and, and what other options we have. Therefore, transformer, I was explaining the different, in the big picture, the different blocks of, of architecture. And then now going to

how does this look like in code? Because all the things I described to you actually are very co- there's very little code you need to, to actually to implement them. First, we start with the basis of, sorry, go ahead.

When it comes to code, I said Python because at least in English, you know, you have words, vocabulary is kind of fixed. But in Python, you have your own variables and so on. So how does tokenization work in that setting?

What would be a tool? Sorry.

So in a programming language, of course, you have programming syntax, keywords, but then you have also normal variables, function names, and so on. So when it comes to tokenization, how does that work?

So yeah, as I mentioned to, to before, we're going to be using, like, a character-level tokenizer because it's going to be easier for this project. But most big labs, they don't use character-level tokenization. They, they use what I was mentioning earlier, the BPE tokenizer or, like, byte essentially they, they, what they do is that they, you, you look at your training data.

Let's say you have, like, this many trillions of tokens and your training data is going to be including, like, code, as you mentioned yourself. And the way it works, it will just look at common patterns. So if you have, like, a lot of training data that is, that is code itself, it will, it will see and then it will realize, okay, four loops seem to be, like, a good candidate for, for that to be a token.

So the four is going to be for sure a token. And then you might have some, like, enumerate. Enumerate, you, you see it quite commonly in the, in the training data. So that will be another token. So it will look at all these different tokens and then create this tokenizer based on the common relationship between them.

Of course, like, maybe some languages that are, like, maybe Pascal that might not be very common in the language. So this, the keywords of Pascal might not be, like, very common. Although I do think that there probably is, like, good representation in the tokenizers too.

But some other of those, like, crazy, like, white space only, like, languages that there's, these ones probably are not going to work very well.

My question was more on the, so one thing is the keywords of the programming language, but then you have your own variable names, which are not common among different programs.

I see that was your argument.

So making them as tokens, I'm not sure how does that help.

So again, like, there's no, there's no specific way, there's no, like, human in the loop in this process. It depends on your training data that you use to train this tokenizer. If your code, like, there's, like, foo and bar, like, that's, like, common variables, these probably are going to be tokens.

Like, if your, your, if your variable names are, like, very strange, like random characters, then yeah, probably that's not going to be in the tokenizer. And when this happens, the tokenizer will fall back to character-level or, like, bytecode-level tokenization.

So if your, if your token is, like, just random, like gibbles, it will just be each character or, like, some, maybe some combinations may be together as different tokens, but most of them are going to be separate tokens and that's going to be a bit of a pain too for doing inference and, yeah, maybe not, not the best if you want efficient inference.

Thank you.

But yeah, going back to, to the transformer side of things, as I said, the, the models are generally quite, quite similar. And the code for actually implementing these transformers is also quite simple and, like, easy to write. It's like maybe, maybe 100 lines, actually usually more than that, less than that.

And the first thing we have to, to, to accept is what the parameters of this transformer should look like. As I mentioned before, vocab size is the size of your tokenizer. In our case, it would be 65. Block size is essentially the sequence length or, like, the, the, the context window.

In our case, that would be 256, which is very, very small for these models, but because we're training a model locally, like, that's kind of, kind of what we have to do. Bigger labs would use, like, one million context size for stuff like that.

But generally, like, 16,000 is, like, a common middle ground. Then we have the layers. How many, as I mentioned before, like, transformers have multiple layers and then you run activations through each one of those layers. We're going to go with a modest number of six.

And then the attention heads, like, the, the way that attention works is that you might, you're going to have different heads for attention that are going to be attending for different things. Like, one of the attention heads might be looking at punctuation.

Maybe another attention head might be looking at the grammar. So all these different attention heads are attending to, like, a specific feature of the text or if you're using audio, a specific feature of the audio, et cetera. And lastly, that's the embedding dimension.

That's how big the actual vectors of those tokens that you create are. In our case, we're going to start with 384. That's, like, the standard for GPT-2. But a bigger value, of course, will have more information per token.

A smaller value would have less, but that's, like, a pretty standard value you can start with.

As for the code itself, feel free to copy-paste part of this if you want to spend more time understanding it. I don't want to go too deep into, like, all the little details of how things work, but essentially, like, you generally have, like, this overarching module.

In our case, we're going to call that GPT. And that overarching top-level module is going to include all the other modules that I described earlier. In this case, this will take the config.

In this case, this, this will take the config that we have, we had above. And then it will create this using, we're using Torch module dict here just because it's, like, the easiest way to implement this. But this is all just math.

Like, you can, everything you see here would just be, like, calculations, like, through matrix multiplications that Torch allows us to abstract and make things easier. And pretty much all the things you see here are, like, you see here are, like, neural networks, like, smaller or bigger that are combined together.

Can I ask a question with regards to this very first?

Yes.

If you could scroll up, so you have

the length of the sequence,right?

Yes.

Let's say we have a use case where we have, we don't have a large sequence,right? So we have however many 256 is fine. But this is very much related to the number of parameters you have in your model.

Not necessarily. Sorry, maybe you can repeat the question with a microphone.

So the, the, the size of the input sequence is related to the number of parameters in your model?

Not, not necessarily. You can have, like, large sequences with, like, small models or, like, small sequences with, with, with bigger models. The, the main thing that this shows is essentially there are two ways you can train a model.

You can train a model, like, that looks at, that's a full attention model, let's say, that looks at the full attention. So it looks at the whole past, essentially, which is what we're building here today. Or you can have a windowed model.

That's where most of the, like, bigger models that are outright now that only look up to, like, this much in the past.

And this parameter is the size of that window that, okay.

Yes. It's what we're going to be using to train. So the model will not have seen anything greater than 256, 56 tokens in a sequence. If you try to go above that, it will kind of, it will spazz out.

It will have issues.

But so let's say we have, like, computational capacity to increase the number of parameters to make the, help the model reason better,right? Which number would you crank up here to say it? Like, would you increase the number of layers, attention heads, or?

I would increase all of the, all, all of what you see here except of maybe the, the embedding size and many dimension. I don't remember exactly what other models do, the exact numbers. I think that seems reasonable to me.

But everything else, like, works well for, like, a small token, a small, a small model. 256 block size is tiny. Like, that's, if you try to run this on GPT, like, it would just forget things that you wrote, like, 10 sentences before.

But the, the bigger you make this, the harder it is to do the training. And that's what I was talking about, scaling laws is that you can't just say, go here and say, okay, actually, I don't want 256.

I want 2 million context length. You can't train this like that, at least with this current architecture. So that's what, like, like, researchers after, like, GPT-3.5 came out, they were like, okay, people are complaining that we only have 16K context size.

We want 1 million. How do we do it? You can't just change this number. You have to change the architecture to allow training to be able, otherwise you're just going to go, like, out of memory, like, instantly. So that's, like, one of the things that researchers now try to, to, to figure out how we're going to be able to increase these numbers here while keeping training stable.

How do you increase these numbers here while keeping training stable and still able to, to have enough compute to be able to do this? I hope this answers your question.

Awesome. Thank you.

So yeah, the, the first thing we'd have to do is, as I mentioned before, like, create the, the top-level representation of our model that you can see here as the GPT class. I'm not going to go again to, to, too much into the details, but, like, you want to, one would be the embed, how the, your model, like, understands embeddings and also how does it understand embedding position.

I don't want to go into details, but essentially you need, the tokens need to be able to understand both. Then you'd have essentially here all the layers that I was mentioning earlier, and each of them is configured as, as a block, which we're going to go at the definition of block late, that's a little bit later.

But essentially a block is like a layer of the transformer that includes its own attention and its own, like, layer norms, and it continues the, it feeds into each other the, the one you do the forward pass. And lastly, you have the LM head.

And essentially the LM head takes all the outputs of the above and connects them together into what we call, like, logits, which is the distribution of what the, the next token should be that should be generated. Because as I mentioned earlier, like, maybe I did, maybe I didn't, but the way transformers work is that they predict next token.

So you take the previous, the previous context, you predict the next token, and you have to sample this token on, on, on this distribution. And the LM head creates this distribution that we, we sample from.

Then the, the, the next important bit that, like, is again, like, quite, quite straightforward, like, it's quite similar in most of these models, is the forward pass, which essentially allows the model to, to go, to, to take the input, which is going to be the tokens that we mentioned, and push them through all these different components that, that I mentioned earlier, and just does some,

does some, some preprocessing, turns tokens into embeddings and positional embeddings,

adds them together, then goes through all the blocks of a transformer, all the different layers of a transformer, and just runs the, the forward pass of that transformer, and then does the linear norms and then passes through the LM head to get a distribution of those logits that we mentioned earlier.

And this is if you're doing training. This would also do cross-entropy loss in this case. And then at the end, you're going to get logits and loss as the outputs of your, of, of your transformer model. And I do have some diagrams here that you can see, like, they're quite, they're quite basic, but essentially they show the flow of, take token IDs, you pass them, you make them into embeddings, token embeddings and positional embeddings, you add them together, you pass them to the transformer, then through the layer norm that just makes the outputs in a space that the LM head can comprehend easier.

And then you return this distribution of, like, what the next token should be that has, that has this, this size. And as I mentioned before, like, 65 is the size of, like, your vocab, our, our, our token size.

And now self-attention is, like, a little bit more complicated. I don't want to, again, to go too deep of how, how that works, but essentially attention is there to, to understand the relationships between the tokens.

Essentially what, what is the important, like, if I say the sky is blue, blue and sky have a very big correlation. So that's what attention does. Based on how you've trained your weights, you will understand what tokens should be attending to each other and put higher emphasis on those specific relationships.

Now, again, going back to what I was saying before about the, what, that's why the tokenizer matters a lot, because sky and blue are very easy, very easy for the model to make this relationship. While in our case, you have to, like, combine different groups of tokens together, different characters together, that's quite a bit harder.

But essentially that's, that's what attention does. It's, it's the, what, what each token should be attending to in the past and what has most importance for it.

And it has also a forward pass as all of these different blocks do. And in your implementation, feel free to just copy-paste this, these blocks. Maybe you can spend some more time on understanding the diagrams and, and how things work.

But as you can see, even something as complicated as attention has a very small, it's just a few lines of code to, to implement.

And yeah, lastly, the, the MLP block is again, as I mentioned earlier, also, like, again, I already talked about the multi-head attention, why attention has multiple heads, because each head attends to different parts of what makes language what it is.

And then the MLP block takes the outputs of attention and, and makes sense of all those different relationships into something that the model can then understand a bit better, which again is just essentially just a, a neural network that just combines things into something that's a bit more understandable for the LM head to then be able to, to generate distribution for logits.

Sorry.

Yes.

Can I interrupt? I think I'm sure there are questions for.

Yeah, yeah. No, you can interrupt.

Would have the same MLP in all individual transformer blocks? Like, is it the same architecture and is it the same weights between the different transformer blocks?

No, each block has its own weight. Each block, the, the way each layer essentially you'd have, each layer that we would call blocks, they usually have, like, a different section for, a different prefix for the weights of, like, MLP would be a different, like, maybe usually they're called FFN blocks instead of MLP, but it would still be the same.

So each block has its own weights. And the ML, each, each layer.

Is it different FFNs?

Yes. Each block has its own MLP. So it's, you have, each block has its own, as you can see here, its own attention, its own linear, linear norms, and its own MLP. That's what basically makes a layer of a, of a transformer.

And that's what I, what my point was going to be later. Everything comes together into this block that we can see here, which is basically a layer of a transformer that has some normalization, running the attention to get the relationship between the models, and then having the MLP that combines the relationship of, sorry, the relationship between tokens, and then the MLP that takes those relationships and turns them into, like, a representation that's easy for the model to, to create the logits.

And this, oops.

This is what I was showing. Sorry. I have two different screens. I shouldn't be doing that.

But yes, this is, this is the transformer block. This is the, the building block essentially of, of a transformer. It has a layer norm, the attention, and another layer norm. Like, not all of the models have this, this kind of configuration, but this specific one has this.

And then the MLP that takes everything and creates it into a representation that makes sense for us to be able to make the generations. And you can see here, like, a bit more of a, like, a simple diagram of how this, this works.

Yes.

Did the original NanoGPT have the residual connection as well? Do you remember?

So the residual should be there.

No, no. I mean, on the, on Andre Karpathy's NanoGPT, did it have the?

I don't, I don't remember. It's been a while since I—

Okay.

Like, I had the idea, but from there, but I built everything from scratch, so I don't, I don't really remember if it did or not. But yeah, residual, as you can, the, what, the idea of residual would be, as you can see here, instead of doing X equals the attention, you do X equals X plus attention.

So you, you get the difference. The, the, the values of the activations don't change that much. That's the idea of, like, doing residuals.

And yeah, so we have this, like, if you wanted to implement this yourself, you can copy-paste all these different classes into one file that you can call model.py. And this is essentially all the maths of how the transformer works.

And as I mentioned before, then we have to decide how big the transformer is. Like, the parameter count that you can see here is just the 10 million parameter based on what I was showing above. You can just find yourself by just summing all the, all the different parameters of all the different parts of the different blocks that we had above.

The token embeddings is 65 times 384. We said this is the vector for each embedding, so it's 25K. The positional embeddings is 256. 256, remember, is the sequence length, the max sequence length of the model. So that's another 98K parameters.

And then where the, the biggest part of the logic is, is in the actual transformer blocks themselves where most of the parameters live. So you have, you have 4X. So the, the 4 there comes of the way attention works where you have key-value, key, query key and value pair.

So it's, it's part of the attention has like four different parameters for, for, for 384, which is the amount of, of tokens times the relationships between those tokens. So 284 per vector times 384. So that's 590 that we have for the, for the attention per layer.

And then for the MLP, you have again the similar logic of, I don't remember what this 1,536 is, but yeah, in the end it turns out into being 1.2 million. So total amount of parameter size we're going to be training is going to be 1.8 million parameters, which should be good for, which should be easy to train for, like, most devices.

So yeah, you can go back to this. It has a lot of details that, that makes,

that, that will, that will make sense for you to go back and understand a bit deeper, do your own research on, on this. But this is like the general, like, very, very high-level idea of how these transformers work.

And we're going to go now to the, the training loop. And that's where most of the meat of this project is going to be, where we have this, this transformer, how we're going to train it and do what we want to do.

The objective of this training we're going to be working today is something that, one, has to be very easily recognizable, that you know when the model starts working and, and grocks, which means that it, it understood what it's trying to do.

It's going to be very easy to understand that the model actually works now. And second, it should be something that's, like, quite easy for the model to learn. If you teach the model how to, to write a Python, like, a competitive programming for you, then yeah, that's a, that's, that's a, that's a very hard task.

So we're not going to be able to do it here. But in, in our case, like, the, the objective is going to be to create a Shakespearean-like LLM that can create

verses from, from Shakespeare.

And as we mentioned before, the, these models, like, are learning as next token prediction. So the way that cross-entropy works is that you take your current, the current tokens that you want to train. In this case, let's say that it's T0, T1 to Tn.

And then you want it to predict T1, T2 to T plus 1, Tn plus 1. So the way this cross-entropy works is that you take your sequence and you just offset it by one as, like, what the model needs to learn what to calculate.

And of course, the last one, you don't have it, so you have to, like, cut it off. And then the model learns how to predict the next token based on, based on this logic.

Now, for the actual training code itself,

first we need to load the data. And we have this function that, that loads the data. The data is already in the repo itself. It's in this data directory. It's a, it's a collection of different, different, like, lines and verses from, from Shakespeare.

It's, it's about 1 million tokens, so like a million characters.

And we first load the data. The data loading is, is, is very simple, too simple. So that's one of the things that I think you can optimize. Essentially, it just takes all the tokens, splits it into a validation and a, and a training set, and then just essentially shuffles it and takes part, 256 batches of sequences of, of, of, of tokens that are, that from, from the text itself, and,

and just essentially uses it for training. Batch size is going to be 64. So it takes 256 token sequences, 64 of them, stacks them together, and passes it to the model to, to, to teach it how to, how to train.

So a very simple data loader. Usually, these data loaders can be quite complex because, of course, like, if you have, especially when you have higher context lengths, like, the way you load data is, like, a very big part of how the model, like, needs to learn.

But in our case, we have a very, very simple implementation.

Next up is the way that this is how the code works for, we need to use a device. This code works for NPS, it works for CUDA, and it works for CPU, depends on what your, your laptop supports, or like, if you run things on Google Colab, like, if you run Google Colab, it will detect CUDA, and that will be quite fast.

NPS is also quite fast. CPU will be the slowest, but that's, like, that still would work decently well. Next up is the learning, the learning rate. And that's, like, a big part of how the, the models are able to learn.

The way the models work is that you first start generally with as high of a learning rate as, as you can, as you can afford, without making the model become unstable. Sorry, it basically makes it go crazy. And the way it works, you start at a very high learning rate, which is essentially the, the amount of the model is able to learn per step, how much the, the weights need to move into the direction that you want.

If you have very high learning rate, you would offshoot your target, so it can go off the rails very fast, your model. So you want to use a very appropriate learning rate

for your model to be able to train well. And usually, you have this, this concept of a warm-up where you start with a very small learning rate so that all your optimizations, the, the model weights are able to, to, to stick to places that are, that again, that, that don't go into, like, they can, they can start in places that are appropriate for the training to begin.

So you start at very low learning rate, increase it slightly until you reach the peak, and then at the peak, you continue, you start reducing the learning rate. That's what we call weight decay. And until you reach the point that you're, you're satisfied with it.

Some people, for some people, this is zero. Like, I prefer to not go to zero because then it's hard to restart the training. But that's the idea. You want the learning rate to be less when the model is, like, close to being perfect.

So small, you, you want it to start with very big changes to find, like, good local minima, global minima, and then it, for it to calibrate as training goes, goes further. So we'll start with, like, a small warm-up of 100 steps, and then we're going to use a cosine decay until the max steps that we're going to, in this case, it's going to be 5,000.

So you're going to start from, like, very low, peak at 100 steps, and then start going down, down to, to 5,000 steps. And that's what the AdamW normalizer does. It essentially allows this, this, this concept of, like, controlling the learning rate using this cosine decay.

And that's the most common normalizer people use, at least used to use. Now there's better normalizers, but this is the most simple to, to start with.

And here's how the full training loop would look like. Again, like, not, not, not that much code. You just

initialize your config. Like, in this case, it would be like six, six layers, six attention heads, 384 embedding size, and then 256, that's the sequence length. You initialize your model, and then you create your, you, you start your optimizer, you start your steps.

TQDM helps with, like, tracking, like, the losses and all that kind of stuff. You want your loss to start high and then keep going down until you reach levels that are acceptable.

And then another important part is evaluations to make sure that your model actually works well. That's what the val loss is here. So it's very easy for models to, like, overfit, especially like this, this, this small models, because you don't have that much data.

So the loss might keep going down, which means that the, the, what the model predicts and what the training data is are very similar. So your loss can keep going down a lot, but actually, at one point, maybe your model actually overfits in this case.

And when it overfits, even though the loss goes down, actually, the performance of the model is worse. That's why we have this val loss, which is a, is a part of the data the model has never seen. And we run a forward pass to get the loss of that specific part of the data set.

And if this is very low, it means the model can, is, is performing well because the model has never seen this, this data, so it can't memorize things it had never seen. So that's, that's what we have here with the val loss.

And

I'm not going to explain the concept of, of, of, of backwards losses, but essentially, that's the way the model train, the model weights move towards the direction of being optimized. So for, for each step, we have a batch size of, in our case, 256, 256 tokens, batch size 64.

So we put this matrix through our model, and then it learns, and then the optimizer does an extra step, and now the learning rates are adjusted depending on what the steps you're inright now. And lastly, just to have an extra way of, every, every 1,000 steps, you save your checkpoint to be able to restart your training if you need it from that point.

And we're also running inference on the current checkpoint to see what the model actually predicts at this point. So what we're, we're going to start seeing is

when we start from, from the beginning, because it's a model that we train from scratch, the loss is going to be essentially random. And in that case, that will be natural log of 65. So it will start at around 4.17.

That basically means the model, like, knows nothing. It has no clue of what this data is. And slowly, we're going to start seeing the loss going down to 3.3. That's when the model is going to understand character frequencies.

It will still not be able to do words yet, but it might understand things like "th" as part of "the," which is a common word. Like, "th" is going to be part of the things that it starts generating.

Then at around 2.5, it's going to, it's going to get a little bit better about this "th," and then it will understand the word "in" and stuff like that. Then at about 1.5 to 2 losses, it will start actually creating words.

And then at around 1, 1.0 to 1.2, that's when the model is going to start being decent at, at, at this task. It will actually be able to understand names from the text. It will start creating things that start making sense.

But then when the loss starts going below 1.0 for this specific data set, that's where we're going to start seeing overfitting. The model will still be, like, producing, like, reasonable things, but it will no longer start be getting better at it.

This is, like, an example. At 200 steps, when I was testing this, it was just producing, like, complete nonsense.

And the val loss was around 3.5. Then at about 800 steps, it started producing, like, decent things. Still not, not, not great things, but it was, it was starting to get there. And at 1,000 steps, it got better.

And there was one point that the val loss actually started increasing instead of decreasing, and that's where we know the model overfit. So at around 2,400 steps is where the, that was the optimal performance of this model. And then if we kept going, the, the performance was actually maybe not decreasing, but the model started becoming less creative.

So that's one of the things to, to keep in mind. Now, val loss is, like, not the best metric. Like, if you're actually being serious about training LLMs, usually you might have, like, some benchmarks that are running as part of your training, and you can see, like, if the benchmarks are getting worse or not.

But for us, that's, like, a very easy and cheap way of, like, understanding how the model is doing.

So yeah, that what next steps now, like, would be to actually train this model yourselves.

Considering that the internet is overly good, I would suggest probably using Google Colab for this.

You can, you can copy-paste stuff from, from, from, from here. You can glue things together. Copy-pasting will probably, like, allow you to get 90% there. There is a lot of room for, for improvement for what I've written here.

Like, on purpose, I made it, like, super simple, and there are things that you can improve yourselves. But the idea is to, to get something working. I hope everybody will be able to get something working if, if you're interested in do, into, into working on this and have something that starting from nothing and getting, like, a model that can actually produce, like, a result that seems reasonable.

After we have trained the model, the next part is text generation, which is, like, the inference side of things. We're just going to be using, there's multiple ways to, to do, to do inference. One way is, like, 3D decoding, which is what I was mentioning earlier.

You have all the logits, which is just a distribution of tokens, and you just take the most likely token. And that's what 3D decoding means. So let's say that you have, like, your, the token T and the token H are, are both in the distribution.

One has, like, 80% probability. The other one has, like, 15% probability. You will always be taking the, the top, the top one.

That's 3D decoding. 3D decoding doesn't work very well for LLMs. It can work well for other models, but for LLMs, like, this essentially makes them very boring and not very creative in what they generate. So you pretty much never want to use 3D decoding for LLMs.

You would want to use it for, for other models. Like, transcription, for example, 3D decoding is the best because there's usually, like, only one way you can, you can transcribe something. You don't want it to be creative in transcription.

Like, that's not, that's not a good idea. So that's what 3D decoding is. In our case, we're not going to use that. We're going to use temperature. So essentially, what temperature is, is that you're not always choosing the highest probability token.

Sometimes you might choose the second highest probability token or, like, the third highest. And even though it doesn't make sense, why would you choose, like, a worse token for this situation? It has proven that this actually makes the model perform better.

It like, sometimes it might, like, go into, like, this weird loop. So there are, like, techniques you can make sure that it doesn't go, like, crazy by generating some nonsense. The worst thing is if, if it predicts, like, an end-of-transcript token, an end-of-text token, and then it just, like, stops the generation, which maybe you have seen sometimes when using ChatGPT where suddenly it just, like, stops for no reason.

You sometimes it's because of that. But there are ways you can, you can, you can prevent this.

Generally, like, a 0.7 temperature is, like, the best, the best middle ground to make inference work well. And then you have top-k sampling, which is essentially, like, it prevents the model for if you have, like, let's say, five tokens that are very likely and then the sixth token is, like, completely unlikely, top-k sampling prevents the model from predicting this sixth very unlikely token, even though the temperature might, you might get unlucky and the temperature might hit it.

So that's what top-k sampling does. And this is, like, our inference function is, like, it's very straightforward. It just does, it just runs, takes all the tokens as the input, passes it through the model, takes the logits out of the model, and runs.

Soft, this is what I was describing before, it's called softmax. It takes, using a temperature, the probabilities, and then it, it decides what the, the next token should be based on, on those probabilities.

And one thing you can do, you can use seeds. And essentially, seeds is that, like,right now, everything would be random if you just keep returning it. But if you, if you keep retrying it, but if you use a set seed, then your inference would always be returning the same value, which will be relevant in later.

Then putting it all together, if we put it all together, we should have three different files. One is, like, the model.py that includes our model architecture. One is train.py, which includes the data, data set loading and the training loop.

And lastly, the generate.py that includes our inference. And in total, this should be, like, maybe a few hundred lines of code, very straightforward. And that's with, even with this much code, like, if we have a lot big hardware, we could train a good LLM using this architecture if we have enough data and enough resources.

That's, that's all you need, essentially. And that's, but basically, like, GPT-3 and GPT-2, like, when OpenAI released it, I remember when OpenAI was about to release GPT-2, and they were saying, "We're not going to release it because it's too dangerous for, for humanity."

And I was like, back then, it, that was the code that they were working on and, like, a lot of data and, of course, a bigger model. Now, this seems to be, like, kind of, kind of funny, but for them, like, that was, like, a very wow moment that we did this and then the model, like, actually performs really well on tasks.

But in the end, it was just, it was just what you see here.

So yeah, putting all together, you can, if you use Google Colab, you can use this, this snippet of code to download the, the data set, and you can use this pip install command to install the different dependencies. And then it should look like something like this where you'd install your dependencies, you'd copy your code,

and you might need to do some connection, some connections together. But essentially, like, you can run a, a train, you can use a train command to run, to, to, and use the data set that you want to use, and that this will start the training.

And then you can see how the performance of the model improves step by step. Like, for me, this took about, like, 15 minutes to train on, on Google Colab until I started getting good results. With some improvements, you can make it go faster, and maybe you can make it go slower, but actually get better results.

But essentially, like, it's, it's very simple to get, to get working. One thing to remember is that you have to change your runtime, change the runtime type to T4 GPU because this, this, this is free and it will, it will run quite fast.

And yeah, feel free to train. Like, you can try, like, maybe starting with a very tiny model, the 0.5 million parameter that only has two layers and only two attention heads and a smaller embedding size for, for each token.

And then you can try bigger and bigger models until you get some usable results.

And as I mentioned before, you can try different context lengths. I started with 256. You can try bigger, 512. And then you can, if you want to monitor and see how your losses went down, like, in a nice way, you can use this, like, this pyplot lib to see, like, the graphs.

The things that you want to look for is if your train loss is not decreasing, it means your model is not learning. That probably means you have a bug in your code. If your, it also could be, like, a low training loss.

If your train loss is, is decreasing, but your val loss is not, is actually increasing, that means you overfit. And if you have very weird spikes in loss, like, loss should be, like, very smooth in general. If you have very weird spikes in your loss, it means, like, again, there's some, some kind of bug either in your data or in your training.

And when the model starts plateauing and not getting any better, it means you kind of, you have pretty much exhausted the usefulness of your current data sets. So either you're going to use a bigger model or you're going to need, you essentially need more data.

Now, one, one, like, interesting part about what I would like us to do is, I think it would be cool if we had, like, some sort of competition of who can train the best model here, if anybody is able to, to get things working.

Hopefully, the internet is good enough. And also for further reading, you can see, like, a lot of the resources for this, for this workshop. But essentially what the challenge is, is we can, we can vote all together to find which, which model actually produces the best verse of Shakespeare, or it could be if you use a different data set, a best poem or in that kind of category.

The rules are you have to train the model yourself, like, here and today. Like, it can't be, like, some, you can't ask ChatGPT to give you, like, a good verse. You have to, like, use your own model. And to prove this, you'd use, like, a seed with a specific prompt and see what outputs it gives you.

You're free to, like, regenerate things as many times until you, you get the best results.

And you, I'll have, like, a QR code that you can submit your results and, like, I can go around and, like, help people out that if you need any help getting training running. And of course, this training is, like, super bare bones.

There are many ways you can optimize this and make this better. So I guess for people that are a bit more experienced, they can, they can implement this, this improvements and maybe get a better outputs of your, of their model.

The winner is going to get some free swag from, from ElevenLabs, maybe like a hoodie or like some, some free credits. I'll see what I can actually give. So yeah, this is the submission. It needs to be creative and then I'm, and it needs to be essentially like a good verse.

And we can maybe use, like, a kind of, a kind of bracket. Oh, what happened? Oh, cool. We can use a bracket and we can vote which verse, like, sounded best. That could be funny. That could be, like, like, well-made.

Like, up to you. And the reproducibility should look something like this where you run Python generate on your best checkpoint, the prompt that you decide that you can, it's up to you, temperature, and then

a seed that, that proves that you can actually reproduce your results. You can try different model sizes.

Like, I guess, like, 85 million parameters would be a bit too big for this, but if you have the resources, you can try it. You can try better tokenizers. Like, this one, of course, is, like, character-based, but perhaps you can train your own BP tokenizer based on that data set.

And there's also other tweaks that you can do, like bigger context. Like, there are some training optimizations, like using a dropout value. You can stop whenever you want when you feel like the model is good enough, change the learning rates, and essentially get, like, as make the model as good as possible.

And yeah, that, that is the idea.

I'm going to go around and help you out. Sorry, go ahead.

Can you say that reasoning models is a value very different for training or is it similar but just some small, some more small?

So reasoning models, like, to, to repeat,

you're asking, like, if reasoning models are quite a bit different in training. The base, the building blocks are very similar. Like, you can train the same exact model, you can post-train it, which is how, like, usually reasoning is, is, is being taught to this model.

You have a good base instruct model and then you've post-trained it to be a reasoning model. This is very data set driven. So you need very, very high quality data and you're going to use a loss that's, like, good enough to be able to, to learn this data in a, in a very good sense.

The complication of reasoning models is finding this good chain of thought data. That's why, like, OpenAI has, like, all these labelers that are, like, PhD students that they write down the reasonings of, like, how to, how they solve problems.

Because this data needs to be very high quality because it's, it teaches the model how to think. So you can't just go on, like, Reddit and just get random posts. You're not going to learn how to think this way for sure.

You need, like, some very high value, good quality data to teach this reasoning process. But in the end, reasoning is, is essentially just adding to the context of the model, like, this, to the attention essentially, this, like, logic that then the model can, when it generates the response, it can go back and attend to those reasoning tokens and get a better response out.

So it could be, like, describing the model a bit better. Then it goes back, sees those tokens that it described and say, "Oh, actually, I already figured this out. I'm going to write it down now."

The microphone is not working.

Is it? Yeah, perfect. So reasoning and non-reasoning models share the same base often.

Yes.

And then one is post-trained in a different way and adds this, you know, in a way different post-training.

Yes. So a lot of the labs, like, let's say the Qwen model that was released, like Qwen 3, usually they release, like, a base version and instruct version. And usually the base version doesn't have a chain of thought reasoning.

It's usually, like, the same model that they first pretrain to be, like, quite good, maybe do some fine-tuning as well. And then the next step is doing post-training to teach it this kind of, like, this, this, this knowledge.

So that's why, like, you see a lot of, a lot of big improvements that happen very fast in the industry. Like Gemini 3 to 3.1, it's essentially giving it, like, better reasoning data and, like, like, better fine-tuning post-training data for, like, specific problems.

It's a little bit like benchmarking. Like, usually this data is, like, very similar to what the benchmarks, the current good benchmarks are. But yeah, essentially just taking a base model, using this new data to improve it to the next level.

Thank you. And second question is, compared to what you are showing us here, like this very bare bone model, are there, like, have there been fundamental innovation in the main training of the model, like powering nowadays models, or is it mostly the same, but just with smarter tricks, small tweaks, better data and so on?

So I, like, are everyone still using the same attention layer and so on, or are there, like, some fundamental shifts that have been happening on the latest and maybe, you know, maybe don't, I don't know.

No, no, so you're correct. That, that, so they, a lot of this base is the same. They might do some changes in terms of how attention attends to those different tokens because sometimes reasoning can be quite large. So you need big sequence lengths to be able to get good results.

So there's a lot of, like, tricks that the new labs, the labs do to make this, to, to make the attention more efficient. But overall, you can take, like, a very, you can take this model, GPT-2, and make it into a reasoning model.

Like, if you have the data and, like, a big enough model to actually learn from that reasoning as well, because, like, these tiny models, reasoning won't help them too much. But if you have a big enough model that can get stuff out of the reasoning, there are people that have taken, like, older models, like, let's say LLaMA 1B, that are small and weren't trained for reasoning, and that made them into reasoning models using the exact same architecture.

Yes?

How much effort do you put in, you know, getting your golden data set and what is that process that you follow?

You mean for post-training?

Yeah.

Yeah. So, like, as I mentioned before, like, usually what most labs do is that they go to companies like Scale AI. That's probably the biggest one. And Scale AI has an army of, like, people that you can, you can say, "I want physicists.

Give me data from physicists." Then Scale AI is going to find, contract physicists, pay them, like, as much money as they need. And then these physicists might, like, write things down. They might be contracted to do different things.

But essentially, like, a lot of these companies, they, like, Scale AI provides data for, like, Anthropic, another good bot by Meta. So probably not that much anymore. But essentially they, they, these data sets are provided by these, like, labeling companies.

Scale AI is one of them. But a lot of the, the big labs have their own labeling teams too, that they hire contractors to, to, to generate this data set for them. But as, as I said before, like, you want, you, if this data set has, like, even some small issues, it can really make or break your model.

So these data sets are, like, kind of the most expensive. Like, they will cost tons and tons of money, but they actually are the ones that make the models, like, as good as they are.

Just a quick follow-up. So even on that, I mean, you still have to evaluate,right? The answer wouldn't be exactly the same. So are people are really reviewing this or it's still you're relying on LLM to evaluate?

You, you, you rely on people for this kind of stuff. Usually the way it works, a big part of my job is actually, like, doing this organization. Usually the way it works is you have, you might have, like, one person that generates the lab, this is, like, very high quality label data.

And then you have maybe some label that has graduated into, like, a QA position where their job is to essentially make sure that all the, like, other labelers, the more entry level, their outputs are correct. And it's, it's, it's quite the tough job because, like, if your QA is not good, you get fired.

So it's, like, it's, that's the way they keep the level quite high. Yes?

So LLMs are non-deterministic, but how does that seed parameter work?

So the, the way the LLMs are non-deterministic is because they, they essentially use random number generators for, of, like, the, of, of your machine or, like, of the, the, the, the GPU that you're currently using or the system.

The seed essentially, like, makes all those different calculations to be always return always the same value. So things are no longer random. Like your, it is not only for, for, for this. These seeds can be used for any random generation.

So it could be, like, for password hashes.

Other systems, but I wasn't sure.

Yeah, it works the same exact way. Yes?

This is, like, you mentioned predictive coding,right? So we use that, that they are deterministic,right?

Yes. Yeah. If you use grid decoding, there might be still some things that, like, like, it's good to use a seed even if you use grid decoding because there might be some other stuff that maybe you don't, you don't, you don't control.

Like, maybe you're using some, like, sometimes grid decoding might not be actual grid decoding, might be 0.01, like, temperature, like for VLM, for example, that they do some, like, tricks like that to be able to get good outputs.

So in general, it's good to use seed even if you're using grid decoding.

Yes?

This is obviously all in text. ElevenLabs is mostly audio,right?

Yes.

How different is this versus, like, doing this with audio?

It's surprisingly very similar.

Really?

It's more complicated for sure, but parts of the, the stack of, like, most audio models, they also can do text because the, you need, the models need to understand language itself. And the best way to teach something language is text.

Of course, you can train an audio-only model. Like, and again, it's like what I was talking about the tokenizer. Like, how do you tokenize audio? Like, what, what is the concept of, like, a sound? Like, how, how big should your tokenizer be for audio?

What, should it be only human speech or should it also be, like, music? Like, do you take the notes of different music instruments, like, different tokens? This is the kind of problems that you have to solve. Like, if you're creating an audio model that you wouldn't necessarily need as a text, it's a bit easier.

But the fundamentals are still, like, the same. Like, if you want to generate audio, you would, you'd generate, like, an audio token. And then that audio token would be using a tokenizer. And then you, you, it's not exactly the same because the audio token, then you have to process it in some different ways to make it into, like, actual audio.

But a lot of the same general ideas still apply.

But do you have a lot more loss or something? Because with text, you know, like, this is the text. The text can be down to very clear, like, a one is a one. But in audio, like, a particular pitch or frequency or soundbite, like, there's ambiguity there since already lost straight away,right?

Yeah, yeah. So the, the, the way it, it works is that you don't use, like, this cross-entropy loss. You use different types of losses. You can use cross-entropy, but usually it's, it's losses that are more specialized for what you're trying to do.

Like, for example, there's a loss called L2 loss, which essentially takes, like, two Mel spectrograms and tries to see the difference between those two Mel spectrograms, which is essentially like sound waves encoded in a certain way. And that's, like, a very common way for TTS models to be trained.

You train on these specific types of loss. And the same thing I was mentioning earlier, cross-entropy doesn't really work as well for, for things like post-training. You might use different types of losses there. Or if you're distilling a model from a big model to a smaller model, you don't use cross-entropy loss.

You might use a KL divergence loss where you find the, the token distributions of, like, the bigger model and you try to match the logits of the smaller model. So there's different types of losses for different use cases.

Not all of them work the same way.

Yes?

So in a sense, you have, like, an audio architecture,right? And so you train on that. But what do they do with multi-model models? Do they just, like, direct regular text vocabulary with audio vocabulary with pixel vocabulary together and train that?

So the way multimodal models work, usually you don't use tokens in the same sense.

Like, okay, this is where, like, it becomes more complicated because, like, these models are not really built for this, like the GPT model. Like, the newer models, they have also, like, an embedding input, which essentially, as, as, as I was mentioning earlier, like, you have each token then corresponds to, like, a vector.

But these vectors, they don't have to correspond to a specific token. You can take these vectors from other places too. And what a lot of these labs do is that instead of having a tokenizer for, like, video, for example, what they do is they have another transformer that they call a video encoder.

And they put the video first through that video encoder. And this video encoder will be taking, let's say you have a, a 30-second video, it will take one frame per second of this video. And it will take those frames and then put them through this, like, new transformer, this encoder transformer that works quite a bit differently than this one.

And what you do is that you take the final layer of this transformer, the hidden values, which are also vectors. You take those, those vectors out of this encoder and you're going to input them in the embedding layer of the, of the transformer model.

So what the model is going to see is going to, usually it's, it's like prefixed. So you take the, you take the video, put it through the encoder, get some vectors, and then put those vectors in the embedding input of your transformer that does text.

And the way it would look like, if you look at the sequence, it would probably be like a prompt and then it would be probably like a video token representation. And then actually, but the embedding of that video token is going to be overwritten by the output of the encoder.

So that's how, like, these, these multimodal models work.

Produces a vector which is the same length as, like, a vector for words?

Yes. It's exactly the same vector. The same for audio. You have an audio encoder and do the same, but, but, but for audio. And, but for, in, in terms of how, what the model cares about, it just cares about, like, this embedding side.

It doesn't care if it's text or if it's audio or if it's, or if it's video. It cares about these vectors. And that's how you, you represent them into the, the same dimension as the model, the transformer expects.

So is there no relation between, like, the video of the horse jumping and the word horse? Is there no similarity between those two vectors?

Maybe there is. I'm not, it depends on how you train it. Like, it's, these are the kind of things that are a bit black boxes. Maybe there is, maybe there's not. Actually, that's a good idea to, like, that would be a good research paper to, to see, like, for video encoders, do they actually match in the same dimension as the text encoders?

I, I don't know. I imagine there probably is some connection. Sorry, you had a question as well?

Yeah, I was wondering about, like, because you do both the normal speech, but you also do some music generation. Is that, like, a very different model or will the architecture be, like, similar? And also, like, when you have this harmonics and stuff, are you still able to do it with just, like, a basic, like, auto regressive transformer or do you have to, like, do diffusion because, like, things depend more on each other and you might want to, like, generate everything at the same time?

You, you can do both. There's music models that are auto regressive. There's music models that are diffusers. Like, it depends on how you train it. Like, I think some of the, some of the Google models are, like, like transformer-based.

Some of the open source models are, like, diffuser-based. Like, both can work, can work very well. It's just that it's, it's a little bit, as I said, like, it's a little bit more difficult to, to, to put into perspective this, the concept of, like, odd music.

If you tokenize and predict the next token, it's kind of hard because it's very abstract. So usually diffusers work a bit better in this, like, image modalities for generation or, like, music or, like, even audio for some models.

Like, they have some kind of diffuser, diffuser, diffuser part of the process. Both can work. Diffusers are generally a bit easier to get it working. I hope this answered your question.

Does that make sense?

Yeah.

How do you tokenize audio? Because I imagine for text you get the repetitions, like, repetition happens quite a lot. It's easier to combine those different tokens. Create bigger tokens for audio. You have different voices. I don't know, like, how long, how long a token would be.

It's, it's very hard. And it's not really something that you do by, by just sitting down and thinking, okay, I'm going to tokenize this word to that. It's not something that you, like, sit down and, like, decide. You use some kind of process and you train, like, an audio tokenizer through, you have a, it's a very similar, like, case, I guess, in how you train, like, a text tokenizer.

You'd use your training data and you'd find, like, common patterns in the audio and then tokenize those patterns. Of course, like, when we say audio, we don't, don't always just mean, like, the actual, like, sample rate and, like, the, the, the audio waves.

Usually convert them to something that's a bit more easy to, to tokenize, to, to do this processing. The most common one is, like, is Mel spectrograms. First you convert your audio to Mel spectrograms and then use this, this, this, essentially arrays of, like, of numbers to train your, your tokenizers.

And it will be very dependent on, on your training set. Like, if you wanted to tokenize music and use a music data set, your audio tokens for music are going to be very different than if you had a voice data set that's going to be, like, focusing on, like, human voice.

Now, the hard part is what if you want to do both voice and music? That's where, like, that's, that's very hard.

Any more questions?

Okay. Awesome. Yeah, if you, if you want, you can start working on training the model if you have a laptop out or if you have already started. You can follow this. You, you can, you can follow the, the workshop that gets something working.

And if we do have enough submissions, we can do the, the competition and, and see who, whoever wins is going to get some, some nice work.

What about for, for getting these things in?

Sorry?

What's the cutoff? The voting begins shortly after.

So, because we don't have that much time left, let's just say 5:45.

If you have any, any questions or you need any help, please call me and I'll, I'll come over.