Low Level Technicals of LLMs: Daniel Han

Welcome to the AI Engineer World Fair. This is the first workshop; there's a few others running, but thanks for coming. We just arrived from Australia, with my brother—I think he's over there somewhere—but yes, we just came here. Yes, we didn't know a lot of stuff about SF, and I think maybe the U.S.

is a bit different from Australia. But yeah, we're very excited to be here. We're going to stay here for a few months, so if you want to meet up, you can just hit me up via email or Twitter or wherever.

So today I'm going to be talking about low-level technicals of language models. Yes, yes, I'm Daniel. So we do have a website called Unsloth.ai if you want to look that up. There's cute sloths and stuff, so my brother designed that.

We'll be using two tiny URLs. Oh, did it? Okay, it's still working. Yeah, so we'll be using two tiny URLs. Now, the first one is—oh, wait, I should—yeah. So the slides are at tinyurl.com/unsloth. Hopefully that works. And there's also Q&A, so I'll be monitoring Q&A.

You can type any question that you like, and I will be answering questions as we go. And that is at tinyurl.com/unsloth/qa. So if those two work—so they'll be on the bottom. If anyone doesn't get this, on the very bottom of the footer we'll re-show these links.

Okay, just—does it work? Yes? Okay, good, good, good. Okay. Yeah, so you might know me from my tweets. So Gemma kind of released an open-source model a few months ago, and we just found a few issues and bugs for different implementations.

So, for example, the first tweet that we ever did was about some sort of the approximate Jelu bug issue. And so multiple implementations of Gemma that had different implementations. Some of them used exact Jelu, some of them used approximate.

And so which one is correct? And so that's the question. And so we just tweeted about this, and that was our first issue that we found. We thought this was just one issue, but actually there were many issues.

And so we found more bugs. And so I'm assuming maybe you know me from this. We did get partially recognizable through our Gemma bug fixes. So yeah, so today we'll be showing you how you can actually find these bugs and issues in language models, and how you can actually analyze and do this yourself, without us just doing it manually ourselves.

And hopefully this can be an open-source project where everyone can find these issues automatically and help us to solve these issues. I always thought about, can we automate this? I don't think so can be automated. There are actually many issues with these implementations, and it's not just Gemma.

For example, we also analyzed Grok, and there's some weird things in their code. They're scaled by 30 times 10h x over 30. It's just a clamping mechanism. But you can see I also make mistakes sometimes. I said it's division and not multiplication, so sometimes I misread the code.

That's because I spend—when the code gets released, I quickly try to analyze them, and sometimes I mistakenly say stuff. So I have to lodge showcase corrections. So yes, I am still human. But yes, we analyze lots of models and stuff like this, and hopefully by the end of these workshops—well, actually this one—okay, it's actually not one workshop.

It's going to be multiple things in one. So I decided to talk about three things, but I'll tell you about that later. Hopefully you guys can analyze and learn about how to find bugs and stuff like that by the end of today.

So another one I did recently was NVIDIA's Nematron. I don't know if you saw this, but NVIDIA released a 340 billion parameter model, which is extremely large. I'm assuming this is in

preparation. They have to release this earlier before Llama 405 billion,right? So they have to do this quickly. But there are some weird, interesting things. They used the squared value and not the normal swigloop and the other types. So that was very interesting.

They were actually the first—well, actually not the first, but the first big model trained to be using these other activation functions. And there's other weird quirks and stuff like that. And hopefully you'll be also able to analyze whenever the code comes out, just read it and you'll get it.

It does take some practice. Okay, I take—okay, the first time when I read this code, it took me many, many days to read through all these architectures and understand exactly what they are. But now it takes 10 minutes, so I'm sure you can just—the code comes, you just read it.

That's the whole goal of today. And also, language models, if you don't know, they're not just issues and bugs and analysis of these architectures. The tokenizer is a totally separate beast from language models. Tokenization is extremely annoying. So I also tweeted about there's different types of tokenization issues, like Mistral, Llama, Mixtral, all these different types of variants of Mistral from the Mistral team.

They have different tokenizations if you didn't notice. So the smiley face, the sun smiley face, is a space. And if you actually tokenize them, depending on the model, you will have different results. And the question is, which one is correct?

Unfortunately, I do not know. I did ask the Hugging Face team for this, and according to them, some of them are correct, and some of them are just because the Mistral team forgot to update the model to the fast tokenization variant.

We will be talking about this later as well. But you can see, even before you can train or run the model, the tokenizer is broken. So what are you going to do? It's a multi-pronged problem. So we don't just do language models.

Our experience is a bit broader than that. So we actually—I used to actually major in maths and computer science. So yes, very fun. Well, actually, I kind of did very badly in maths. But anyways, yes, very, very fun.

So SVD, I don't know if anyone—has anyone done normal machine learning here? Oh, yes, very good. There is a few people. So the SVD, I'm assuming most people know PCA. Yes? Principal Component Analysis. Yes. Okay, very good. Data visualization, it's a very powerful technique.

More people should know about it. SVD, okay, I don't know if people know about SVD. Okay, yeah, so it's a bit less well-known. I'm actually a bit confused why people don't know SVD. It's actually the algorithm that's one of the most important algorithms in all of maths and computer science.

It literally underpins many applications, and it is extremely important. Maybe we'll talk about that. But yes, I'm a huge proponent of telling people to learn more about SVD. So please do the singular valley decomposition. That's the must, must, must, must, must.

Okay, that's the most important algorithm. It's because it's one algorithm. It can spawn many other algorithms, and it can be used for many purposes. There's also the QR decomposition. Okay, probably shoot. Okay, probably don't. No one knows. The LU, Cholesky, there's a lot.

Randomize SVD, yes, that's extremely important as well. And yeah, so we don't just do language models. You can ask me any questions about stuff, maths, or computer science. You have a question?

Real quick. So do you think—so for the Nemo Tron 340B, is it a unique architecture? Because you can only use Nemo loaderright now to load and train, do the—I think that data is the most valuable part. But we are attempting to try to convert it to a Hugging Face transformer safe tensors.

But we've had issues because we don't have the modeling file. So I was wondering, do you think that it's similar to—so at the same day, they uploaded a 70B of Llama 3 that's Nemo Tron as well. Do you think that we can get some clues as to how to build a Hugging Face implementation

from there?

So the question was, for Nemo Tron, the code was not released for the actual inference and training. You have to go through the Nemo training framework from NVIDIA.

Yeah, well, what I mean is that I can dump the weights. Weights are great.

Yes, but the code, yeah. I was actually planning on doing something like that, but as you know, we didn't have time to do that. So I probably—yeah, I might take a crack at that.

Is there going to be a Q&A? I don't want to ask any more questions.

Oh, yes, no, no, there is. So you can log questions on Q&A. There will be Q&A. So we—no, no, yeah. No, you can—anyone can raise their hands and ask a question. I'll just repeat the question. But there is the slider.

If you want to—random questions, I will keep monitoring. Yeah. And yeah, so—oh, yes, the other one was another paper called "LoRA Learns Less and Forgets Less." It shows that fine-tuning via LoRA does not really work for learning new knowledge.

And well, it depends. It depends on how you actually read the paper. Some components were incorrect. They didn't actually train on all linear layers. They kind of forgot a few. And they also—you need to do some sort of special parameters to make this work.

And we will also be talking about that as well. But I was just trying to show you that we don't just do language models. So we have a whole wealth of knowledge across different industries and stuff. Well, not industries, topics.

And you can ask me any question that you like. So Unsloth, yes, we launched this last December. So I launched this with my brother. It's a bit outdated, the stuff. But anyways, I think we have 11.9 something K or something.

I don't even know now. But anyways, we launched this last December. It generally makes fine-tuning of language models like Llama, Mistral, Gemma faster, two times faster, generally speaking. And we have 80% less memory usage now. We have some new methodologies which reduce memory even further.

And the trick is there is no degradation in accuracy. So we don't do any approximations. That's the whole purpose of doing the optimizations, is we don't want to lose any accuracy. And so we do Triton kernels. So this is from OpenAI.

It's a language to do CUDA programming. Essentially, it's like an intermediary between the CUDA code and Python itself. And we'll be showing some Triton code. I don't know if we have time for programming Triton, but that will be another topic.

And yeah, so the whole purpose of Unsloth is to make everyone be able to fine-tune their language models with very bad GPUs,right? So Tesla T-Force, the free Google—does anyone—people do know that Google Colab has free Tesla T-Force,right? Yes?

Yes,right. 65 teraflops,right? It's actually not that bad if you use it properly. Just a reminder, there is a common misconception that the P100s on Kaggle is faster. That's actually not correct. P100s, I think, are five times slower than Tesla T-Force.

So although it's actually more expensive as a GPU, I think, I think, but it's actually slower for—so please do not select the P100s on Kaggle,right? So Kaggle has 30 hours for free per week GPUs, and you get two Tesla T-Force.

So that's 130 teraflops per week, 30 hours. And that is actually very powerful. I think that's the same as RTX 3070 or that. Yeah, I can't remember exactly, but yeah. So Kaggle has 30 hours for free per week.

Google Colab, it depends on how much you use. Normally you get four hours per day, I think. I guess the Pro is not that bad. It's like $10 per month. You can actually get—yeah, it's pretty good. Yeah, so probably get Pro.

I do not suggest—I mean, you could use RunPod and Lambda Labs and stuff like that. I guess that's another option. But we do actually share a pricing comparison. So what you need to be careful when you use GPUs, there is a big issue is like, "Oh, look, this is the most—I want to use a H100."

Did you actually check how much flops the H100 provides? Be careful of NVIDIA's marketing. It's times two because it has sparsity. So just be careful of that. And also, you have to be careful of the flops when it's like float 8 or float 16.

So just be careful of those. I do have a pricing comparison where we normalize by the flops with no sparsity. And we look through Lambda Labs, RunPod, Google Colab, AWS, Google Cloud. And I think RunPod is mostly pretty good.

Yes, question.

Why is it times two?

Oh, the sparsity?

Yeah, why?

So okay, so the question was, why is times two for flops for the sparsity feature in NVIDIA on GPUs? So the sparsity feature, what it does is you take 50% of the weights and make them go to zero.

And NVIDIA essentially allows you to train this two times faster by not doing matrix multiplications on the zeros,right? So you're like two times zero is just zero. So you essentially don't fire the transistors. And essentially, this makes it two times faster.

Well, that's actually not—that's just a high-level overview. But essentially, you compress the matrix into this special format, and then this NVIDIA special format allows you to do matrix multiplications two times faster. Yeah, so it's sparsity.

Is it only on the H100?

So it's on H100s. It's on A100s as well. So your RTX 3060, your RTX 30 series has that feature from RTX 30, the 30 series. I think it has. Yes, I think it does.

We can enable it ourselves.

So if you want to enable it, the biggest issue is that the language models, most companies do not train their language models with sparsity enabled. If you set the weights to go to zero, you actually ruin the behavior of the model.

So there are papers which show that you can actually turn on the feature, and then you can do fine-tuning to make it work. So there are actually papers which do that. So in theory, you could enable this, but it depends on what models are released from the large companies.

Yeah, I'm assuming—I think I do know if Facebook is—they did implement sparsity in their PyTorch and XFormers library. So I'm assuming they might be focused on sparsity because you get times two faster. And if you know OpenAI, they keep saying, "Oh, it's two times faster."

Hmm, I wonder why. Why is it two times faster? Why is it the two times faster,right? So could it be sparsity? Could it be float 8,right? Float 8 is generally two times faster. Okay, not exactly, but approximately two times faster.

So all these things, when you hear two times faster, where does it come from? Could it be these things? Although we don't know, but we're just guessing. Yeah. Any other questions? Okay, just remember, you can raise your hand, or you can do slide.

Wait, are there any—I'm assuming there is no slide or questions yet. Yeah, okay. Yeah, just raise your hand. So for Unsloth, we do benchmarking against Hugging Face plus FlashAttention 2. And we just show our benchmark. This is kind of old already.

The memory reduction is much more now.

And we did a blog post with them. So thanks to Hugging Face for the collaboration. And essentially, all you need to do is, from Unsloth, import fast language model, and we try to make it as easy as possible for people to fine-tune a language model.

And yes, we will talk about Unsloth a bit later. Oh, there is a question. Is it a myth or solid hypothesis that linear versus cosine, then short 1 to 2 epochs versus 3 to 5 epochs is highly generalized?

I think it depends. So for training methodologies.

Sorry. So is it a myth or solid hypothesis that linear versus cosine, then of course short epochs versus long epochs is the highly generalized best way to train any standard-based model?

I think it depends. So there are some research papers which show that cosine or linear schedules. I mean, it depends. To tell the truth, I think it's a toss of a coin. I don't think so it's actually that important for the learning rate schedule a lot.

I think it more depends on the data set, the number of parameters. There are research papers which show that if you just simply change from untied weights to tied weights, you can get better accuracy for smaller models. So I think it's—so the learning rate schedule is not that important.

You might get accuracy plus 0.1%. Just train for more data. There we go,right? Get more data. Oh, wait, I pressed back. Just train for more data, and I'm assuming it will be similar. But to tell the truth, I think it's best to do small experiments and then test which schedule is the best.

But I don't think so it's that important. I think for the number of epochs, that's actually important. These big companies, to tell the truth, I'm not sure what Llama—15 trillion tokens, is it actually 15 trillion tokens? Or is it like 5 trillion tokens times 3 epochs?

I do not know,right? These questions are very important. If it's 5 trillion tokens times 3, that's actually very different from actually 15 trillion tokens in total. That's actually very, very different. But in general speaking, if you train for more epochs, 3 is generally the, well, good, approximate 3 epochs.

1 is actually the best for pretraining generally. You shouldn't retrain your data multiple times. But yeah, did you have a follow-up question or no?

Well, so basically, learning rate was one of the big issues you fixed with the Gemma implementation.

Oh, yes.

So that's why I kind of—that's where my pitfall was when I was training my 2B for Gemma. And so I actually trained it pre-your-fix, and somehow it turned out the benchmark after your fix is better than—I don't know what happened, but it now is one of the highest-ranking benchmark 2Bs.

Oh, okay.

So I don't know what—do you have any theories about what could have happened? I trained it on the Transformers broken version, and then subsequently using Oxylotal and using a very hard—we brute forced a learning rate, basically. But it turned out surprisingly well.

And we also—we didn't use Unsloth, but we used AGPUs, so.

So this is after the fixes that we did? It does.

Before.

Before? So before does better?

No, well, no, before it was usable. Everybody else's was unusable,right?

Yeah.

But it was usable, which surprised me because everybody else's was unusable. But then after your fixes, we are now the top—well, my company has on the OpenLLM leaderboard has the highest.

So you didn't even retrain? You just.

No. So after the fixes, somehow it.

It made it better? Okay.

Do you have any theories on that?

To be honest, I do not know. I think because the fixes that we did for Gemma are multi-pronged. It's not like one fix is like 9 or something. So I don't know which fix caused the change. There could be all of them, maybe.

I don't know.

It's kind of like just for me, when I did the training,right, and it turned out good, and then I heard from all my friends, "Oh, I can't do this. I can't do this." It's like it just shocked me, I guess.

Okay, yes, that is quite shocking. If you didn't—yeah, we changed the code, we kind of fixed all the issues, and then you don't need to retrain it, and it does better? Okay, that's a very interesting phenomenon. I do not know.

I can send you the code.

Oh, yeah, okay.

It's actually open source on my.

Okay, yeah, great. Yes, to be honest, language models, I mean, these are all active areas of research. Please, someone do a research on that.

I'm amazed at this interpretability now.

Okay, yeah.

So I think these are super low-level stuff.

Yes, I cannot say anything other than I just read the code and fixed the bugs. I do not know. So.

Sorry, sorry.

Yeah, yeah, don't worry. Okay, we also do long-context fine-tuning. So we show that if you use a new methodology which does gradient checkpointing and offload it to system RAM, you can randomly increase your context size by 4. And the weird part is if you offload correctly to system RAM from the GPU, weirdly, the time of execution is just slower by 1 to 2 percent,right?

So this is very weird. It's like if you can do non-blocking calls and offload the GPU memory into system RAM, if you do it correctly, it's not slower. Some implementations, unfortunately, offload incorrectly. I do not want to name anyone, but they offload incorrectly.

Sometimes they offload to disk. I don't know who came up with the idea of offloading to disk. But anyways, please try to offload to memory first and then disk,right? Disk is extremely slow. And if you can offload to memory system RAM, you can actually get away with a lot of memory usage.

Okay, so I should have put this at the first slide, but anyways, so today we'll be having three approximate topics. And I wanted to make them into three different separate topics, but I guess I just mixed them together.

Whatever. So you'll be learning about low-level technicals of language models. For example, backpropagation. Why is Transformers not O(n2)—sorry, O(n3) for training and rather O(n2)? And there is a lot of maths, but I will try my best to reduce.

I think I already tried my best to reduce the maths, but there are still some maths. So please handle the maths. I will try my best to explain as simply as possible. That's the whole goal of the workshop.

So not that bad maths. You will actually understand the formulas very well. Just a reminder, I kind of nearly failed my maths in university, so do not worry. Do not be scared. It's very fine. We'll be talking about Unsloth fast fine-tuning, the best tips and tricks for fine-tuning, how do we write the fast kernels for fine-tuning, how do we actually make it two times faster, use 70% less memory, like how, with no accuracy degradation.

And we'll be talking some Triton, OpenAI's Triton language and stuff like that. And we'll be doing finding and fixing bugs. So this will be a constant phenomenon and theme. How do we find and fix bugs in Llama, Mistral, Gemma, Phi?

We'll be talking about Mixture of Experts as well. Oh, wait, maybe not, but it depends on time. And we'll be doing lots of bug hunting, bug fixing, and more. And everyone here will be a fantastic bug hunter and bug fixer.

And we can essentially open-source our effort to fix open-source models to everyone here. Oh, yes, and we also have stickers. Yes, I don't know where they are, but oh, yes, yeah, my brother has some stickers. And we bought a few of these stickers, which look pretty cute,right?

So you can—wait, my laptop has some,right? I put them on my laptop. And they're pretty cute. I really like them. So my brother has them. We'll be handing them out, yeah, as well at the end.

Okay, so let us start. So the Transformer,right? So what is the Transformer? I'm assuming everyone knows what the Transformer is. Does anyone not know what the Transformer is? Yes or no? You can simply.

I will.

Okay, yes, okay. So the Transformer is just an architecture that is behind all language models. So like GPT-4, GPT-3, Llama, Mistral, Gemma, all these open-source models, what are they? What's the architecture behind them? And all of them rely on the Transformer.

And the Transformer is essentially an architecture, which seems to be very good for sequence modeling. So it's not just for languages. It can be for any sequence modeling,right? So if you know Sora is a Transformer—well, not just a Transformer, probably plus diffusion, but it's generally a Transformer.

And there's other different types of models, which doesn't have to be language modeling. Okay, it's just sequence modeling. And I probably should send some pictures later. I probably should have explained it a bit better, but just assume that Transformers are the method behind all language models.

Okay, GPT-4, GPT-3, GPT-5. Okay, I don't know if GPT—okay, who knows what GPT-5 is, but I'm assuming it's a Transformer. The Transformers just seem to be very good at learning new knowledge, injecting knowledge into the model. It seems to be very, very good at changing the weights to fit the training data, which is very interesting.

And the GPT-2 architecture was actually very popular for most decoder-style Transformers. That was very, very popular. It's still used to this day. And it kind of got reincarnated by adding extra components to it. And this new architecture is called Transformer Plus Plus.

I don't know if people have heard of this, but Transformer Plus Plus is the GPT-2 Transformer architecture plus RoPE embeddings plus Wiggloo plus RMS layer norm, and with no bias. And I think it's untied weights or the—I'm not sure if GPT—I can't remember that exactly, but I think it's plus untied weights.

And Transformer Plus Plus is the architecture which most people think is the best Transformer architecture for now. Yes, for now. There are probably some other tweaks and little small things that Transformers still can do, but in general, this would be considered the best architecture.

And how does the architecture look like? It is just a list of maths equations. So I just wrote down the entire Transformer architecture. Well, this is Llama 2,right? So Llama's Transformer architecture in one slide. And all you need to do is get some inputs, like some sort of inputs, do some layer norm, do some RoPE embeddings, do some attention, plus some residual, do some layer norm, Swigglu, whatever, residual, layer norm, and you get logits.

And you essentially repeat this middle section L times or many times. And that is the Transformer architecture. Okay, then maybe the maths equations—I'm not sure. Does the maths equation scare anyone? Or I'll be explaining each one. Okay, so hopefully I try to make the maths equations as reasonable as possible.

In theory, if you write this down in PyTorch—oh, yes, if you write this down in PyTorch, you actually have a working implementation of a Transformer architecture. And yeah, so we'll be talking about each component separately as well. Yeah, is anyone scared for the maths?

No. Yes. No. No. Okay, very good. Okay, let me just check questions. Does anyone have any questions? Okay. Okay, so did you have a question? Okay.

Well, so from my understanding from the layer level for Transformers, so it's almost comparable to cosmic Planck in a way. I mean, of course, there's the hard math behind it, but.

Cosmic, sorry, what?

Cosmic Planck, like the drop, because there's 80 layers,right, in a grid,right?

So like low-energy something? Sorry, I'm actually not familiar with that. You have to explain to me. What did you say? Cosmic what? Sorry.

Cosmic Planck.

Cosmic Planck? Do I know that? I'm not that smart. So you have to explain to me what that is. Should I search that up?

Not gotcha. It's the arcade game with a bunch of pegs and you drop a ball.

Oh, that. Oh, yes. It's a Windows XP.

Output on the bottomright to visualize it,right?

Yes.

If you take this and visualize it, and then visualize the math, for example, let's take Llama 3 80B,right? It's 32 layers,right?

Yes.

And then layer zero, or I guess it doesn't matter. There's layer zero and then layer 32, which is the output layer.

Yes.

Now, when you drop a prompt into this cosmic machine, which it's not actually cosmic. We just don't understand.

Okay, yes, it's not cosmic. Yes, it's just maths. Yeah.

But so far, it's cosmic to us.

Yes.

I don't think we understand. Personally, I don't think we understand 50% of it yet.

I know. I think we've said it a lot, but okay, yes. Okay, it's just maths.

So you're the expert, not me.

Okay. So you're just trying to say an analogy, like it's kind of like the game.

Yeah, so if you look at visualize the math versus.

I think.

A fair comparison.

Oh, yeah, I will talk about that in the slides, but I think it's more like an analogy. I think it's more like you're going through a maze. Okay, it's not a maze. I would say it's more like you have every single layer has someone trying to make you change clothes.

And then each layer, there's a fashion designer trying to get you to wear different clothes. And each layer, the fashion designer doesn't like the previous fashion designer's choices, and they will change your clothes. Something like that. I think that's more like a Transformer.

It's like each fashion designer has views of their own.

So it's actually quite similar.

Okay, yeah, I got you.

Go does have kind of pegs on it.

Oh, yes, yes, yes. Wait, it is in Windows XP,right? Is it that? I'm even confused. Windows XP, like the game? I think I played it before.

It does.

Oh, okay. Anyways, okay. Yeah, sorry. Yeah, question.

In the notation, is there a way to tell what's a matrix or a vector or just a?

Oh. So when I put subscript i, it generally is a vector. Well, technically, everything is a matrix, but if you see any submission signs with subscript i, then it generally means row-wise or column-wise. The W, if you see a small W, that generally means vector.

But in general, everything that's capital is a matrix. And why is it a matrix? Because it's just faster. I mean, in theory, you could convert this all to vectors, but for speed purposes, this should be matrices. Yeah, any other questions?

Okay, next. So why do they put this? Hello, my name is Daniel. Hi, my brother's name is Michael. I hope everyone will have tons of fun. AI Engineer World fairs is the best. Why did I put this? Well, does anyone notice any similarities between these sentences or differences?

Exclamation mark at the end.

Sorry?

Exclamation mark at the end.

Okay, yes. Okay, yes. Okay, except for the first sentence. Okay, anything else? Just say random stuff.

I mean, hello and hi are the same thing, just kind of different.

Interesting. Yes, okay. Hello and hi are the same thing, but kind of different words. Okay, yes. Yes.

Semantic embeddings. Basically, this is an example of semantic embeddings.

Okay, okay. Somewhat. Okay.

Verbs are all similar. Is, is, will have are all to be verbs.

Okay, yes. Okay. Okay, well, to tell the truth, I didn't have any intention. The intention, yes, go on.

I mean, it's the king plus queen.

Yes, yes, yes. For word to fact. Yeah, yeah. I know what you're talking about. Yeah.

So it's king minus man plus woman equals queen. I think that's what you're trying to show a little. I mean, maybe you don't have that intention, but this kind of shows that a little bit.

Okay. Well, in theory, I guess you could have just seen the next slide. There are just five. If you simply just look, the first one, if you consider punctuation as combined with the word,right, like hello, comma, treat that as one separate component.

And if you do this, ignore all spaces, the first one has just five components,right? What do you think the second one? Okay, I guess you can already have the slides, but what do you think the second one has then?

Right? Anyone? I already wrote out the answers there. Okay, yes, I wrote all the answers,right? 6, 8, 7. Right, 1, 2, 3, 4, 5, 6, 7, 8,right? 1, 2, 3, 4, 5, 6, 7. Right, so if you do this, just assume all punctuation from now on is combined with the word and ignore all spaces.

And this, we just invented a tokenizer,right? So this is a very general, a random tokenizer we just invented. And each one has an essentially, the reason why we want to do this is because computers, they don't understand words.

They only understand numbers,right? So you have to essentially assign each of these tokens as a number, like an ID,right? So hello is ID zero,right? Hello, comma, actually, is ID zero. My is ID one. Name is ID two, and so on,right?

So you have to assign an ID to each of these components. If you don't do that, then a computer doesn't know what you're actually doing,right? Computers only know numbers. So we just invented a tokenizer. I would not suggest you to use this tokenizer, but in general, it's actually not that bad.

Can anyone see any issues with this new tokenizer we just created? What are some issues? Yes.

You include the punctuation, the tokens for Michael with and without punctuation will be different.

Yes, very good. So what do you think we should do? Okay, so the point was we included the punctuation for the words, and that is not helpful,right? So like Michael explanation mark or Michael not explanation mark. So what would you suggest to fix this?

I would assign tokens to the punctuation.

Interesting. So hello and then hello will be one token, and then comma itself will be a.

Yeah.

Okay, interesting. Anyone have any other suggestions? Yes.

So the point of a tokenizer is to reduce the vocabulary, but here you haven't reduced the vocabulary,right? Because every new word will be a new token.

Yes, very good. Exactly. So do you have any suggestions for how to improve?

You could stem the word that will reduce the vocabulary.

Oh, yes. Very good. Yes. I haven't heard that in a long time. Very good. Yes. Old natural language stuff. So the idea was to stem the word. So essentially, you can remove, for example, skipping can become skip,right? So skipped, skipping, skip, they're all kind of the same.

Well, I wouldn't say the same thing, but in theory, they're the same thing. Any other suggestions?

Well, my with a capital M are treated differently, but they mean really the same thing. So we just lowercase them all first.

Very good idea,right? If you lowercase them all, then you can reduce a lot of issues. Is that a good idea? Capital my and small my, what do you think is the difference? If you do capital my, it generally means it's the start of the sentence.

If you do my with no lowercase, then it means it's the middle of the sentence. So good idea, though. Actually, I think, so but is an old, okay, people still use but. I think there is a lowercase version of but.

So they essentially lowercase everything. And I think it does okay for semantics and stuff, but it doesn't really do well for decoder-type style. So don't lowercase, but good idea. Any other suggestions? Yeah.

You build a vocabulary by starting with just the individual tokens instead of size for how large your vocabulary would be.

Yes.

And then you start building word pieces from the most frequently observed.

Oh, you just said the name of the algorithm. It's called word piece or BPE tokenization. Yes, exactly. So that's actually the correct, well, I wouldn't say it's the correct way. I actually don't like that approach, but yes, it's the most recognizable and most industry-standard approach is to do what you suggested, which is to start off with small little individual characters and then combine them together based on some sort of statistic,right?

You shouldn't like, hello, maybe hello is a word that we have to select,right? So like, but like hell, okay, hell, okay, that might be a popular word as well in the dictionary, but anyways, or he,right? So he, hell, hello.

Each of these might have assigned different tokens, but they might not be,right? So it depends. And the correct industry standard is to use these methodologies to build up the component, which we won't be discussing today, but that's for later research.

Yes. Okay.

So let us just look at one sentence,right? The first one is, hello, my name is Daniel. Assuming our tokenization is useful, okay, let's just assume the tokenization which we created is helpful. Okay, so remember, it is just put all the punctuation together, ignore spaces, and don't do lowercase or whatever.

Just tokenize it as it is. It's not very useful, but who cares? We'll just say this is a good tokenizer.

The question now is, if I select the first token, hello,right? Let's assume, I don't know if, okay, the color is not very good, but my name is Daniel is grayed out. I think, I'm not sure if you can see that, grayed out.

Pretend someone types in hello, comma,right? A language model should predict what's the next word,right? So how does it know to predict my? Right, you know when you type ChatGPT, it goes from left toright? You type something, you type some sort of instruction, and then ChatGPT will print the words from left toright,right?

So why is it printing words? Can anyone tell me why is ChatGPT printing words from left toright? Why is it not doingright to left? Or why does it not just spit out everything in one go? Yes.

Because it's basically a sequence modeling, and it's just taking for instance, hello, and then predicting what's most likely the next word depending on the prompt you gave it.

Exactly, correct. So because it's a transformer-type, decoder-type architecture, it is predicting the next word based on the previous words. So very good. And so the point is, we only can have the language model see previous words, not future words,right?

If you accidentally put the future data in, oh, your accuracy might go to 100% if you use future data. So please do not do that. This is actually a very common mistake. I'm not saying this jokingly. It's a very big issue in research.

So please read the paper before you actually, if you read research papers, please see how they do the methodology. Always read, did they put the, always ask the question, did they put future data in when they did the training or during the research paper?

Okay, this is a pervasive problem. I'm not joking. You can see weird accuracies. If you see papers which have 98% accuracy, question, question, question, question, question, okay? Or 100% accuracy. How is that even possible? How can something be 100% accurate?

Question,right? And so the most likely scenario is they use future data. Yes.

Do you find the paper itself will actually catch this? Or I feel like I often find that the logic in the paper makes a lot of sense, but then I'm really questioning how likely their code actually implemented what they think they are doing,right?

Do you find it more like the paper actually gives you that issue or that the code has some bug in it?

That is a very good question. So the question was like, is it the research paper, like the methodology itself using future data, or is it the code implementation the problem? Now, that's actually a very good question because it depends.

So I think if the researchers, I think normally in research papers, one person does the coding, some other people do secondary coding, and then the main researcher makes the idea. They don't actually do the coding. So the coder might have misinterpreted the methodology.

To be honest, I don't think so that's the case. I think it's actually the methodology that's the problem. I think if the researcher, like the main researcher, if they find out that their method has 90%, 98% accuracy, why didn't they question the results?

Wouldn't that be very sketchy, like 98% accuracy, 100% accuracy? I'm not joking. This is actually a very serious problem. So if you read enough research papers, you'll see this problem is always the same problem. It's always using future data.

So yeah, I think that, yeah. I think if you don't, yeah, okay, I won't comment on any papers. Anyways, yes. Okay, does that kind of answer your question? So it depends. I think in general, I would say it is the researcher's, the lead researcher's responsibility to correct these mistakes.

Then you shouldn't be the lead researcher. So that's my take on that. I think the coder, I mean, the programmer might have some issues, but I don't know. I think I blame the lead researcher. Any other points?

It seems like you're saying the main red flag is a huge leap in performance.

Yes. If you see huge leap in performance, question, question, question, it's probably future data. Actually, I wouldn't say it's probably. It's like 50% sure it's future data. Yes, question.

Do you think in methodology, they do a train validation test split, and what if the weight is sort of like deducted and you shuffle the data?

Oh, yes.

The tokens, like whatever we are calling future data, what if the tokens, name entities, and stuff, they didn't actually shuffle that,right? And that was all in the training data and the test data or the validation split, whatever that, that just essentially have tokens that they've already seen, so it's able to do a good job at 98%.

Exactly. So the point was, in the test and train split, sometimes, although you might not be using future data, you might accidentally put different components in the training set from the test set. And depending on how you split the data set, you might actually mix the data sets.

Is that kind of correct?

Yeah, it's like basically, are you biasing basically your.

Yes, the distribution itself.

Yes.

Yes. So that is why you have to be very careful to how you split the data sets,right? For training and testing, you have to use stratification. You have to inspect the data before you do it. You must enable random shuffling,right?

There is actually a very, very common question on PyTorch. How do you not enable random shuffling for Hugging Face? How do you not enable random shuffling for Hugging Face? They purposely disabled this. Think about why, because people might forget to randomly shuffle.

And actually, this is a pervasive problem in Kaggle competitions. People like to train on the tests. They give you 20. Normally speaking, you should have 20 test, 80 train,right? But people like to, the final submission, they use even the test in the submission.

So it is a pervasive problem. Any other points and questions? Okay. So what is a language model? Given the word hello, can you somehow predict my name is Daniel,right? So given the word,right? So given that token, can you predict these extra tokens?

Remember, carefully, I purposely only went from left toright,right? So hello can only predict my, can only predict name, is, and Daniel,right? It cannot predict,right? You can't use the word hello to predict previous words,right? That's cheating. So that is sequence modeling.

And the point is, when you chain them together, let's assume that you predicted the word my as the next word,right? So now you have two pieces of information. The text that you see is hello, comma, my. Using these two pieces of information, you want to predict name is Daniel.

And you keep doing this. And that is a language model,right? That is essentially a language model. What is it doing? Is you start from the first word, you predict the words into the future, and you keep doing this iteratively,right?

And so that is what ChatGPT is doing.

Does that kind of make sense? And remember, the point is never use future data. Okay, I know I keep stressing this, but this is actually a very big issue in machine learning and AI,right? This is actually the biggest issues I find in my opinion is using future data.

There's so many papers which do this. Yeah. So the second point is you must tokenize each component into numbers. Remember this? So I'm just going to cook up some numbers. Hello will be 0.11 minus 0.123, 102. Okay, I just made those numbers up.

Daniel is 0.11, 123, minus 0.122. Okay, I just randomly made them up. And remember, each component must have the same number of numbers,right? So if hello has three, Daniel must also have three. Can someone tell me how many combinations of, if you assign this case that each number must have three, three numbers, how many combinations do you think there can be for each token or for each component?

How many combinations? What do you think the answer is? So remember, you can choose any single number in the three numbers,right? 0.11, 0.112, 0.111113, whatever number. How many combinations are possible?

Depends on the flow of precision.

Okay, let's assume it's infinite precision.

Oh, that would be infinite.

Correct, the answer is infinity. You can do as many as you like.

But normally speaking, you should use not three. Right, now the question is, why don't you just use one number then? Hello can be 0.11. If it's already infinity, 0.11. If you use two numbers, isn't it also infinity? Right, so what's the problem?

Well, please don't, you should use as many numbers as possible. Try your best to use more numbers. It's because the computer, it's not an infinite machine. So please use more numbers so it can learn which numbers to assign it to.

And when you start training a language model, these numbers will be randomly initialized,right? So all these numbers will be randomly initialized. And can someone notice my initialization? What is the issue with this? There is a glaring issue, quite obvious, very big problematic issue.

Hello and one or two, then you can do.

Sorry, sorry. Wait, you want.

I think there's the same number for hello and Daniel.

Okay, same number. Okay. Okay, good point. Yes, did someone say something? What was the other point?

Different magnitude.

Yes, the magnitude is the most important. Right, 123 and 102 are terrible. When you randomly initialize, please do not initialize with random large components. This will destroy your training. That is why your training might have infinities, and sometimes the training loss goes to zero.

Okay, that does not mean your model learned anything. That just means it's some sort of error in your training data, sorry, your initialization. So be careful of that. Don't worry, Hugging Face does this automatically for you, so you don't need to worry.

And now each component has a list of numbers that it's associated with. Okay, I just use the same number for now. It's easier for me to do the slides. But essentially, hello, comma, my name is Daniel. Each of them has numbers associated with them.

And this is the thing that you're trying to learn for each of these components. And remember, this can be converted into a table of numbers,right? So if you replace all the commas with just columns,right, these are just tables of numbers.

And this table is what you need to train.

And again, remember, given the word hello, you want to predict my name is Daniel,right? So essentially, given that vector of numbers, can you predict the other vectors of numbers?

Just for clarity.

Yes.

The three dots are.

Oh, yeah. You can do as many numbers. So you have to select a option, how many numbers you want to select to represent these numbers. So for example, you can select six numbers, or you can select 1024 numbers, or 2048.

It depends on the model creator's choice. Yeah.

Imagine every row has to be unique,right?

Yes. Every single row must be, well, it doesn't have to. Okay, it might not be the case. It might be not, but it should be, yeah, with high probability, it will be unique. Yeah.

Wait, is it like, okay. And so when you do training of a language model, there is a trick that you use. And remember, we want to predict the next word,right? So hello, you want to predict my,right? So can someone notice any pattern with this?

Like, why did I do the arrow and what's the pattern? Can anyone see this?

Any special

things with this? No? What happens if you take hello, my name is Daniel, and just shift it up by one place? Is itright? If you shift it up by one place, hello is now aligned with my, my is now aligned with name, and so on,right?

And there'll be a gap at the very bottom,right? So simply, we just put eos, which means end of sentence token. That just means it's the end of the sentence. We just put it there because it's a gap,right? So remember, machines do not like gaps, and you must use all numbers.

So that's the reason why we did that. And this is kind of the training mechanism,right? So essentially, we have that list of words, and we want to predict the shifted words.

And the transformer, all it does, it's a function to predict that. So given hello, can you predict my? Given my, can you predict name? And so on,right? That's the transformer architecture. It's kind of a bit wrong, but something like that.

Okay, the FX is like this gigantic model that can be like, there's lots of turning knobs in it. Okay, let's check if there's any questions. Okay.

And the point is, remember, remember the point is we can only use predict the future words,right? So hello, can only predict my name is Daniel, and so on,right? That's the purple component. And the blue box is called the attention mechanism.

The FFX, I factored it out, and that is called the multi-layer perception, or MLP layer,right? So there are actually two components in a language model. One does the prediction of the next word, and the FFX, which is the MLP, just does this changing component.

It just makes it a bit better than just simple predicting the next word, the attention. Oh, okay, there is a question. Oh, I didn't see.

Oh, okay. Wow.

Yes.

Yes.

We are here. I know we are focusing on the most canonical approach when we are predicting one token, but you've got papers now on predicting multiple.

Yes, yes. The Facebook.

Comment on this. I don't want to go into the mechanism, but how.

So the comment was, normal transformers predict only one token at a time. How about transformers which predict multiple tokens at a time? Yes, so actually, you can predict multiple tokens at a time. It just depends on what is your training objective at the very last layer.

You don't have to, remember, we shifted by one place,right? Why don't you shift by two places and three places? Then you'll be predicting two tokens in the future, three tokens into the future.

Yeah.

Exactly. So before, it's just one objective. So like one column of the output, we just add more. And yes, you could do multiple tokens. Was it the, it was the Facebook paper,right? I can't remember. Yeah, it was a Facebook paper.

I forgot the accuracy though. Yeah, but yeah, you could do that. But I guess I don't see any, I guess it's just good for inference time. Like you can predict, you can make inference, if you do four tokens in the future, you can predict four times faster.

Also, I think they managed to

scale model down a little bit.

Okay, yeah. I think it's mainly for when you do inference, you can do this. If you do two tokens in the future, you can have like two tokens in one go. I don't know. I don't like that approach.

I think predicting one token is better because you're already forcing the language model to do so much. You're making it even more problematic. So I think predicting one, okay, maybe for inference time it could work. Yeah. Yes, question.

So for the multiple dimensions, is that basically the, for each token, is that correlated to the hidden dimensions in the layer? And because tokenizers do convert it into a single number,right? Like you actually use natural tokenizers.

So token, yeah, tokenizer converts it to ID. So like.

ID, yeah. That represents then the dimensions, and that's going to be equal to the hidden layers, or it's going to be.

So the tokenizer, when it has like, when it says like it has 32,000 words in the tokenizer, essentially it's an ID from 0 to 31999,right? So like hello, we'll have ID 2557. So then what you do is you take 2557, go to the hash table, which has this, which has this table,right?

I made the table,right? So hello has an ID, my has an ID, name has an ID, and so on. And you essentially, you hash, you go to that specific row, and then you, that is your sample in your training data.

And you do this like for the whole thing. So the tokenizer does, you do get a number. It's an integer. And you just have to hash it to the embedding matrix, and you'll get like a vector of numbers.

If that does that kind of answer your question?

It will correlate to like basically a 2048 or 6144.

Oh, you're talking about the embedding dimension? So if you see 2540 or whatever those numbers, that is just how many columns, how many numbers you want to represent for each component.

Okay, that's the question I had.

Yeah. Okay. Any other questions?

Yeah, yes.

I have a ton on the, I have a ton on the.

Oh, okay.

So you can answer them way later.

Okay, okay. Yeah, yeah, you had a.

The language is following up on that. I never understood what's the significance of language? What is the difference between 2048 versus.

4096?

Yeah.

I guess it just depends. Like if you make, so the point, so remember we said how many combinations can we do? Infinity, because we can do,right, because floating representation, in theory, if you have infinite position, it can be infinity.

But someone mentioned how depending on the position of your float, it's actually limited position,right? So like the point is you want your model, you want the training objective to actually learn, give it as much freedom as you like,right?

So like if you're trying to restrict the model when it learns, it might actually not be helpful. And that is why normally people have like these large numbers, like 6144 embedding dimension or 8192 embedding dimension,right? The more numbers you give it to the model, it just has more freedom to move.

So if.

Is it better?

In theory, I mean, in theory, it should have better accuracy. In theory, everything's in theory. I don't know if there are research papers to show this. I think someone should write a research paper on that. Each embedding dimension, test, do 3 trillion tokens.

Okay, that's probably too many. And then see which one has, I'm assuming the more you add, the higher accuracy. Yes.

Well, I mean, people used to do like one-pot of coding where your embedding layer would be the same size as like your vocabulary. And I think that like a denser representation does work better. So I think there's like a sweet spot in somewhere in the middle.

I don't think as you get larger, it's just always better.

I agree on that. I think someone needs to do a research paper. Yes. That should be a new research topic. I've never seen this paper before, so like that would be very interesting. So yes, question.

I actually have a question. I think questions as well. Is there anything, or sorry, is there any reason why you'd add unused tokens like Gemini did or Gemma models in the beginning of the program? There's a bunch of.

Yes.

Unused tags. What's the reason behind that?

The reason for that is it just makes training faster sometimes. So depending on, so there is, I think Andre was the one who tweeted about depending, if you pad the token, if you pad the vocabulary to a specific number, you can actually make training faster.

Because in the NVIDIA GPUs, when you use Tensor Cores, if you pad it correctly, it can get the data and cache it more appropriately. So like, for example, the cache size is like 64. I think it's 64. Okay, maybe I'm making stuff up.

But like essentially you have to pad it to a multiple of 64, something like that. And so sometimes that happens. Another one is like some people want to add, when you want to do more fine-tuning, when you want to like train the model for more, you want to use one of those unused tokens for your own purpose.

And so they left some holes in there. Yeah. Does that kind of answer your question?

I was wondering if it maybe caused any weird issues with the model, if you hit those on other occasions or.

So when you do tokenization, assuming you don't encounter these tokens, you won't have any problems. But if you do, then there are problems. Yes. So for example, if you do Llama 3 fine-tuning, if you use the base model for Llama 3, and you accidentally use one of those tokens, you will get NANS for fine-tuning,right?

So you have to be very, very careful. And so like I think what we did is for Unsloth, we actually find these untrained tokens first, set them to the mean of all the embeddings, and you won't have these issues.

So I think that's actually a model creator's problem. It's like they probably should have not set it to zero. I don't know why they did that, but anyways, yeah, they should have set it to like a normal distribution or like some just random initialization.

Yes, this is kind of, yeah. Okay. Any other questions? Okay. Oh, yes.

Yeah, just to go off, why was there an eos and not a vos in the beginning? Like what if I want to, you want to learn that hello is like for most sentences, they start with hello. So why don't we have a beginning of sentence token?

Oh, yeah, yeah. You can put a beginning of sentence token. I just didn't do that. You should put a beginning of, most language models will put a beginning of sentence, like what is, I put the end of sentence, you should probably put a beginning of sentence as well.

That is actually very important as well. Most models do that now. They found that to be very helpful. To be honest, I don't think so it's actually that effective. I think the beginning of, so the bos token came from the old style, the old style, the CLS token for, I think it was the first token for like BERT style.

So like they had a classifier token at the very start. I think it was at the very start. I'm not 100% sure, but I think that's where it came from. I don't think so the beginning of sentence token actually makes that much difference, but you should put it.

You should put it. Giving the model more freedom to move is always better. So yes, I probably should have put the beginning of sentence, but yeah, for demonstration, I did not do that. Yes, okay, we did that. Right, so like the green one,right?

So like the attention block is kind of encoding the stuff that we described,right? Predicting the next word based on the previous words,right? And so like the attention block is that first part. The MLP block is just the mixing component.

And this is kind of the transformer architecture and kind of like in visualized. And you just repeat this L times. That is a transformer. Oh.

Now, another question I always have is, why is training language models not O of n cubed? Because like aren't you like, given the word hello, you're predicting my,right? And now we have hello my, you're predicting name. And then you have hello my name, and you're predicting is, and so on,right?

Shouldn't this be the training data? Why is the training data just hello my, my name, name is, is Daniel, Daniel eos,right? This is the training data that you actually see. Why is it not this? Does anyone know why?

Complexity?

Sorry?

The complexity?

The complexity?

Yeah, so it will be.

Yes, the complexity, yeah. Very bad complexity,right? So like if the sentence is like one, okay, but if the sentence is like 100 words, what do you think? How many?

Quite bad.

Quite bad. Yes, basic one. Yeah. So like one plus two plus three plus four plus five all the way to plus 100,right? So like n divided by two, one plus 100, I think. I can't remember my maths, but yeah, something like that.

So it ends, yeah, it's very bad. And that's if you have one sentence. If you have like 10 sentences, oh my. Yeah. But like, does anyone know why language models don't need to do this? Like we don't actually need to do this,right?

So like we can skip, essentially, instead of having this as the training data, your training data is simply my name is Daniel and shift it by one up. And that's your training data. Why is it not this?

Is it position encoding?

Oh, yes, we haven't talked about position encodings yet. Yeah. Okay, but you actually, you don't need position encodings.

Attention.

Oh, okay, yeah, attention. Oh, yeah, the attention mechanism.

Yeah, because of attention.

Oh, correct. That's the answer. Yes. It's because of attention. We'll actually specifically mask attention. So that's a trick. Okay, we'll be talking about that. We'll be talking about it a few times. And I'll give you the code again.

Well, actually the maths formulas for transformer architecture,right? So like attention block, we will be now talking about the attention block,right? So like the Z is equal to the softmax of QK transpose over root H plus MV. And as you mentioned, it is the attention mechanism which allows us to skip the O of n cubed complexity and make it O of n squared.

Why? Because remember, we want to mask out future tokens because we don't want to predict on future data,right? So like by using this mask, weirdly, this mask allows you to train more efficiently. It's funny because like attention is O of n squared, so the longer your sequence is, the worse the complexity.

But actually, there is a special trick, which you use a mask, and this actually makes attention not that bad.

So instead of doing hello to predict my and so on, so on, so on, the attention mask acts as this methodology,right? So the attention mask itself acts as you don't need to do like all the complicated, all of the words to predict the next word.

Okay, this is, okay, probably should have. So we'll be now talking just about the attention itself,right? So like softmax QK transpose over root DV. Just a reminder that whenever you see QK transpose, the query and keys, I do not like to like, there's like explanations like what is a query, what is a key.

I do not like that actually approach. I would like this to be a maths approach. So my view is, given the matrix X, which is your embeddings,right? So remember, hello is a vector of numbers,right? You multiply this by some weights, WQ, WK, and WV, and you get back QKV.

Q is query, okay, keys and values, but that's a very vague interpretation. I don't really believe, like I don't really trust those interpretations. It's not that clear. Just assume it's just maths. Okay, just like get your X matrix and multiply by weights and you get some extra weights.

That's my view. And so that is kind of, so like if you see why I stacked it like this, does anyone know why I stacked it like this? Like why did I present it like this specifically? Why is it like the presentation like this?

Any points?

Do you want to put the composition?

Sorry, what?

Do you want to put the composition?

Composition?

Decomposition.

Decomposition. Interesting. Okay, that's a very interesting point. But no.

You're just lining up the dimensions of the multiplication?

Correct. I just, yes, that's correct. So I just lined it up such that it's easier to see. And if you take the matrix X and you multiply by WQ, you will get Q,right? And this is actually the correct maths dimensions and stuff like that.

And so like I like to normally tell people to like visualize transformers as maths. It's actually, in my view, it's easier. Okay, I'm not sure for other people, but my view is easier. I do not like it when they see, oh, queries and like you're trying to do keys and values.

I don't even know what that even means. Anyways, and the yellow components are the ones you want to train. X is what you want to train. WQ is what you want to train. WK and WV. And QKV are just the components afterwards.

When you have the, so remember, you have the Q, you have the K. All you need to do is when you do K transpose, you transpose the matrix and you do Q times K transpose and you get this big square matrix called QK transpose,right?

Hello my name is Daniel and so on,right? So like that, that's kind of what I want to visualize is like, it's actually a, when you do QK times K transpose, you get a square matrix.

And all you need to do now is do the softmax divided by root D,right? So softmax, essentially each row you normalize to one,right? The sum of the exponentials must be,right? You need to like normalize them. Does anyone know why you should do that?

And why should you use softmax?

Any clues? Why do, yes.

It's not correct.

Yes. Okay, that's the answer. Yes. But like why? Like why?

Because when you multiply them, you can get LNAND?

Sorry. When you multiply them, you can get NANDs?

No, you can get LNAND, you can get a bit of exponentials.

Oh, yes, very good. Oh, that's, okay, do you know how to fix that?

Well, that's one of the softmax, isn't it?

Close.

Again, I know how to fix that.

You have to minus the maximum of the row. That's how you fix it. Yes.

Yes, we're going to sample from that distribution.

Oh, yes, very good. Okay, yes, we're going to sample from that distribution. Okay, sample from that distribution. But what happens if you don't do the softmax? Doesn't this still work or not? Like what happens if you just do QK transpose over root D?

Remove the softmax. Like why do I have to do softmax? Yes.

But is the exponent gradient?

Interesting. Then you can fix that with like minus max of the row as well. We've exploded. Anyone else?

Maybe have a nonlinearity then.

Okay, what happens if you don't have a nonlinearity then?

Constrain it for like—

So it doesn't have to be softmax? Can it be something else?

Theoretically, it could be something else.

Yes, it could be. Yes. That is another active error research which people should focus on, which is like, why do we need to use softmax? Generally speaking, research papers show that is actually the most accurate. If you use other activation functions, it might actually not be that accurate,right?

So like, but this also is the bottleneck of transformers. It's because it's a softmax, it does the row sum of the exponentials. This means that you can't actually decompose this,right? You can't actually bring the matrix multiplications out. And so if someone can find ways to make this faster, you'll get like millions of dollars.

Okay, maybe like much more than that, but yes. And V is just, remember, the V comes from here,right? So we just take the V, multiply it up again, and we get this matrix at the very end. And that is,right, oh yeah, that is the final component,right?

This empty box is what you get out from the attention mechanism. For the layer norms, I don't really want to explain too much, but the layer norms essentially, you take the, you square all the elements per row, you sum them, you divide them by the square root, and you take the mean and just do one divided by,right?

All this does is just normalizes the rows to make it easier for the language model to learn,right? So like why do people do layer norm? It just makes training easier. It's more stable. There's no other like, there's no other like point.

There are like some theories like batch normalization, like you know, out of distribution, you want to make like shift towards the distribution for out of distribution data. I just like to think of this as an optimization method. Layer norms just make training easier and more stable.

And layer norm is simply, remember, as I said, as you take the X matrix, you do a row sum of all the squares and you take the mean and then you just divide it and then you multiply it by some weights.

It's a vector of weights. And that's just layer norm. You don't worry too much about like what is layer norm or what it does. It just does training better, more stable. Please add as many layer norms as possible.

Yes, add everywhere. Layer norms everywhere. And you'll make training much better.

Okay, I probably, okay, I don't know if you can see this. But in Triton,right, in order to write Triton code for the layer norm, this is the forward kernel. We will not be talking about Triton today, but it's actually not that complicated.

If you read more intently, ignore all of the like components, there is only very few lines for the layer norm. It's actually not that complicated. The rest is simply just how to load the data. It's actually not that hard.

Yeah. The backward kernel is when the problem comes. How do we actually do the differentiation of the layer norms,right? Remember, you have to train the W,right, is yellow. You actually have to train that. How do we find the derivatives of the W?

It is very complicated. And if you want to learn in your own time, you can have fun learning the derivatives. It is extremely complicated because there is like sums, there is like, row sums. How do we do the derivative of a row sum?

It can get quite complex. I wanted to talk about backpropagation today, but I thought like it's probably too heavy maths. So no backpropagation. But we'll be showing, but I do have tutorials on that. So if you go to the Triton tutorial, I followed that.

That's actually quite helpful. And the backward kernel is just very, very problematic. Now, up to the rope embeddings. Why do we do rope embeddings? Does anyone know what is the rope embedding? Yes?

It's a, well, it's a way to extend, I mean, context,right?

So you could use the rope embeddings to extend context, yes.

Okay, that's not what you—

Do you know how? How does that extend context?

So, well, how does it work? Versus yarn or like—

How would you use rope embeddings to extend context? What would you do?

How would I, how would I do that? I would create a basically, what I would do is I would basically create kind of a—

You just multiply the base by two and then you get two times longer context. You multiply the base by 10.

The problem is, if I'm looking for like 1 million context,right? Then the model, the part of it is trained at like 400.

Correct.

Sorry, 4K,right?

Yes.

So that's where yarn might kick in instead of like—

So is that the dynamic, dynamic?

Well, that's static or dynamic either way,right?

So how would you solve the problem if it's like you want to train, if you want to have 1 million context length, but your data set is only 1,000 words, how would you solve that problem? How would you think of solving that problem?

Because like some people have said they do 10 million context length. Is there any data sets which is 10 million tokens? How would you—

15 trillion tokens, fine wine. So we have that.

Oh, no, no, but that's 15 trillion tokens for like the data set. I mean, like how do we do long context? Remember, when you do long context training, you have to have a document which is at least 10 million words for it to learn how to predict the 10 million plus one token.

So how I would solve the problem would just be to gather better and more diverse data sets, but—

Yes, that's the—

That's the ideal.

So what happens if there is a, there is no data set which is 100 million tokens? Then what would you do?

If there is no data set that's 100 million tokens, I would

synthesize.

How would you synthesize if the model, it's like a chicken and egg problem.

Huh?

How would you do synthesis?

So, no, no, no, I would basically just create, I would basically use like Claude or like any of the state-of-the-art models with like LoRA and then get, and then basically turbulate the data.

But are they trained on 10 million tokens?

Huh?

If the model itself wasn't trained on 10 million tokens, does it do long context?

So if I was to try to solve this problem for like a client, for example, like let's say their code base is in 10 million tokens or, and they want a 10 million something context or whatever,right? Then I would basically like create a like—

Synthetic data set?

So not synthetic, but a derived data set from what we have.

Okay, interesting.

So assuming we do not have, but I can't assume that we have no data,right? So like—

Good point. Okay. I don't know. I think it remains to be seen. Like many claims by companies, 10 million contacts, 100 million contacts. I question, question, question, question, question.

Well, I've only seen 1 million actually work. So, I mean, and that's bringing attention in theory,right?

Okay. Okay. Now we're going into— okay, yes. Okay.

Sorry, apologies.

Okay. No, no, no, that's fine. I was asking the questions, but okay, wait. The question was like, what is a rope embedding? Someone did mention like positions. What does that actually entail? What do you think is the point of a rope embedding?

All it does is you want to tell the model to learn what is the position of the words,right? So like, hello, my name is Daniel. It actually has a meaning like hello is like the first token,right? But then if you put hello as the third token, what's the difference?

There is a difference,right? So like depending on where the word is in the sentence, it matters. So the whole point of embeddings, rope embeddings, is it tells your model to learn where is the position of this component. And old style, they use absolute, like relative, like absolute positions.

Rope embeddings does like some special tricks, like times a cosine, times a sine, and does some sort of like special rotation and stuff like that. The paper found that if you do rope embeddings, it actually has high accuracy.

And everyone does rope embeddings now.

So, yeah, why do we need the probability cases then if we already have the positional information in the embeddings?

You mean lower case, sorry, the position?

At the very beginning, you mentioned that there is a lower case version of BERT.

Yes, there is. I think BERT did not, I don't know. Did BERT use rope? I don't think so. BERT used absolute. Yes, that's the problem. I think BERT used absolute. I think, I don't remember anymore, but—

Rope didn't exist at that time.

Oh, yes, yes, exactly. So rope did not exist. Yeah. And so like this paper, the ROFORMA paper shows, so previously people used absolute position encodings, which simply just adds a position. Like you can literally just add, like if the position is zero, just add zero.

If the position is one, add one. If the position is two, just add two. That's literally what they do. Well, actually, well, not exactly, but like, you know what I mean,right? You have to divide it by some sort of normalizing factor,right?

If the position's 30,000, don't add 30,000,right? You would like destroy training. But that's kind of what they do. And what they show is if you do rope, you can essentially increase accuracy somehow. And we just use this as gospel.

We just treat this as true. Everyone uses rope now. Yeah. Yeah.

In that case, do you have an opinion on yarn versus rope versus, and yarn diarization?

So yarn is kind of rope. So yarn just does, I'm assuming yarn is rope, but it does like actually, I don't think it's actually common at this because I'm not an expert on that. Doesn't it just like, it does, yeah.

I mean, not as, so I'm no expert on, obviously.

I'm not an expert on long context.

On rope versus yarn, but since VLM only supports static yarn, unless you're planning on trying to go to 1 million context, it's actually kind of terrible.

So is yarn the one which it does like the base, like randomly changes? Like if you have position, if you have like up to 1 million context and you do 1 million and 1 context, the base changes with that, like the factor changes.

Is that yarn? Dynamic changing. Yeah.

So and that's the issue is that in like let's say PyShop,right? That's not, a short PyShop,right? That's not anywhere close to 1 million context. In theory, or maybe like let's say a three-shot or two-shot. Like, so, but obviously dynamic yarn in theory could fix this rope issue where like we just take this rope as gospel.

Yeah.

Where, but are you following kind of my question here?

Yes. No, I think dynamic yarn is just rope though. Like, okay, that's weird. Okay, maybe I'll unplug and replug. So like the screen kind of went away. Let me just do this again. Is this like, is it, is it, is it not working or is it like screen or

no screen? That's weird.

Can you answer questions while we're at it?

Oh, yes. Okay. Yes. I'll answer some questions.

Allright.

Yeah. Anyone have anyone else have questions? Wait. Okay. Wait. I need to refresh the, oh, if anyone has like take a break, you can take a break now if you want. And if you have like other questions, yes.

Okay. Question.

So I don't know yarn or what yarn is, but I've seen some of the discussion there's like a separate context window of like rope and like yarn thing.

Sorry, sorry. Yarn is.

Yeah, so I don't know what yarn is, but I was hearing some stuff about like extending the context window and I'm just wondering that like what is the issue with like rope when it comes to like the context window?

Why isn't it like not just have like a completely like a very big context window? Is there a problem with like encoding that?

So the question is for context windows.

Like what improvement is like yarn on like rope? Like what is like.

Oh, okay. What is the improvement? Okay. So the point of yarn, like what we were talking about is like how do we make a language model learn, how do we make it do long context without training on long context?

Kind of. Yeah. And so like what yarn does is you can essentially extend the context window automatically by dividing the base of the rope embeddings. You change the scale factor. I don't actually have slides for this, but it's just a methodology which allows you to scale the factor and you essentially magically make the model learn new context, long context.

That's kind of what yarn does.

And you originally wouldn't train on the long context if it was expensive?

Yes. It's extremely, if you do 1 million context length, then your O(n2) is 1 million squared, which is horrible. There are like some other methodologies, like they want to do like linear transformers and like, yes, I guess you could try that, but I don't suggest that.

Yeah. So hope, yeah. Yeah. Sorry. Question.

I would be very interested then on your comment for the fine-tuning versus in-context learning from a low-level perspective.

What is the difference between in-context learning and fine-tuning? Or what's the benefit?

How does.

That is a good question. So fine-tuning changes the grade, you have gradients for fine-tuning. So you up, yeah.

I'm less interested in explaining how it works. It's just.

Yeah.

Your opinion, what would you do when?

I think it depends. I think I would do in-context learning first. So if you have like few-shot prompts, you shove it in to see if it works. But I would still resort to fine-tuning if you want to be more efficient.

And if your model doesn't seem to be learning, then you have to go back to fine-tuning. There was a paper which was released yesterday, I think. Was it yesterday? It showed that in-context learning is very useful and it like learns kind of like how to be like a random forest or like a tree.

I think it was yesterday, the paper. That was quite useful. Very interesting. But I think fine-tuning is still very important, especially if your model is not learning anything and it doesn't seem to be working, then you have to like use fine-tuning to change your behavior.

I don't really have a comment on this. Like I just feel like you should do everything and try everything. Yeah. Any other questions?

Well, I have a couple on the app. I don't know if.

Oh, did you, okay. I think my app kind of glitched.

Do you want me to read them off one by one?

Oh, maybe I'll take, okay, wait. Yeah, wait.

I'm sorry I asked you to.

Okay. Okay. Maybe, okay. Let's just, I'll continue and then we will do the questions.

Allright. It's back. So we're good.

Yeah. Okay. I probably have to, okay, I don't know. Okay. I did not see time. I've been speaking to, okay. Anyways, that is the rope kernel and it might look horrifying. It's actually not that bad. It's literally just the formula that it did, Q times cosine plus Q times the rotation matrix times sine.

And that is just rope. It's actually not that hard. Just the code is a bit more annoying, but it's just like moving the data, it's just data moving and data, moving data and stuff like that. All of the code is just related to data movement.

So not that complicated. The most complicated part is the derivatives for rope. And you have to use something called rotate half, which essentially rotates half of the, so okay, just read the code. It's like minus X2 concatenated with X1.

So you're like, you essentially take the matrix X, you divide it by two, you take the first half, you put it on the second half and you switch the ordering and the first half becomes minus. And the code generally is reasonably well, hopefully, for understanding.

But the question is, and so you're like, this is Hugging Face code,right? So like Q times cosine plus rotate half Q times sine. The question is, how do we actually find the derivatives of this? This was actually a very complicated phenomenon because I could see many implementations not doing this correctly.

And it is very special, the derivative. Simply, if you notice, is rotate half, the function is literally a matrix multiplication,right? It's Q times R, where R is a rotation matrix and the rotation matrix is minus identity and identity and zeros on the diagonal.

And if you do this, because it's a matrix notation now, simply the derivative is the transpose. And so if you do the transpose, the minus sign just flips. And if the minus sign flips, it's literally the same as your previous example, just for the minus.

Okay, I probably like explained this too quickly, but the point is, if you do matrix multiplication, you can derive derivatives very simply. My suggestion is shove the derivatives in Wolfram Alpha or like your favorite tool. You can use ChatGPT as well and you will get the derivatives back.

But you must put it in the form that's useful for the computer to see.

Now we've been talking about the MLP component,right? So we completed the attention, we completed the rope, the layer norms. The MLP is just a mixing component,right? So it's an activation function times some weights, multiply some of the other weights and stuff like that.

All it does is it mixes the signals to make it like more fancy. In theory, you don't actually need the MLP component. Like most attention, you don't actually need this part, but you must put it for the model to have more freedom to learn.

And the famous paper, GLUE Variance Improved Transformer,

very famous author, I'm assuming most people know him, but he showed that if you add GLUE, Swigglu, and all these other variants, you can actually increase accuracy. Once again, this is just treated as fact in the machine learning community.

We should be doing more experiments than just using this methodology, but we just treat it as fact. And this is in Transformer Plus Plus, the architecture which everyone uses. There is a very big difference though. In GPT-2, they don't use the GLUE variants.

They simply just use a normal MLP,right? So X times the weights up, do some sort of activation function, and then you down-project it. So the weights down,right? So that's, but then if you do Swigglu and these new variants, the GLUE variants, you essentially add this component where you do element-wise multiplication and then you do a down-projection.

So it's actually very similar to the GPT-2 architecture. You just add an extra component. And so I try to like, there's also like a naming change, like up and down, up and gate and up, like changes and stuff like that.

But in general, you can see it's very similar, just the extra element-wise multiplication component. Yeah.

But there is like a new, the Newmatron paper, for example, oh sorry, the Newmatron, the new model by NVIDIA, did not use GLUE and instead they used squared value,right? And they showed that you don't actually have to do GLUE anymore.

You can just use squared value and it seems to do okay. Although it remains to be seen if it actually is good. But yeah. So like they showed that if you do squared value, you can remove this, you can essentially go back to the GPT-2 architecture,right?

You don't need to use Llama architecture anymore. So Llama and Mistral, Gemma, all used the second equation. GPT-2 used the first equation. You can go back to the first equation, but the trick is the F must be very special and it's called squared value.

And they showed that if you do this, your accuracy does not degrade that much. And so yes, the paper. And interestingly, if you see one of the authors, it's the same author. One of the authors' names is similar.

The name is the same. So they were also the ones to showcase a squared value. You don't need to do GLUE anymore. And you can simply just use squared value as well.

There is a research paper which I highly suggest people to read and it's called The Physics of Language Models. It is extremely long though, but it has many nuggets inside and I highly suggest people to read this. They show example, they actually did so many testing tests and experiments and they showed that if you used GLUE variants or gated MLP, it actually reduces the model's capacity to learn on small models.

Okay. So that's the point. It's on small models. On small models, on small models, if you do GPT-2, the first formula, it does better than if you do Llama, Mistral, Gemma, the second formula. Only on small models,right? That's the point.

Only on small models. And there are other special things inside the paper which I highly suggest. It's extremely useful. For example, they say that if you do, if you change, if you don't, if you, like for example, the activation function, which activation function did you use?

Should you use the sily or jelly or value or whatever? That's not that important,right? So like it's not that important. If you use like biases, it's not that important. So like there's so many different things that you don't need to do, the paper shows, but people just treat it as gospel.

Oh, we have to use this specific component. My suggestion is we should do more testing in the AI space, like which variants are the most important. But I think this paper is pretty useful.

The code for like the Swigglu kernel is, that's the forward kernels. Again, it's not that complicated to do. Like the second part is actually useless. The Swigglu kernel is literally three lines. It's just the three lines which you commented.

The rest is just data loading. How to actually load the data into the GPU. It's not that important. And if you use Torch to compile now, you can simply generate these kernels automatically and this makes your training much faster.

So that's what I suggest people to do. Just use Torch to compile. You don't have to rewrite tragic kernels. Yes.

Can you talk me through the slides back to the actual sort of equation there? And the slides?

Yeah, this one. So what's like W_gate?

Sorry, W?

W_gate?

W_gate?

W_gate?

Oh, gate. Oh, it's just another matrix. So W_gate, W_up, you train this and W_down. These are all you train them. W_down, W_up, these are all you train. They're just numbers. So like matrices, X times W_gate is like, remember the.

You should make it longer.

This thing. Right. W, this is for attention. X times W_q, W_k, W_v. Assume it's just W_gate, W_up, and W_down. And it's the same thing. So you just train these. Does that kind of answer your question or?

So is W_up. Like basically when you shift up or is it?

Oh, no, no, no. It's just a naming convention. W_up is just, it's just, it's actually called up-projection. The naming convention is like W_b, W_a, W_c. It's just a naming convention that people like to use. W_up, W_gate, and W_down.

They do have meanings. So W_down is a down-projection. Up means up-projection. So essentially you take the matrix and you like make it larger and then you project back to a smaller version. It just makes the model better to like make the model have capacity to learn.

So it's just a naming thing. Any other questions?

Okay.

As I said like before, the derivatives are always a pain. If you do the derivatives of Swigglu, it is a nightmare to do. And I do not suggest you to do this. But I had to manually do all the, you can see all my comments.

So like the comments are actually there. If you do see more carefully, I wrote it in math formulas of how to actually take the derivatives and it's extremely painful. I highly suggest you not to inflict pain on yourself by doing this.

It took me many days to do. So do not, I don't suggest this. Yes, question.

How do you verify what's in the matrix things and what are your tests?

That is a very good question. So I use Desmos. So Desmos is a graphing, graphing, online graphing calculator. You type all these equations in and then you can see does the graph align.

No, but along with all the data loading, what if you made a mistake in the data loading?

Oh, every single component, you have to be careful. So every single component has to, you have to check. Oh, is this component correct? Is this component correct? Check all of them. And so like I normally, so like you isolate each component separately and then test it.

I will talk about that actually. Yeah. Any other questions? No? Yes. Okay. And this is the cross-entropy loss kernel. I'll probably just skip this. Don't have enough time. So I wrote this as if you want to inspect the formulas and stuff like that, how do we do the derivatives for this?

You can do this. It's not that complicated. Actually, it is very complicated. I did spend a lot of time trying to like work out the derivatives. It might be a bit foreign for some people for the derivatives. The main reason why it gets complicated is when there's sums,right?

For like whenever there's sums, I just like doing derivatives when the sums is always painful. Matrix differential is actually very easy. If you do X times W, the derivative of W is just X transpose. It's very simple. But if you do derivatives when the sums,ugh, it's horrible.

Yeah, it's quite horrible. What you can do for the sum when you do derivatives, if you transform the sum into a matrix multiplication. So a sum is just X times like a vector of all ones. So that's called the row sum.

And essentially if you do this, you can actually make differentiation much easier. But I won't be talking about that. That's for another topic. And someone was talking about stability for softmax. If you minus the maximum of the row, you can make softmax much more stable.

And this is to like reduce exponentials of large numbers and then you like essentially it takes over the entire exponential,right? And so like if you do this trick when you minus the maximum of the row, this makes training much more stable.

Always do this. Yeah, always do this. Yeah. And that's the code for the forward. Not that important. Oh yes, I wrote the code for the backward as well. Oh wait, sorry, this is the forward. The forward. And it is quite long, but I wrote all of this down for your own leisure if you want to read and implement this.

Have fun. But I wrote this step by step,right? So like take Y is equal to log sum of X. Then I simplified it out. Like if you exponentiate both sides,right? You can do exponential of the Y is equal to the sum of exponentials and so on,right?

And so like I wrote this all down.

But there is a methodology which we showed in Unsloth as you can use chunked cross-entropy and this is actually very helpful for large models. Your logits are very large. So if you chunk them, you can make multithreading much better for the GPU.

And so like the problem though is like the derivatives, the forward propagation, you have to be careful now when you do chunking. So like essentially you divide these into slibbers and parallelize each component. I also wrote some maths and stuff like that for you to review.

And how you actually do the chunk sum is very interesting. The chunks, the log chunk sum of the log sum exponentials is just the log sum exponentials. There's like, you have to do some manipulation and it's actually very interesting.

You don't actually need to change that much code to make it work. Okay. Now we'll be going to the next component, which is to investigate the Llama architecture. Hopefully this works. Yes, question.

Chunking, is that something that you guys wrote as a part of Unsloth or is itright now a part of PyTorch or something?

It is part of Unsloth currently. I think I heard that the PyTorch team will be including this in Torch to compile, although I'm not 100% sure. This reduces memory. This just makes large context, oh sorry, large vocabulary sizes work.

So the biggest issue why you have to do chunking is CUDA, NVIDIA GPUs has a limit. 65536. I think that's 2 to the power of 16. Oh, they're not. I think so. Yeah. Yeah. So there is a limit.

And so if your vocabulary size is larger than 65536, you must do chunking. Yeah. You have to do chunking. So like if you're like I think, was it Gemma? 128,000? Oh, I'm getting confused. I think it's 128,000. You have to divide it into two.

And so like your chunks would be two. So it should be in PyTorch in their future release maybe. Yeah. So now we learned all of this. Now you can read the code for Llama. So if you go to, if you go to the code by modeling Llama.py,

it should be in the slides, but you can also type in Google like modeling py.

This,right? There is your, if you go to line 94, you have your root embedding, which we talked about,right? Just assume, don't worry. Oh, okay. I need to, okay, I'll probably use my mouse. Yeah. Okay. I don't really like how GitHub does the, it's kind of annoying sometimes, but I think I can disable it if I log in.

Anyways, the symbols. So if you go to this,right? This is not important. If you go to the first one is line 74, the Llama RMS norm,right? This is the layer norm kernel,right? This is the code for the layer norm kernel.

It's just this much,right? This is the, you take the row, you take the sum of the row, you take the squares of the row, like each row, you sum them, you divide it by the mean. Remember, this is the only reason why you do layer norm is to make training more stable.

And it's actually not that complicated. It's just these few lines. The rest is just bloat code for like, you have to set stuff up, you have to do random initialization, blah, blah, blah, comments and stuff like that. But that's the layer norm kernel.

The rope kernel, the rotary embedding is a rope kernel. This is just setting stuff up. Thisright, this is setting stuff up. Forward,right? This is the most important component, which is the forward component. Now don't get scared by this.

That's because we fixed a bug. So this is actually our bug fix that we did. So this is all in transformer architectures is like you have to be very careful when you downcast to float 16 if you use b float 16,right?

This is actually very important. Before, you have to be very, very careful when you use float 16 and b float 16 training, mixed position training, because it will downcast incorrectly and your rope embeddings will be wrong. So it's actually, it's not supposed to look as ugly as this, but unfortunately it looks ugly now.

Before it was just this,right? But this is just setting up code that we have to like fix the bug and stuff like that. Yeah. So it's actually not that long,right? So like it's just this. Yeah. After here. So whenever you see these architectures, it just looks complicated, but it's like, yeah.

Can you do a Torch to compile on this earlier? You get similar performance at the writing kernel that you're supposed to write?

No, no, no. So there are like some, if you do Torch to compile, we're still like I think 30% faster. So no matter.

Faster than your kernel that you wrote?

Sorry, if you use Torch to compile plus Unsloth?

So if you Torch to compile versus you use the writing kernel that you wrote in Unsloth?

Oh, we're still 30 times faster. Oh, not 30, 30%.

Oh, I see.

So we're two times faster than Hugging Face plus FlashAttention 2, but Torch to compile, they're adding kernels from multiple packages in. So yes, they could learn from Unsloth and put it in. I'm assuming they're doing that. I did have a talk with them.

So yes, they're probably doing that already. But yes, we're still 30% faster. There was like some, I think someone tested this last week. Yeah. Is that, any other questions?

Okay. This is not important. This is for, this is the one we're talking about for linear scaling for rope embeddings. If you want to extend the context length, that's kind of what they do,right? This is for scaling, not that.

There is a rotate half, rotate half part, which I was talking about,right? How do we actually derive the formulas for the gradient? How do we actually do the rope embeddings? It's just this. The MLP, which we talked about again, remember the MLP layer,right?

So there's a gate projection, the up projection, the down projection. This code is bloat again. Ignore this. It's just this. It's just one line. That's the rest is bloat. Ignore that,right? So like the down projection is just the down projection,right?

The activation function, the gate projection times the up projection,right? So that is the MLP. That is the Swigglu. One line, not all of this. That's just for training purposes. Repeat KV is just an, that's for like, okay, I won't explain this too much, but that's for the attention part.

This is just to make inference faster. When you do QK and when you go back to slides, if you go to, where is it?

Right. WQ, WK, and WV. Instead of training WK and WV, you train a small slipper and you repeat this. And this can make inference faster. And so like we don't actually train the full matrix size WK anymore. We train a small slipper and we just repeat this.

The attention, again, all of this is just preparing,right? This is just preparing,right? Okay. Ignore, ignore, ignore, blah, blah, blah, bloat. Get rid of this. Don't look at that. There is QK and V. That's the matrix part. There is some, the only problem I find that people struggle with is there is like dimension manipulation.

You have to like manipulate the dimensions of the output. That is actually kind of annoying. I do agree this is actually very annoying, but just assume it is the this,right? That's all we're trying to do is this. And that is just these three lines do this, do this.

And then we want to do QK transpose,right? Oh, you have to, sorry, you have to apply the rope embedding. Don't forget to apply the rope embedding. And then QK, that's a repeat KV. Repeat KV is the one that, the trick that I said to make inference faster by repeating the, so if you go back here, the K and the V, the K and the V, you take only small slibbers, you only train small slibbers and you repeat them four times.

And it does not reduce accuracy that bad. And that's repeat KV. This is the QK transpose,right? Torch to map out QK transpose. This is the attention part. Softmax, dropout. Okay, no one uses dropout anymore. Get rid of that line.

Matrix multiplication,right? So this is, so up to here, up to here is

QK transpose over root D softmax times V,right? So up to here. And then we do some sort of, and then we have to do some output projection as well. The rest, okay, whenever you see this part, just ignore that,right?

Whenever you see like if self.convig.pretraining TP is more than one, this is just for, this is just for faster training. So you can get rid of that. Well, not for faster training, for like training across multiple GPUs. You can ignore that and just assume it's just that one line.

Now there is more code like FlashAttention 2. Ignore. That's just for faster FlashAttention. Ignore, ignore, ignore. Right. No, you don't need to see this. scale.productAttention is a faster version of Attention that is native to PyTorch. Also ignore. You do not need to see that as well.

Pretend you didn't see that. And then finally we get to the decoder layer,right? Remember each, we show, okay, wait, maybe I should exit the slides.

Where is it?

No.

So remember the decoder layer. So this,right? Remember we said we repeat L times,right? This is repeat L times. This is just shoved in the decoder layer,right? We call this one decoder layer. And again, we do the layer norm.

Remember, put layer norms everywhere. Do layer norm, do attention, add some residual. This also makes training more stable,right? When you add residual, it makes training more stable. Do more layer norm, do more L, do an MLP, add more residual, and then we complete it,right?

That's just one, that's one component of the decoder. And remember, we repeat this L times. And the rest of the code is just doing this L times. Now where is the L times? Comments, comments, comments. Forward,right? You go to forward.

Sorry.

Yeah.

To the previous one. So does it make any difference where you put the layer norm?

In my opinion, put it everywhere.

Yeah, but in terms of calculating the residual and then adding it by.

Oh, that's just by.

The norm after.

Yes. Why is the ordering? Why is the ordering? I think it's layer norm first, then add residual. Is that correct? Or maybe I'm getting confused.

So it's get residual, then norm, then calculate the attention, add back the residual, and then there's probably another norm term somewhere down the line.

Yes. The ordering to tell the truth, it doesn't really matter. I think there are some research papers which show if you switch the ordering, it maybe increases, like it decreases accuracy by like 0.01%. To be honest, we need to do more testing again.

Like in my view, shove layer norms everywhere. This should make training more stable, but it makes training slower. That's the problem. That is the only problem. But I suggest you to put layer norms everywhere. Wherever you see it, just shove layer norms.

Yeah. Layer norms always work. Okay. And.

Sorry, just one more thing. Why isn't this an issue that you get the residual before applying the norm and then after, like you calculate the attention on that layer after the norm? So you kind of combine the residual before the norm with your calculations after the norm.

So why do you add the residuals of the previous? So you do, you take the residual, you save the state before the layer norm, and then you do a layer norm, and then you add back it in. So why do we have to do that?

Is that your question?

Yeah. Why the normalization is between that rather than before.

Oh, you could do it before if you want to. You could try it. It depends. I think it's for, yeah, it depends. As I said, all of this is like gospel, like, oh, why do we do this? It's just people tried it and they said it works.

I just meant why this isn't an issue, but I guess.

Oh, what do you mean by not an issue? Like why does this work or why is this like not a, what do you mean by issue?

I mean the normalization kind of changes the, in some way, the representation.

Yes. It scales it.

Yeah. So you're combining something from before that change with something after that change. And why is it compatible basically? Why it's.

It's just maps. You just let it, just, it just works.

Yeah, yeah, yeah. I get that.

Because like if you do that, the order grid engine will still know that you did it and the derivatives will still be applied correctly. So you just assume that it works. To be honest, that is another research topic.

You should try that. I mean, I'm being like serious, like all the so many research questions that are like open, like why doesn't everyone just put layer norms everywhere? Like why don't you put a layer norm after the multi-head attention?

Put a layer norm after the Swigglu. Put a layer norm after the inputs. Put a layer norm, you know, everywhere. And my hypothesis is it makes training more stable, but like it makes it slower though. Yes.

In this, in theory, could mechanistic interpretability solve, like not solve, but explain this?

Do you mean like, what do you mean by like explain this?

Mechanistic interpretability where you take the.

Which method? Do you mean like doing the order encoder style, like sparse.

Or lens or like just direct weight, just looking at the weight and what happened at inference time or training time.

Do you mean like the, which method, like the, because I know there's like many methods for mechanistic interpretability, like there's different types of methods. This is sparse order encoder one or the.

Well, transformer lens is Neel Nanda's tools.

Okay. I have a doc. Yes, that's true. I'm not that, I'm not an expert on that, but.

No, but you can see the literal like activation.

Yes, yes. The activation one. Yeah.

Yeah. So basically what, so I mean, I'm going to not diverge from this, but so basically, can, do you think we can figure out why this works this way? Like, because you said it's kind of an open question,right?

But do you think that using tools like transformer lens where we can look at training activation or not activations, but like steps where in fact we would have to like, I'm not sure if I'm explaining myselfright correctly, but like do you think mechanistic interpretability is a path to understanding this?

Good question. Could mechanistic interpretability, okay, it depends. I think my view is it just layer norms, if it's specifically on the topic of layer norms, it just makes training more stable. I don't think it has any like meaning, like that's my view.

Okay, that's fair.

Like I think like the master questions don't show that it has any meaning. I just find it to be just stabilize training. There was like papers like the, what was the one? Batch normalization. I forgot what the term was.

Yeah. Like there was like a theory which shows that batch normalization reduces problems of out-of-distribution data and stuff like that. Oh, it reduces internal covariate shift or something. That was the phrase, which, yeah, I don't know what that even means, but anyways, does anyone know what that means?

There was like a, there was like a video for that as well. The, yeah, does anyone know what that means or? Yes.

What do you mean by making training more stable?

So layer norms, my view of layer norms is when you do, if you don't do layer norms, if you keep, okay, let's say you take a number, two, you multiply by two, you get four. Remember the 32 layers,right?

If you multiply by two continuously, you will get infinities in the end,right? Because you like go out of scope of the float 32. So what layer norm does is it makes your number go back to a good scale.

So if you do two times two is four, let's divide it by four, go back to one,right? And so now it's one again. If you times two, it's two again. Let's divide it by two again, go back to one.

So all layer norm does is it makes a scale go back to a good scale. Like it doesn't, your numbers don't like diverge on both sides. That's what layer norm kind of does. Does that kind of, okay. Any other questions?

Okay.

So all we, remember the decoder style. Oh, I think we actually kind of finished reviewing the Llama architecture. There's nothing else to do. The decoder,right? You do this 32 times. Remember like four decoder layer is in self.layers. You do this 32, I think it was 32, I can't remember, multiple times.

That's the decoder. You just do, you apply this multiple times. You do a layer norm. And finally you get your logits. Where is it? Your LM head,right? This outputs the probabilities of which token. Remember we're trying to predict the next token.

We output probabilities for each token. And that is called the LM head. And where's the forward function? The forward,right? There's a forward. Always read the forward. Self, you do, you go through the model and then you do, okay, remember ignore this,right?

Ignore this. And you just do the LM head. That's just one line. One line. Okay. One line. And then you do the float. Now another question people have is like, why do you have to do the float? Does anyone know why you have to do, you have to upcast a float?

Why?

Any clues? Have a guess. I remember we made this big up. Have a guess. Have a guess. Have a guess. Why do we have to upcast a float?

Gradients.

Sorry, gradients. Okay, clues. Why? Why gradients? It is related somehow to gradients. Anyone have a guess? Okay, it's for training stability purposes. So the softmax, you should always upcast a float 32 because it makes training more stable. If you take the derivatives and the gradients, if you do not do float 32, you might get NaNs as well.

Remember the exponential can be very large. So you want to take the float 32, which has larger precision than float 16,right? Float 16 is the maximum number is 65536, I think. I think it's 65536,right? But float 32, the maximum is like some large number to the power of 38 or something, 10 to the power of 38.

So that's why you have to upcast it to float 32. This just makes training more stable,right? So all of these things that we do tricks, it's just to make training stable. Yes.

You said you're doing the operation 32 times. Is that just like an arbitrary thing that people figured out works, or is there a reason to put up?

Oh, that's up to you. So like if you want to do more parameters, you can do 300 times. Up to you. That's just make your model 10 times larger. So like when you hear like, you know, Llama's.

Which time you do it, it's generating like a weight for the model? Is that each layer, like each time you iterate, it's going to generate like a set of weights? Is that what it is? Or is it?

So the weights you train, when you take the tokens, you go through the architecture and like it changes the, it changes the tokens and these tokens keep shifting to like some sort of like new direction and you keep doing this.

So but if you do it more times, do you get, you said you get a larger model from that?

The problem is you have to train more weights.

Oh, that's so just.

So each 32 times has different weights.

Got it. Right,right. Okay. So each iteration has its own.

Correct. Each of the, each of the iteration. Well, sorry, yeah. Each, there'll be 32 different weights for each layer.

Got it. Okay.

And so like, yeah, normally people just, if you see like there's like, you know, GPT-4, what is it? Like one something trillion tokens. I'm assuming there's more layers, larger embedding dimension, larger this, larger that, more layers. Normally speaking, the more layers you do, the model can learn more.

So that's the whole reason why you wanted to add more layers. You just want to increase the capacity of the model to learn. Again, it's to make training more stable again.

And so this, remember the shifting trick that we did in PyTorch? The shifting trick is just this and this. That's the shifting trick. That's the thing that makes it learn to predict the next token.

And then you pass it through the loss function, the cross-entropy loss, which we discussed about. And then that's the Llama architecture and that's normal. The rest is not useful. The rest is, yeah. So in theory, you could write the entire Llama architecture in like I think 50 lines or something.

The rest is just unnecessary bloat,right? This all of this is 1,600 lines of comments and stuff like that. But you know, this is for Hugging Face's implementation. It's highly respected. And this is what you should look at first when you read a new architecture.

So we just kind of went through the Llama architecture. Hopefully you can kind of get a sense of it. Obviously, if this is your first time reading the source code, it's actually not that hard. It's not that complicated.

You just have to see which components you can ignore,right? It's not that scary. Yep. Does that kind of get it? Or do you guys kind of get that feel? We're going to do more. Obviously, this is the first one.

Any questions?

We have done the questions. Is there a major architectural difference between Llama 2 and Llama 3?

No, not really. Other than more tokens.

I think they changed embedding. They did change some of the numbers, like how many numbers you want to represent for each number. They changed that. Large vocabulary. They did much larger vocabulary and more tokens. Other than that, no, there's no change at all.

Yeah. Yes.

I think a lot of your writing was in Triton. Do you find like any reason to ever drop into CUDA? Or is like Triton pretty much like always?

The reason why, it's funny, I used to work at NVIDIA. Why shouldn't I be writing CUDA,right? The reason is I see CUDA as extremely annoying to write. And if you want to optimize for just NVIDIA hardware, okay, go ahead.

You can do CUDA. But my view is like, I don't think so that's going to be forever. So as a, as like a, as like a safety precaution, let's just do Triton,right? Let Triton handle the compiling down to CUDA or AMD or whatever, Intel or whatever,right?

And Triton can be the intermediary. If you want to get like 10% faster, yes, you should do CUDA, but it's only 10%,right? If you do fine-tuning two times faster, it's already like, it's already nearly at the ceiling. You can only go so much.

And so like if you want to go down the extra mile, yes, more than happy to welcome you to do that. But I do not, I do not like, it's funny because I used to do CUDA all the time, but I don't suggest it.

You will get more performance though, but I don't suggest it. Yes. Question?

You never had to drop down CUDA,right?

Oh, sorry. Yes, what?

You never had to drop down CUDA. You could live with Triton.

Yes, you don't. Yeah. So Triton, you write it in Triton, then it compiles down to CUDA. Yeah. Sorry. Wait.

Yeah. Let me follow up on this. Why doesn't Unsloth work on AMD or the other targets that Triton?

Actually, it could work. The only problem why it doesn't work on AMD is Triton. Oh, I think. And XFormers. Actually, if, so if Triton works on AMD, we work. If Triton, if XFormers, so Facebook's FlashAttention library, if that works on AMD, then we work.

Oh, funny. We work. But anyways, it depends on those conditions. So if AMD has those to work, then yes. In theory, you can remove XFormers and just use scale.productAttention. So there's only one dependency, which is Triton. I think some people have gotten into work.

So it depends. Yes.

I kind of have an answer to that. I've trained on a MI300 instinct with one card with Triton and it worked with AMD. So.

Okay. I mean, if Triton works, then yes.

It just works. So I just have an answer. Sorry.

Okay. No, good. Your answer. Yeah. Okay. Yeah. But we don't, so officially we do not support AMD, but I guess it works. Okay. That's interesting. Yes. Okay.

What's next? Where's my, where's the Gemma one? Yes. Okay. So we're going to be talking about Gemma bugs, specifically Gemma. So if you go to our blog post, I actually have, we wrote a blog post about all the issues that we found in Gemma.

For example, you must add a BOS token. There is a typo in the paper. Yeah. So we don't just find bugs and, you know, we have to read the paper first to understand the model. Now the problem is sometimes when people release models, they don't release papers.

That is very painful. That happens a lot now. So please, model creators, please provide papers. Otherwise it gets more complicated. There's also like some other issues. And we have a Colab notebook which provides all these. So if you open up the link, Gemma details in the, remember, if you don't have access to these slides, it is tinyurl.com/unsloth,right?

That's the slides. If you open up the Colab notebook,

this is actually runnable in Colab. Please log into your Google account for this to actually work. But we show that this is the log L2 norm. So we check the, so this layer number,right? There's like 18 layers. We check every single layer, the output of the actual good implementation, so the DeepMind implementation with the Hugging Face one, with the PyTorch one, with the other ones.

And if you do the L2 norm, you find that the error is very high. And we, what we did, we showed is that you can actually move the error down by doing multiple changes,right? So each line, you can see there's like multiple lines.

Each line is actually a method that we apply to make it better,right? So like we finally found that approximately either the blue line, I mean, it depends on which one you like, either the blue line or the black line makes training much better.

Does anyone notice any interesting things about that, this graph? Anything interesting? Like, do you see the, you know, the, so remember each line is a fix that we did,right? So like there's many lines and we did a fix and it changes the error.

What, and we selected the black line to be the final one. Does anyone have any, what is like anything interesting?

There was the one that did the most, like there was a huge jump that the rest of.

Yes. So one of them caused a huge jump and that is a float 32 fix that we did for all architectures. Yes. And the other ones are less prominent. But anything else? Anything else interesting? Yes.

Yeah. So the dash line, you know, finally goes up in the letter hap, like why?

Yes. Fantastic. Why? I do not know. And that is a good question and I don't actually know. I think it's just language. You know, I have a theory. The theory is.

You need to tell us.

Yeah. But unfortunately, I can't say everything. Like, I mean, my theory is, and there was also a jump as well in the middle. And the blue line, you know, it starts from very low, it goes up very high and everything does this,right?

So like there is a some weird transition boundary in the Gemma model,right? And so like I'm just going to guess, my guess is that when you do it, when you train a transformer, the later layers get harder and harder to train,right?

The earlier layers actually get very easy to train. And so this transition boundary is when the model probably was not really trained that well. So I'm going to guess, this is just guessing, that maybe the model should have been trained for more data and the boundary should disappear.

This is just my guess. So there is a phenomenon essentially, it's like more data, the model, the last layers are much harder to train. And that's kind of my theory, but I don't think so that's correct, but okay.

And the blue one kind of drops for a moment,right?

Yes.

Right before the last one.

Yep. Exactly. So in the end, so now the question is like, why did we choose the black one then? Why didn't we choose the green, the blue line? So that's adding the exact JLU that we found. So if you add the root fix plus the exact JLU, you get the blue line.

But we in the end decided to do the black line. And why do you think that is? We did not choose the blue line. We should have chosen the blue line,right? But with final, all the fixes that we did.

So essentially the answer why we did not choose the blue line, the blue line should actually have had lower error,right? The reason why we didn't choose that is because there was not just one error. There were two errors.

There was many errors. And all of the errors combined together, we finally chose the black line because it matches the original implementation. So because remember the trick is you have to match the original implementation of whatever the Gemma models created, the Gemma model creators did.

So you can't like just look for this error. Maybe like, I mean, maybe like if someone chose different like fixes that we did, you can probably get even a lower training loss. I guess you could. But we decided to choose the black line because that's what the original implementation did.

Any other questions?

For my own clarity here, so I know the weights, they're trained. And

when you kept saying the model could have been trained, you're not talking about the weights for you.

Oh, I am talking about the weights. So the weights are the ones in the, so the model weights are the ones training,right? So the rest you don't actually train. It's just the weights itself.

Yeah. Because it's a bit meta,right? Like at the end you end up with a model that's the transformer and the weights themselves have to be trained. So anyway, that leads me to my follow-up question, which is,

do you have examples of that weight training? Like is it, I guess, what are the training based off of? Is it iterative where you're just going to see the training loss over time and you're just trying to find theright weights that give you the least amount of loss?

Yes. So remember the goal of a transformer is you want to predict the next word,right? So the sentence, "Hello, my name is Daniel." You're trying to predict, "Hello, predict my,"right? My predict name and so on. You have this data, correct?

Like you have, just take novels, you shove in the novels, you already, you essentially create a data out of thin air. And then you change these weights using like backpropagation, do derivatives, and try to like change these weights such that you get the highest accuracy.

And this training procedure is called backpropagation. And so like I was trying to show you like how do we actually derive the derivatives. When you do backpropagation, you need to derive the derivatives. Just use PyTorch. PyTorch will do the derivatives for you.

And yes, but that's, does that kind of answer your question or?

Yeah. I think that gives me a glimpse of some of that stuff. Yeah.

Okay. Yes.

So you mentioned that middle layers are, you know, middle layers are usually undertrained.

Yes.

And often for fine-tuning, you see that the first layers get overtrained. I know that people are doing layer freezing now, but is there a way to change the learning rate dynamically based on the layer?

Yes. Unsloth actually has that. So you can actually, depending on your layer, so for now what we do is your embedding and your final layer, you can change different weights, different learning rates. So we found that if you train on, if you train the last layer with the embedding weights and the first, sorry, the embedding weights in the LM head by a factor of 10 smaller, the learning rate, you can actually have increased accuracy.

So you actually, so yes, you should, you should change the, you should change the learning rates for each layer. But people don't actually do that. I think it's because if you set a learning rate for each layer beforehand, you're kind of like, it's like there's, you're like doing subjective bias.

So like that's why people just set one learning rate for all the layers. But I think in this case, I'm just going to guess, okay, this might be a transformer. This is transformer general. This is not just for Gemma.

This is for all transformers. Maybe, I guess, layer-wise learning rate could work. I think there are like some papers which do that. I think it's called LARS. I think LARS does layer-wise, I think it's called LARS, layer-wise learning rate.

I hope that answers your question. Yes.

Yeah. Where does the Y-axis end?

Oh, the Y, oh, it's a log L2 norm. So it's, you take the DeepMind implementation, you code it up correctly, then you take the other implementations like PyTorch, Hugging Face, even DeepMind's own implementations, and then you check each layer, the output, you compare it with the original, the correct implementation and check what's the error.

And that's the thing that I graphed. And your goal is you want the error to go to zero,right? So like you want it to go all the way to zero. So like, you know, on the bottom and not like very high.

And that's log scale,right? So the error is not like a small number. It's 1,000,right? So like every single line, every single step you go down is a log difference,right? It's not, it's not like a, it's, I essentially logged it.

If you did not log it, it would look very bad, but I just logged it. Yeah. Does that, okay. Any other questions? Yeah.

So let's say if there's an issue in a tokenization kind of part, a fundamental thing, or we find some optimization for a bug and you have to change the way you're tokenizing your sentences and everything, would you have to retrain your models to like incorporate these fixes and changes to the tokenizer?

This actually happens a lot, very frequently. And I think like, so like for example, Tiny Llama, someone tried Tiny Llama and then training already 80% completed, they found a bug for tokenization. They're like, so it happens very frequently and it depends on what you want to do.

I think it depends to the model creator. If you already spent millions of dollars, maybe just keep, just train it with the bug and then you release the bugged version. But it should still work, hopefully. Yeah.

So in theory, let's say if OpenAI would have a lot of difficulty shifting, if they found, like if somebody else found an optimized tokenizer or something like that, they would have trouble shifting to that model because they would have to spend like.

Yeah. To retrain everything. Correct. So just assume it, just leave it. If you already spent like billions of dollars, I'm probably not a good idea to retrain.

So even if it offers like a, like 2x optimization for somehow, like they would have to spend, like they would have to retrain and spend.

Yes. You have to retrain everything from scratch. But that's why like, I think like, that's why you should do like small scale experiments, you know, get like a smaller model, train it for less data, test it, and then see if the accuracy is good and then you scale it up.

Yeah. Any other questions?

Okay. I will, yes. So there's a notebook. So we show step by step exactly what we did. And if you inspect the code, okay, the Gemma code is now, the Gemma code,

if you, oh, okay. Wait, no, it's modeling Gemma.

Oh, okay. Maybe I should just go to Hugging Face itself. Wait, let me go to, you can actually find this if you copy paste this,right? You edit the, you go to Gemma and you go to modeling Gemma.

Oh.

Right. This is, oh, did I not, okay, let me just, okay, maybe I typed it wrong. Did they not, oh, okay. Maybe I did two Ls. My bad. I always get confused on that. Oh, what is this? This is interesting.

Okay. This is like new. So, okay. Yeah. I did not, yeah. So all of this, so we wrote inside the, like, you know, Llama does, so we showed, for example, in the code now, if you go to like Hugging Face's code for Gemma, we wrote, I tried to write some comments, you know, for it to be more clear on why we are doing this.

And so, for example, the layer norm,right, you have to be careful to where you upcast and downcast. And we write this in here. Where is it? I think it's, no, no, no, not, oh, wait. Is it? No, I'm pretty sure I wrote it somewhere.

No, it is here. Yes. Okay. It's a bit unclear. I need to make this bigger. Okay. It's a bit blurry. But you can see that depending on the model in Gemma, you have to actually upcast to Float32 everywhere.

You must use Float32 everywhere because that's the original implementation used Float32,right? So you must always follow the original implementation. If you don't follow the original implementation, then you will get wrong, like, you know, somewhat worse results. And the problem was other implementations just copied Llama and Mistral's code and they did not do this.

And so we found that you actually have to upcast correctly over here,right? You have to upcast immediately. And then you downcast at the very end. And so we wrote a few comments like Llama does x.2 Float16 whilst Gemma is x, you know, it really, like Llama does that,right?

But Gemma does this,right? So there were like small little issues downcasting, upcasting. Another question is like, why do we have to do downcasting? Does anyone know why? Like why is there always like downcasting, upcasting, Float32, Float16, Float8? Does anyone know why we have to do downcasting, upcasting?

Performance gains, but lower precision.

Yes. Correct. It's for faster speed. So do you know how much faster? Like.

Eight? Yeah, I don't know.

So Float32 to Float16, what do you think?

It depends what you do it with,right?

It depends. Who said two? Okay. Good guess. Why did you guess two?

Is that in the form?

Well, that's for sparsity. Okay. Okay. Yes. Okay. Float8, approximately two. Actually, it could be more. So Float32 to Float16 is actually not two. It's actually much more. I think it's five. I think. Or is it six? The reason is because the representation of the float is different,right?

So Float32, I have float, floating point representation, Wikipedia. I think it's in here somewhere.

Oh, and maybe I go to beat Float16. Where is beat Float16? Brain float, be float, beat float. Yes. Right. So like, there it is.

Oh, there's more pictures now. Oh, they edited this. I did not, okay. This is new. I didn't see AMD FP24 format or Pixar. Oh, okay. They have like weird formats now. This is Float32,right? And Float32, the exponent has eight numbers,right?

Eight bits. And the fraction of a bit has 23. And when you do matrix, when you do matrix multiplication, does anyone know how to calculate the number of transistors you need for Float32? Does anyone know? It's a formula that's related to the exponent, the fraction, and the, just the exponent and fraction.

What do you think the formula is? Have a guess,right? I said that it's approximately, so if you have beat Float16, the fraction is seven,right? Beat Float16 has 16 bits you can use. The exponent, the exponent is used for the dynamic range of the number,right?

So if you want larger numbers, you have to have larger exponents,right? So this means beat Float16 only has a range of two to the power, okay, it's not two to the eight, but like just assume, you know, it's two to the power of eight.

Okay. That's notright, but just assume that. Two to the power of eight,right? But

it's, yeah. And this one, Float32 also has two to the power of eight. There is another format called Float16, which is two to the power of five. And then the fractional component is 10. So all of these numbers you can scale,right?

How many do you want for the exponent? How many do you want for the fraction? You must include the sign bit. And the trick is you must have 16. You need to fit, you know, 16. So you could have like exponent one and fraction could be 14.

That could also work. But does anyone know how many transistors you need to use for Float16, for example, and beat Float16? Remember I said it was like around five times faster. It's actually notright. I think it's even more.

What is the formula? Have a guess. How many transistors do you need to use to do Float16 multiplication, approximately? Or Float multiplication? It's a formula related to exponent and fraction.

The answer is exponent plus fraction squared. That's the answer. So what does that mean? That means Float16 is five plus 10 squared,right? And Float32 is eight plus 23 squared. So it is not two times faster. It is much faster,right?

So like, I don't know what that is. What is, so it's eight plus 23 squared, 500 and, so you need approximately, okay, this is approximately, you need 537 transistors for Float32 multiplication.

Is that for the same three squared or just 23?

Oh, it's just 23 squared. So it's eight plus 23 squared. Yeah. And so what is the other one? I think it was, was that, I can't remember. So it's eight and seven,right? So eight and seven. This is Google's format.

It is 57. So what does that mean? How many times faster?

10 times.

Yeah. So it's actually 10 times faster,right? So Float32 to Float16, beat Float16 is around 10 times faster,right? Float16 is five plus 10,right? So five plus 10. So beat Float16 is approximately two times faster than Float16. Although to tell the truth, no one really notices any difference.

But in general, beat Float16 is actually faster,right? So that's why it's not two times faster. It's 10 times faster. And that's why you must use Tesla T4s, as I said, because it has Tensor Cores, which does this,right? The Tensor Cores does Float16 multiplication very effectively and very efficiently.

And so do not use P100s again,right? P100s do not have this methodology. Yes. Question.

Why don't we go even smaller than 16?

Yes. Float8. So Float8, I don't know. There are two formats for Float8.

Oh, wait. I don't think so. It's in Wikipedia. Float8. Oh, okay. Floating point. There is, it's called EEM. Oh, I'll just use Minifloat. Does it, they have some. Yeah. There we go. Allright. So you get to decide, remember, if you want to have eight bits, you get to decide how many you want to do for the exponent, how many you want to do for the fraction or the Mantisa part,right?

You get to decide. And depending on the company, you know, it's unclear. There's no standard. So this one's 1, 4, 3,right? So like, what's 1, 4, 3? One plus four,right? Four, three squared,right? Is that four or three? Yeah.

13. So Float8 is, I think it's around, yeah. So around four times faster than beat Float16. But in general, it's not. Okay. In general, it's like two to three. It's not going to be four. The reason is because you're packing so many transistors in.

You also have to do like energy. You have to like do the data movement. There's like other transistors you have to do. I just, approximately, it's two, two to three times faster. That's Float8. Can you go even lower?

Yes. Why don't we go one?

One bit?

One bit?

So you must have the sign though. So you can't do one bit. So 1.58 bits, some people have been talking about. Two bit. Two bit could be possible. The problem with two bit is it's problematic because when you do two bit training, yes.

Okay. So let's see. Let's do two bit,right? So what do you want to do? How many exponent? Zero? Remember, you have to have a sign bit. That's the most important.

One?

One for the exponent and then fraction zero,right? Because that's, because remember it's squared. So plus one. Oh, wait. No, it's zero plus one. Okay. So it's one? Okay.

10 times faster? I don't think so. Okay. Maybe two bit is probably too low. Maybe four bit. Four bit could work. Yeah. Yes.

Question because, you know, how does this map to the flops on, say, NVIDIA's spec sheet? Because I understand this map, but they, for example, have the same TensorFlops for BF16 and FP16. They don't do it.

Oh, that's just because they wanted to do that. Just for easier calculation. Like.

And then for their Tensor32, they do list the two X. So that's why I would guess.

Tensor32 is not 32. It is, they have it somewhere. NVIDIA TensorFloat, it's 19. That's the trick. They like to like do marketing and they say it's 32, but it's actually 19. Yes. That's why it's the same. Okay. Any other question?

Was it someone else raise a hand or, okay. But yes, I was going to say like you can do four bit,right? So four bit is actually a new, NVIDIA's new GPUs. The B100s do have four bit. So that is approximately two times faster.

Now, the reason it's not, okay, let me just try four bit. I think it's one plus, it's probably like two plus two or something. I don't know. Six. Okay. Right. It's not going to be that much faster because, as I said, there's power transistors and there's other transistors.

You can only go so far. Just to jump from Float32 to Float16 was very large. Yes. Yes.

So quick question. So for the example, the one bit bitmap, that.

1.58 bit. Yeah.

Yeah. So that would be an example of one bit like in this.

So it's a different, so actually I had a tweet about this. 1.58 bit and Float4 is the same in terms of number of transistors. You'd rather use Float4. The reason why is 1.58 bit is you have to do more manipulation to make it work.

You have to use like the straight through estimator, like it's a horrible mess. You'd rather just use Float4. So Float4 and 1.58 bit are like similar.

You get to create your own base model and then you do a one, then you, if you replicate, if you replicate the paper.

Yes. What do you mean?

Which most of us have never done,right? Which would be.

Technium and the Noah's research probably replicated there some though. It does work somewhat. Somewhat works.

I mean, yeah, it's one bit, but I mean.

1.58. Yeah. It's not actually one bit. Yeah. I think it's like three.

They call it one bit, but yeah.

Oh, yeah. They like to call it one bit. Yeah.

But my question is like, so in theory, like obviously I don't know who works here, but like most of us have never built a base model,right?

Yes.

So like.

Oh, you could. Yeah. Yeah.

Yeah. You can with enough GPU power, but one bit bit, you know, that was, and they even had like a really good great tutorial and like, so, but do you think that that's just like, I'm just asking for your opinion on that.

I don't think so of 1.58 bit would be the future. I think NVIDIA is like the focus is on Float4. They might go to Float, I think Float4 might be the final precision. I don't think you can go any faster with that.

I think Float4 is a final, no more. So we won't be having that much faster GPUs. I don't think so. I mean, Float4 is actually, they don't actually do Float4 anymore. It's actually Float6 for the gradients, Float6, and then Float4 for the activations.

Like, you know, it's very weird. I mean, you could do like Float3, Float2, but like it's your diminishing returns.

So in ARM silicon though, there's been like advances in like super low bit width bits versus.

Fixed point stuff. Is it called fixed point stuff or? I think it's called fixed. I knew ARM has fixed point.

Oh, well, yeah. So it's, I mean, just like the Snapdragon X, like the new.

Yes. They have, so it's like customizable as well or I don't know.

Yeah. Well, okay. So the SDK is broken. You have to pass the, so this is why you can technically run Mixtral 8x7B on your phone at like 20 something FPS, not FPS, sorry, TPS is because you can use UFS 4.0 as flash storage and subsequently use that as memory.

And then the, but the thing is then you're running at two bit precision, which is.

So that's probably why this, so if you use two bit precision, that's why you have memory reductions. But there is actually papers which show that if you do two bit for the MLP plus four bit for attention, that's actually the most appropriate.

You can actually do that.

So that's not an invalid approach.

No, that's not invalid. That actually, it works. It works. The Möbius people did that, I think. Yeah.

That's all.

Yes. Question. Sorry. Okay.

Two kind of related questions on precision. First one is like, why is the negative bit like you must have the.

The sign bit?

Yeah.

You don't have to, but it's generally, generally like standard practice to have the sign bit. In theory, you don't have to. The only problem is if you don't have a sign bit, your numbers will be zero, one, two,right?

But what happens if you wanted to like make the model, like you're trying to not make the model learn negative directions anymore. You could do that. I don't know if there are papers. Maybe you should write a paper about that.

Train a model on that and let's see what, okay, but yeah.

Well, and very related, I think, is all bits. It's all bits,right?

Yeah.

Softmax, you're basically just linearly like fitting stuff down to a certain number of bits. There's nothing special about softmax and things that could be like exponentially big sort of going this way,right? Like.

The reason.

Why this to be better?

It's because remember, when you do softmax, you also have to normalize by the sum of the exponentials. And if you do exponential of 10, you already get like some large number and this probability will take over the entire sum.

Well, but you're not like logging it. You're just square rooting it.

No, no. It's the sum of exponentials divided by, oh, sorry, the exponential divided by the sum of the exponentials.

Yeah. But like the biggest exponential dominates the sum,right?

Yes. That's the problem though. If you do that, then your model's not learning. You're just trying to learn to predict one token. Why don't you just predict that one token then? Like the largest one that you did. That's kind of what you're forcing the model to not learn anything.

That is why you have to like minus the maximum. That's the trick that we showed. It's like minus the maximum and then you can like reduce this effect of this one token or this one issue. So it's for training stability purposes.

I don't know if that kind of, okay, probably that didn't answer your question, but okay. Yes.

From a craft power perspective, how much slower was upcrafting the layer norm? And then do we know whether that's actually more accurate than the way that Llama does?

That is a good question. To be honest, I do not know. I don't think so it changes too much. Layer norms, if you upcast, it's probably, yeah, small effect, small effect. But the reason why you need to upcast is because Gemma did it before, so you have to do it.

Remember, the trick is you must follow what the original implementation does. Any other questions? Or, okay. There are like some other issues which we showed that more, okay, this, it's funny because it's all about upcasting, downcasting and stuff like that.

Each implementation does its own thing. Unfortunately, how do you actually analyze this? You have to open three screens up. The DeepMind one, the DeepMind one. Okay. Okay. Too excited. You have to open up three implementations. DeepMind one, the Hugging Face one, the Keras one.

You have to open up three screens and you see line by line what did they do. And then now you have to guess which one's the correct one. The guessing part is the most painful. So you have to like inquire.

You ask Hugging Face which one's the correct one. You look at the paper, which one's the correct one. You assume the DeepMind one's correct and stuff like that. So there's like some human component. You have to guess. Guessing.

So that's probably why it can't be automated,right? These error checking things cannot be automated. It's because there's a human there which made these decisions. And so you have to, now you have to decide which one, which of those decisions did they choose?

And you can't really automate this away. I guess you could automate this by doing the methodology which we described. Try all combinations and see which one has a lower error. I guess you could do that. But remember, you must have the original implementation first.

That is a problem. So there's like chicken and egg problems. The rope position, so this is the one I was talking about, upcasting rope. This is in all architectures now. You must not downcast rope. If you do, you will get wrong results.

So previously on the left,right? If you see 8192, 8192, 8192, that's the positions. That is definitely incorrect,right? What does that mean? Like, do you know why that's incorrect? 8192, 8192, 8192. Does anyone know why? Remember, this is positions.

Why is it, why is it all the same? Like, does anyone know why this is very bad?

Too often too.

So we kind of like essentially now we, the three words have the same position,right? 8192 is the position. And what is another big error of this? There's actually one more error. Let's assume the maximum is 8192, the sequence length.

What is 8192?

So the end of the answer is the end?

It's out of bounds. Remember, it's minus one for Python,right? It's 8191 is the correct number,right? So if you correct this, you get 8189, 8190, and 8191,right? And you can see all the numbers are like this. So the point is if you use, remember, the whole point of this problem is because we're using Float16 for faster training.

Remember, Float16 is how much times faster?

10.

Yes. Around 10 or five to 10, something around there,right? That is why you have to do this. And these are the issues pop out because of this issue,right? We're trying to make training faster, but then these issues come.

And the Jell-O one, which we described before, this was the first bug that we found. Actually, I think this is the main reason why we were trying to look through bugs is we found that, oh, look, there is this bug in Jell-O in the activation function.

And so the point is Keras used approximate Jell-O. The PyTorch and the PyTorch version used exact Jell-O and Hugging Face also used exact Jell-O. And the question is, which one is the correct one? Is the exact Jell-O correct?

Is the approximate Jell-O correct? So what's the difference between exact and Jell-O activation function?

There is the, where is the, I don't know if they have the exact and the, it's called flex.

Oh, okay. That's night mode. The, oh, that's even worse. Okay. Whatever. Oh, wait, that's Prelude. Where is Jell-O? Oh, wait. No, I have to find it,right?

Yes. Right. So like the exact Jell-O is this one,right? There's an error function. Okay. My thing is not rendering it properly. But if you essentially what you do is you use Desmos. So what I like to do is I use Desmos, Desmos,right?

And literally plot them together, plot them on the graph,right? So like if you have,right, XY is equal to X over two,right? You literally type this in. What is this? One, I think you can do error function. Oh, yes, you can,right?

You can do error function,right? X divided by square root of two,right? That's the exact Jell-O,right? Now you type in this complicated formula for X divided by two, one, one plus than what? I don't remember this. Square root of two divided by what was it?

Pi. Oh, it's pi. Pi and the what? X plus zero point.

0.47.

Can I, oh, we can't. Okay. 0.447. 0.447. What was the, oh, 1.5, 1.5X to the cubed. Was it cubed?

Yes.

Yeah. Okay. Right.

It's only the two that is uneven.

Oh, is it? Oh, you'reright. Okay. Wait, is it, oh, is it just the rendering problem or is it square root? No, no, no. It's square root of two over pi, I think. Wait, is it correct? Wait, something, I did something wrong.

Maybe I did something wrong. Oh, whatever. Who cares? Just assume. Okay. Oh, wait. Oh, you'reright. I put the square root everywhere. Oh, is that what you were saying? Yeah. Oh, okay. Oh, no, no, no, no. What's? No, get rid of that.

Okay. Let me just, no, it's tan of everything. Now I have to do this. Oh, yeah. This probably have to play around with this. Oh, there we go. There we go. Right. So the blue line, if you remove it, the blue line and the red line,right?

They're the same thing. But what's the difference? Remember in Desmos, I don't know if people know this. We can actually do derivatives, D over DX. Did anyone know this? You can actually do derivatives. You get your D over DX and then you can do this as well.

D over DX,right? And they generally align,right? The exact Jell-O and the approximate Jell-O generally align. And guess what? You can also do integration, integral of minus infinity.

Oh, did I spell it wrong? Oh, infinity to infinity,right? Of, I think this works. I'm not 100% sure,right? You take your exact Jell-O, you minus the difference. Oh, I don't think so this works. I, yeah, I don't think so.

Oh, yes, it works. Yes, it works. So what you do is you can take the integral of minus infinity to infinity. So the entire line, you minus exact Jell-O and the approximate Jell-O and you do DX and there is a difference,right?

But the difference is very small,right? It's like 10 to the minus 16. It's very, very small. And notice it's like when you do, when we do fast Triton kernels, I generally use this feature. So you can do integration, integration and derivatives and, you know, you can use Desmos.

So I highly recommend Desmos. And if you do this, that's where we found the problem. It's like, oh, okay, there is some sort of issue. And if you fix it, remember the Jell-O fix does do some effect. It does do some effect.

But remember, we only showed there was only very small effect. So it's not that useful. The rope fix was the most important,right? The rope fix actually caused issues. So you must fix that. And that's the most important fix that you must do.

And finally, there is like some other things that we do. Depending on the precision that you use, there is actually a difference between Float16 and BleepFloat16. And if you do this, we show that Float32, remember we showed before that in the fixes that we did, the lines sometimes go back up,right?

But actually, if you do Float32, it actually does work. If you do Float32 precision, the lines actually don't do separate very well. But once you use Float16, the lines then match up again,right? And BleepFloat16, the lines match up again,right?

So this is just a phenomenon that you're using fast, smaller precisions and that is why you have this problem. But if you do use full precision, you get good results. And the fine-tuning notebook for the Gemma one also works.

So Gemma is two times faster and uses like, I think 60% less memory as well. It's more now. So if you run this, remember you have to connect to, you have to connect to your Google account and you will get this to run.

Any questions on the Gemma one?

Okay. Yes.

Sorry, I came in a little late. Could you just summarize the graph you just showed and.

Okay.

The lines over a number of layers?

Yes. Where did I put the picture? Oh, wait. It's in the blog post. Yes, that's fine. Wait, where did I put it? Oh, it's the first picture,right? Yeah. This one,right?

Yes.

So the X axis is the layer number. So Gemma has 18 layers. So each of those, the X axis just indicates which layer the, which layer it is. The Y axis is log2 norm, logL2 norm. So what you do is you take the original implementation, like DeepMind's implementation, you take Hugging Face, PyTorch, Gemma, like, you know, the other implementations, you check the output of both of them.

So the output, you run the model through, you take output to layer one and output to layer one, the other implementations, and you just find the error. And so this is just the error. And this is log scale.

So when it's log scale, it looks better. When it's not log scale, it looks very bad. So does that, is that better?

So this is you're taking the output of each layer.

Yes. Output of each layer. Yes.

Thank you.

That's called Gemma. So that's for Gemma. For Phi-3, similar. What you do is you open up the Phi-3 implementation, you read through the Phi-3 implementation. And because like you guys like most likely can go through Llama and like just look at it in general, remember delete useless parts of the code, you will see there are differences in Phi-3.

And the differences are, they use other methodologies. They use upcasting, they use stuff. But there was a weird thing that we found in the config file. I will show you

Phi-3 config.

Okay. I'll just use the instruct version.

If you go to, always when you go to like new models, always read the config file,right? config.json. When you open it up, it tells you all the tricks you need to know about the model architecture. And it highly,right, it tells you what is the EOS token ID?

32,000,right? When you look at this, hmm, is that a good idea? 32,000, 32,000. What is the EOS token ID? Right? 32,000. Okay. That's fine. The pad token, no. Hmm, is that a good idea? Like you have to think about like why are these, why are they there?

How many layers does Phi-3 have? It's 40,right? So 40 layers. How many positional encodings does it have? So how long, what is the context length? It is 131072. That's the context length. Remember, it's 100. So this model, the Phi-3 medium is 128K,right?

It's not 128000,right? Just be careful. It's actually 128K,right? It's 131072. There are other issues with this model as well. Okay. This is, okay. That's the, okay. That's probably, okay. Probably don't use the instruct version. The instruct, sorry, we choose the small version.

This is the smaller version. There is a thing we notice is a sliding window. So Mistral has sliding window. Sliding window essentially attends to only 2048 tokens. And this just makes training much faster. And does anyone notice what the problem is for this?

Why is it 2047? Anyone notice any issues? Yes?

Well, it's not a power of two, but.

Correct. So is that weird?

I mean, that's horrible, yeah.

So I did ask the Hugging Face people and they said, yes, it is a bug. So they actually did fix it, but then I don't know why they reverted it back. So I'm a bit confused. They never, they kind of forgot about this.

Oh, yes.

Yeah. So it's actually, it's supposed to be 2048.

Yeah. Because that only makes sense because you're training on 4K, like, you know, two, you're training on the correct context,right? Then this sliding window makes no sense. In fact, I've seen a lot of sliding window bugs recently.

Yeah.

For some reason.

Yeah. Yeah. I'm not sure why, but I'm pretty sure this should be 2048.

Yeah. I'm very confident it's 2048.

I'm actually, I'm 100% sure it should be 2048. Yeah. It's not.

It's very obvious.

Yeah. And yeah. So these small issues are, they need to fix. They still have not fixed. But we Unsloth, it's fixed. So we actually uploaded models which fixed them,right? So if you go to our Unsloth Hugging Face repo, we actually have models which we fixed all of them.

Oh, this is too big. Where is the Phi one? Oh, I didn't put it up. Okay. I need to find the Phi one now. Where's Phi? Oh, there. Phi-3 mini 4K instruct,right? If you go to files, you go to config.json, we fixed it.

Allright. And there's other things that we did to fix it. For example, the pad token ID is 30. Okay. That's actually wrong. Okay. Hmm. Okay. I need to fix my own. Okay. Anyways, there is a bug which we discovered ourselves.

It should be 30. This is actually wrong. Another thing is you must not make the pad token the same token ID as EOS. Never, never, never, never, never. This must be a different token to the EOS token. I do know Unsloth, we automatically fix this during the loading.

It's just the config itself is notright. But that's okay. Unsloth itself is fine. Just the config is a bit wrong. Oh, okay. I found my own bugs, but okay. Yes.

So, okay. Never mind.

Oh, yeah, yeah.

I'm not going to slow down the, you keep going because there's a lot of slides.

Okay. Oh, yeah. Yeah. Actually, this, okay, there's not that much slides. Okay. Actually, there is. Oh, okay. I just noticed there are more. Okay. So another one is like Phi-3 used, they merged the QK and K. Remember we did QK and V?

They're unmerged,right? The weights are separate for the attention matrices. Phi-3 did a very interesting move is that they fused them into one matrix. And we found that to be very problematic for fine-tuning because if you fuse them together, when you do LoRA adapters, you actually, you actually only learn new extra weights and it's very less.

So please unfuse them. And we do this. So our version of the Phi-3 actually unfuses the weights. You must unfuse. Actually, I have to like highly suggest you to unfuse the weights. You can only fuse them if you want to do training faster.

This will make training like maybe 5% faster. It's actually not that much. It's like 2%. You actually increase memory usage a lot. So just be careful of that as well. Oh, yes. They actually did. So this is the sliding window one.

They actually fixed it and then they unfixed it. I think they just forgot about it. I'll probably like push them again to fix it. And this is the fusing of the weights. So we show that if you actually unfuse the weights, so QK and V must be separate.

You must not combine them. If you combine them, you actually have lower accuracy. So please do not do that. For tokenization, remember this slide which I showed you about the smiley faces are like the spaces and each one's a different tokenizations.

There are actually many issues for tokenizations. This is a totally different separate topic from finding bugs and issues in language models. This is a whole topic of its own because tokenizers are very problematic and they're very hard to find and fix.

Did I double this slide? Okay. I doubled that. And also we have new Ollama support, which we have not announced yet, which you can try out. So lots of people have asked us for how do we actually fine-tune a language model and export it to Ollama effectively?

Does people do know what's Ollama or no? Or does anyone not know what's Ollama? Okay. So Ollama is like an interface. When you fine-tune a model, you have to run it,right? You have to run the model somewhere. And Ollama just makes you run the model much easier.

So like you know ChatGPT, ChatGPT is like the running mechanism. Ollama is just like ChatGPT, but they don't have the model. You have to select a model. That's kind of Ollama. Yes.

How did you manage to like, so I've been working on converting, creating model files using the automated pipeline, but we've been found many issues trying to automate model file creation.

Is this using Unsloth though? Or is it using Axolotl or something or other ones? Did you automate their model file yourself or?

Well, we, yeah. We, because we need, we need our own model files,right?

Oh, so we do this automatically now. So with Unsloth, we actually, we spent, I spent like a few, one month on trying to automate the model file creation.

That's why we were struggling so hard as a company.

Yes. We, I have code for that somewhere.

If you could, yeah. Okay. This is a public open source and.

Oh. Yeah. It's already in the GitHub repo. So if you go to Unsloth, you go to chat templates, we have code for that, Ollama. It is still very ugly. So these are the chat templates for, remember the BOS token someone mentioned you have to add it.

Yeah. Add the BOS token. This is the Ollama chat template, which we, so Ollama has a, Ollama has a specific requirement is you must have a chat template because if you don't use the correct chat template, your model will output incorrect, like substandard responses.

So this is the chat template for like some of them. I had to, we had to write chat templates for all of the architectures and we have an automatic one. So these are Vicuna and blah, blah, blah, Alpaca style, Gemma, the Gemma style.

We also have that. We have many, many, even Ollama 3 chat template we have as well. Now for the automatic one, so what we do is we can actually make an automatic chat template, a model file for you automatically.

And this makes your fine-tune much more accurately. Wait, I'll show you the, where is the code for that? Where is the code? Okay. You can see the code is quite large for the, just the chat templates,right? This is just for tokenization.

So it's not even the, yes.

This is Apache 2.0,right?

Yes. It's Apache. Yeah. So it's open source. Yeah. Wait, where is it? Okay. So we have something called parse combined prompt, which does some O of n squared. I didn't actually optimize this. It does O of n squared.

I should have done O of n, but anyways, it's O of n squared. Checking the prompt. Here's the prompt format. So we do, it looks quite ugly, the code for automatic model file creation, but we actually made it so you can actually automatically create a model file from your chat template.

You can see it's quite ugly, but it works. And

yes. Oh, it's even more ugly. Yeah. It's quite ugly code, but unfortunately the model file is very hard to create automatically. And so we have the notebook which allows you to do this. So, so this notebook is in here, Ollama for Alpaca.

So this one's for the Alpaca dataset. And so this is our installation, Llama 3. Where is it? So we, so we'll be using the Alpaca GPT-4, GPT-4 dataset. So you use the Alpaca dataset and you use GPT-4 to create the dataset.

And the trick is though, we also have a CSV file now. So you can actually upload a CSV file and use Unsloth directly to fine-tune a language model. And, but the problem is a language model must have an instruction and output,right?

Only two columns. CSV files and Excel files can have many columns. So what do you do? You have to merge the columns into one. So remember each of those columns in your Excel file, convert them into text. And for example, the Titanic dataset, you merge them to say they have one siblings and spouses and so on,right?

You merge the row into one row and that's what you do. And with Unsloth, you can do this now. It's, I still probably need to edit the, the like syntax calling, but this merging technique says, okay, your first column is called an instruction column and the two double brackets means it's optional.

So if, if the input column exists, then it will say the instruction followed by your input is. And you can like make this very crazy. You can do as many columns as you like. I don't know if the syntax is useful, but like I will probably be editing this.

We're going to make a YouTube video about this to talk about this. This is actually very important for fine-tuning. We noticed that every single provider requires you to use only one column for instruction and one output column. Now you can have infinite columns, how many you like, but you must define the chat template.

And, and we also have a custom, customizable chat template. So before, when you do fine-tuning of language models, you have to use the Alpaca prompt in our other notebooks,right? Below is an instruction that describes a task paired with an input, blah, blah, blah.

You put your instruction here, you put your input here, and you put your output here,right? But notice, what is the problem with this? Is there a problem with this?

So you must only put one instruction and one output or response,right? The input is the problem,right? So how do you solve this? You solve this by merging the input into your instruction prompt,right? So this actually should be removed entirely,right?

And your, your input should be something else. And what you do is we can actually, you're now, we can do this now,right? So you must do, you must put the input and you must put an output,right? You can only use two columns now, but you can use, remember, even though you can only use two columns, you can use this to convert your dataset into two columns.

Yes.

Do you lose any of the semantic meanings though?

Oh, no. I don't think so. No.

You don't think so?

No. I don't think so. So it depends on how you, it depends on how you format the dataset.

Okay.

Remember it's a language model. So you can do, the more you tell the language model what to do, the better.

Of course.

Yeah. But the problem is to do the model file creation, you must do two iterations, repetitions of this,right? You must do instruction response and then you do another one. You must, okay? You must do this for Unsloth. I found this to be very, very important for the model file creation.

If you do not do this, you have dangling new lines and you actually make your model output terrible. So you must do two repetitions of this. Okay. It's a must, must, must. And if you don't do that, we'll error out.

And so once you do this, we also have examples of, for example, this is Llama 3's chat template,right? We again do two iterations. You must do two iterations. Most important. And when you finish training the model, remember you can do run time, run all.

You can do inference now,right? Continue the Fibonacci sequence. Your input is one, one, two, whatever. And the next number in the Fibonacci sequence is 13. I think that's correct. Yes, that's correct. So your language model has learned how to do Fibonacci.

And because it's a chat template, you can also do, you can shove in multiple messages into the model. So this becomes a ChatGPT for you. This is a customized ChatGPT that you can use. And finally, when you want to save the model, you can save it to LoRA adapters.

So this only is 100 MB in size. So once you fine-tune the model, you have 100 MB. But some people also want to like merge the model back and that will take 16 GB. But you must merge this for like Ollama support and GG WAF and stuff like that.

And what we show for Ollama support is you first have to like, you know, install Ollama. You select what you want to save the model to GG WAF. So this is eight bit. We now support multiple quantization methods,right?

You don't have to do eight bit. You can do like four bit, five bit, whatever you like. And this will be saved into one go, much faster. In fact, I think this will save you like 20 minutes of your time.

And we save this automatically. Okay. And this does all the saving, blah, blah, blah, saves. And we also, you see, we automatically create an Ollama model file automatically using a chat template. And I can verify this is actually correct because I tried it.

And then when you want to serve the model file, you can actually print out the model file which we created. And this is the model file. Oops, I pressed run already. Anyways, and finally, to serve it, you can just do a model file to serve it and you can serve this.

And we do have a CSV version. So you can actually use the Titanic dataset. Okay. It's loading. So if you want to use the Titanic dataset, you can upload the Titanic dataset,right? We, I upload the Titanic CSV. You can use the CSV file for this.

And again, you have the merger columns and so on,right? This is a more complicated example. In fact, I should provide this entire example for you for the entire Titanic dataset to merge all the columns into one. And it's the same exact output.

So that's the notebooks that we're sharing for, we did not release this yet. So this is for you guys to like experiment and see if there's any issues. Yeah. And just tell me. We also have blog posts on our website, which you can see our Unsloth GitHub repo.

And we have stickers available and they're very, very cute for you to take. And thank, yeah. And also, yeah, we have Q&A now. Yeah. Yes. Oh.

Did you manage your, did you measure the difference that it makes to write the CSV content in English sentences as opposed to just .co or put them in JSON format?

The problem is if you put them in JSON format, you still need to have instruction and output. So how would you do that? You need to have two columns only for fine-tuning.

Can you do like, you know, your,

your, in your template here, you have instruction and then you add all.

Yeah. You add all of the other columns onto it.

Can you just put, put the same instruction and then JSON of the, of the values of the columns?

Yes. You could, you could do the JSON file. Yes, you can. But we just show you that you can do multiple columns now. So like if you have like 10 columns, you can now make the 10 columns into one by merging them together.

Does that kind of.

There's a big difference in representing that merge columns as an English sentence or like a dictionary?

Oh, no. You can't use, you mean like you shove in the actual dictionary for fine-tuning? I mean, you could do that. I don't, I think you should do English language because a language model predicts the next word. JSON is probably less useful.

Always convert it into English.

I have the same intuition. I was wondering if you measured it in English.

Research paper. Yes. It should be another research paper. Yeah. Any other questions?

So there's a lot of upvoted questions from me on the chat. Sorry. It's a.

Oh, there's a.

I was wondering if you could take a look at them. I don't.

Oh, yeah. I, oh, yeah. I didn't actually check the slider questions. Whoopsies. It actually didn't load. So, oh, there's lots of questions. Okay. I will, okay. Oh, okay. Oh, okay. Oh, okay. I need to, I need to answer each of them afterwards.

I think I'm already out of time though. So yes, thanks a lot.