# Fine tune 20 Llama Models in 5 Minutes: Santosh Radha AI Engineer · 2025-02-09 Santosh Radha, Head of Product/Research at Agnostiq, demonstrates Covalent, an open-source platform that lets users fine-tune and deploy hundreds of Llama models directly from Python without Kubernetes or Docker. By adding a single decorator to Python functions, users specify GPU requirements (e.g., H100 with 48 GB, 18-hour limit) and run them on remote compute, paying only for actual usage (e.g., 87 cents for 6 minutes on an L14, 11 cents on a V100). Covalent supports job submission, inference endpoints with custom autoscaling (e.g., scale to 10 GPUs at 9 AM daily), and automated workflows for training, evaluation, and deployment. Radha shows a workflow that iterates over 20 models, fine-tunes each, evaluates accuracy, sorts, and deploys the best—all from a Jupyter notebook with a single dispatch call. The talk, recorded at the AI Engineer World's Fair, emphasizes eliminating infrastructure overhead for accelerated compute. ## Questions this episode answers ### How can I fine-tune many LLMs simultaneously from a Python notebook without managing any infrastructure? Santosh Radha explains that Covalent lets you add a decorator to a Python function specifying compute needs (e.g., H100 GPU, 36 GB memory) and dispatch from a notebook. To fine-tune many models, loop over them in a workflow; Covalent parallelizes execution on the specified hardware. You avoid Docker, Kubernetes, and manual GPU provisioning. The demo fine-tunes 20 models, but it scales to hundreds. [1:03](https://aie.addtry.com/14d60791-ffee-43d7-a7e9-1159ee3360f8?t=63000) ### How can I deploy a machine learning model with custom autoscaling without using Kubernetes? Santosh Radha describes using Covalent: define an initializer and endpoint like `/generate` in Python, then issue a `cc.deploy` command in your notebook. Covalent provisions the service on its cloud, returning an API endpoint that autoscales per your rules—e.g., when GPU utilization hits 80% or at 9 a.m. daily to 10 GPUs. It scales to zero when idle. No Docker or Kubernetes configuration is needed. [2:36](https://aie.addtry.com/14d60791-ffee-43d7-a7e9-1159ee3360f8?t=156000) ### What does it cost to run a fine-tuning or evaluation job on Covalent's cloud GPUs? Santosh Radha reveals real costs from his demo: evaluating a model on an L14 GPU for 6 minutes cost 87 cents, and on a V100 GPU for 6 minutes cost 11 cents. You pay strictly for the GPU seconds used, with no infrastructure surcharge. This pay-per-use model makes it cheap to experiment with many models. [5:43](https://aie.addtry.com/14d60791-ffee-43d7-a7e9-1159ee3360f8?t=343000) ## Key moments - **[0:00] Intro** - **[0:47] Covalent** - [0:47] Covalent enables running Python functions on remote GPUs by adding a decorator to specify GPU type, memory, and time limit, then shipping code from Jupyter. - **[1:53] Primitives** - [2:20] Covalent deploys inference services with authentication and custom autoscaling via a single cc.deploy command in a notebook. - **[3:00] Demo Setup** - **[3:40] Fine-Tuning** - **[5:06] Workflow** - [5:06] Covalent workflow fine-tunes multiple Llama models on specified GPUs, evaluates accuracy, sorts results, and deploys the best model, all from Python. - **[5:43] Results** - [5:43] Covalent evaluation of a model on an L4 GPU for 6 minutes costs $0.87; on a V100 costs $0.11. - **[6:13] Outro** ## Speakers - **Santosh Radha** (guest) ## Topics Fine-Tuning ## Mentioned Covalent (product), Docker (product), H100 (product), Jupyter notebook (product), Kubernetes (product), Python (product), V100 (product) ## Transcript ### Intro **Santosh Radha** [0:14] So the talk is actually going to be about how you run things extremely easy, directly from Python. And the example that I'm going to show you here is— obviously I just have 5 minutes on my end— but I'm going to try my best to showcase how you can fine-tune pretty much 20 is an arbitrary number here, but hundreds of models that you can doright from Python without needing anything like Kubernetes, Docker, or anything on your side. So before that, you can find the talk and the actual code for what I'm going to do in this QR code. And you will find a lot more interesting examples over there to try out and run as well. ### Covalent **Santosh Radha** [0:47] OK, so what do we do? So Covalent is an open source/open core product on its own. And what we do is we help people write Python locally and ship the code to any kind of compute back then that you need to send it to. So what that means is, hey, you have a Python function that you want to run on a GPU. In your local laptop, open up a Jupyter notebook, add a single decorator on top to say, hey, I want to run this on H100 with 36 gigs of memory for two days, maximum time limit. And press Shift-Enter in your Jupyter notebook. And that's it. The code gets shipped to a back end in a GPU, and you get back the result on your side end. In the open source case, it sends it to your own compute. You can attach your own compute cluster, and it runs over there. In the cloud case, it runs in our GPU cluster, and you just pay for the GPU time that it runs on. So it runs for 5 minutes, you pay for 5 minutes of H100. It runs for 10 seconds, you pay for 10 seconds of H100s on your side end. You can also bring your own compute and attach to us, and we'll help you orchestrate the entire compute that you're handling on your side, be it your own cloud or on-prem systems or whatever it is on your end. ### Primitives **Santosh Radha** [1:53] OK, so Covalent basically has a bunch of primitives that you define in. You can submit in jobs, which are called single functions. So essentially, all you need to do is, as I said, add a single decorator on top and say, what is the compute that you need to ship it to? It goes there, it runs, and you get back the Python object back. And you just pay for the function that you are running in. We also let you run inferences. And again, it's completely Pythonic. You don't Dockerize. You don't run Kubernetes cluster. You don't do anything. You just say, hey, I have an initializer function, and I need an endpoint called /generate. And you define your Python functions. You click a single cc.deploy command in your Jupyter notebook. The entire service gets shipped to us, and we scale. You get back an API endpoint that scales to zero or scales in its request as and when your new request comes in. You can define your custom autoscaling mechanism, like, hey, I want to autoscale it to 10 GPUs exactly at 9 o'clock every day. Or I want to autoscale whenever my GPU utilization hits 80%. Or I want to autoscale whenever the number of requests I get in is 1,000. So you can define whatever autoscaling you want. You can define authentication and everything. And everything happens in the background for you. You don't even touch a single code of Kubernetes or Docker or anything on your side end. And the talk I'm going to give in is a very tiny example of what we do from our side. ### Demo Setup **Santosh Radha** [3:14] But if you go to this link in, there's a whole host of examples that you can run in,right from real-time time series analysis to using inverter transformers for time series, which is like a state-of-the-art time series transformer on its own, running in large systems, large language models on your serving systems, and even building an entire AI model foundry out of just pure Pythonic code on your side end. ### Fine-Tuning **Santosh Radha** [3:40] So without further ado, I'll quickly run through the code example of how you do essentially fine-tune a bunch of huge set of models directly just from Python on your end. And I'll also show you how it looks like from the front-end side as well. So it's rather simple. All you do is, I have written a bunch of normal Pythonic training functions in my local package called fine-tune and evaluate on our end. And what we are going to do is, hey, I'm going to define a Python task which essentially calls in my fine-tune function, which is going to accept a model and data and return back a fine-tuned model. So this is a simple Python function. And I'm going to say, hey, I want to run it on a 24-core CPU with one GPU in it of type H100 with 48 gigs of memory. And I'm going to give a max limit of 18 hours on it. And then I'm going to say, hey, once the model is done, I'm going to accept the model and then evaluate its accuracy on its own. And finally, I'm going to just sort the model among all the best models and then pick the best model in it. And I want this to run on a CPU-based machine. I don't want to waste GPU for my sorting or whatever I'm going to do on my end. And finally, I'm going to deploy the best model that I figured that has performed well on my end. And this is, again, a simple decorator to say, hey, this is my initialization service. And I'm going to create an endpoint called /generate. And I'm going to generate the text and give back the prediction. And finally, this is where the magic happens. To tie together all of these things, what I do is I'm going to create a workflow where I'm pretty much just going to simply loop over a bunch of models to fine-tune, call the fine-tune function, evaluate the task and get the accuracy, make a list of all the models and accuracy, sort the best models, and then deploy the model from my end. ### Workflow **Santosh Radha** [5:24] And this is completely Pythonic. And once you dispatch this to our server, which is essentially calling a single line over here, what you will go back and see is a new job that creates in our application. And all of the functions that you called will run in the respective devices that you just defined. So for instance, here is one of the evaluation steps that ran in. And it has its own machine that we ran in. It ran in L14. It ran for 6 minutes. And you get back just 87 cents to evaluate your model in. ### Results **Santosh Radha** [5:57] And another model ran in V100 on its own. And it ran for 6 minutes again. It cost 11 cents to do it in. And in total, you finally have deployed, fine-tuned, and trained completely in Python without needing anything like Docker or Kubernetes on your end. ### Outro **Santosh Radha** [6:13] And we have a booth over there. Do visit us, and we can have more chat over there. Thank you, guys. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.