Robots as professional Chefs - Nikhil Abraham, CloudChef

Hey everyone, I'm Nikhil. I'm the co-founder and CEO of CloudChef. Today I'm going to tell you guys how we took a general-purpose robot that was not meant for cooking—it was just a robot with two hands—how we trained it, or put it through culinary school, and it's now a professional chef that's working in various different kitchens doing actual, real work like a chef.

So before we get into that, a quick thing about CloudChef: our mission basically is to make high-quality, nutritious food affordable to everyone, and the only way we know how to do it at this point is by automating all commercial labor—or all commercial kitchen labor—with what we call culinary intelligent robots, robots that can act, sense, and reason, and behave in the real world like a chef.

So you guys probably all have seen the Tesla Optimus dancing, and I've seen it too, and the immediate question that comes to mind is, maybe this is how a robot chef will be, where it's in the kitchen, it's beating down equipment, whatnot,right?

But it turns out that those guys are a little too expensive, they're not really there yet, there are lots of problems with humanoids. But on the other hand, if you look at form factors like these, they're also general purpose.

They're basically just two hands and a mobile base that can move around and actually do all the work that a regular chef would be able to do. And as compared to humanoids, these are now way cheaper than human labor.

Humanoids, if you plot them on this curve, you'd probably not even see them at this point because of how unreliable they are, how much maintenance it requires. But these wheeled robots with two hands? No problem. Way cheaper than a human—way cheaper than any human chef.

But what's missing is actually

software. And what we did was we took that—we took this robot, like I said, we put it through culinary school, and now we have chef-like robot labor. So commercial facilities can hire this robot, pay it hourly wages like $12 an hour, it'll always show up, no overtime, no turnover, no calling in sick.

And just—or even better than a human—it plugs in place into any arbitrary novel kitchen. So it learns new recipes from one expert demonstration, and it is robust to ingredient variation, appliance variation, and can cook on arbitrary portion sizes.

This is actually a task that's actually harder for humans to do too. Like, when I say we put these robots through culinary school, what we actually do is—like, what is culinary school for a robot? It needs to learn all the motion primitives that come with human beings.

So how do you pick something, how do you stir a pot? And to do that, we have these robot foundation models that we fine-tune, we have teleoperation to fall back for all these edge cases. But that's not enough.

You still need the robot to understand food. Like, are the onions brown enough? Are the onions brown enough if you're cooking steak? Is it shrinking well enough if you're cooking shrimp? Can you sense when the shrimp is done?

And these are ingredients that vary seasonally, daily. Like, onions today might require 7 minutes to sauté, it'll require like 9 minutes to sauté tomorrow. And we basically have thermal and visual embeddings that are specific to cooking that help us reason through these unseen environments.

And we've basically modeled recipes as state machines based on these embedding models at the core. And now, even if—even after you have this, the next thing that you need is—it needs to adapt to any new kitchen that it has never seen before.

So it needs to be able to see a recipe once, understand what to do, and interact with real humans in a workflow, and actually do work. So we put our culinary understanding to the test, and we at this point do better than even expert chefs in their cuisine of training.

And if you take—so basically we evaluated more than 1,000 recipes across a mix of cuisines. We got given—the task was given live cooking data, an expert demonstration, and a text recipe: can you estimate where in the cooking process you are?

Can you track progress like a human being? We put it through this. Expert human chefs, who get paid more than $150,000 a year, still perform worse than our tiny model that's doing perception in this case. And in fact, when we put state-of-the-art models like Gemini 2.5 or O3, they actually perform way worse than our own models, and that's partly because they don't have any thermal modality.

And the thing is, thermal modality does not have internet-scale data. So what we did is we went and installed sensors in active commercial kitchens, collected hundreds of thousands of—collected data worth hundreds of thousands of live cooked meals in various kitchen environments across various different recipes, cuisines, and seasons.

So we collected this private data, we trained a model, we scraped a bunch of public data, trained some self-supervised models on that, and a combination of this is basically what our culinary system banks on. And it is what, like I said, is now way better than human chefs at just decision-making during cooking.

But motor skills, on the other hand, it's not as good as a human, but it is getting there. So we again put it through all these different evals. Sautéing, it's almost as fast as a human cook. Picking and pouring, slightly less fast.

Grilling, stirring, so all a bunch of evals that we did on top of motor skills. And this is how—in fact, our system isright now about 95% autonomous, 5% teleoperated, and it's way faster and way more reliable than just teleop or just foundation models.

And basically the robot comes into a kitchen, like I said, looks at a recipe once from a chef, and it's just able to do it. So for example here, it's cooking a recipe from a two-michelin-star chef who's based out of San Francisco.

And basically while it's cooking, it's looking at how the onions are browning, it's comparing it to how brown the onions were getting when the chef was cooking it, takes it to theright amount of brownness, it knows exactly what to do for the next recipe, where the ingredients are kept.

It's not pre-programmed to know where the ingredients are, what kind of variation you'll find. It is doing all that reasoning within the system itself. So yeah, so if we

go further under the recipe, we'll see how it's cooking this chicken. It's basically getting clean readings every single—every few minutes. And at the end of it, I will basically show you what happens. And yeah, at the end of it, you have actual—so these are actually recipes that go into the stomachs of actual real customers.

So the robot's cooking at various different facilities at this point. It's

deployed in the real world. And yeah, so it's deployed in the real world, it's being used in all these sorts of kitchens. On theright, you can see it cook recipes in our in-house kitchen. On the left, it's also CCTV footage of the robot doing some operation.

I'm not even sure, I just pulled it off the CCTV before getting on stage and just pulled it up here. And this is video from a couple of months ago where the robot's doing regular cooking like a human being.

And outside of our own facilities, this is how, for example, the robot's working at one of our customers' facilities doing chicken wings. It's basically fetching the chicken wings from some place kept to the side, waits for the cook to be done.

Now it'll basically collect the cooked chicken, put it inside a bowl, and goes ahead, sauces it, and mixes it like a human being. And while doing this, the robot has—a robot is practically a wing scale itself, so it knows exactly what amount of ingredients it has put in, it knows how much it has stirred.

And yeah, so basically we are CloudChef, like I said. At this point, we are hiring—we are a very small team, we are growing super fast, and we are looking for people in software, ML, and robotics. If you know anyone, please reach out to me.

My email address is nikhil@cloudchef.co. And yeah, thank you. If any of you have any questions, I'm happy to take it. Thank you.

You said that it's almost the same as human. How are you benchmarking success?

So for us, success means two things. One is how good is the robot at understanding what's happening in the cooking process. So very simple intuition for that is, okay, if you give the entire cooking feed to a human being, and if you give the entire cooking—like video and infrared feed to our system, which estimates state better?

Because once you have a cooked recipe, you can use that as labeled data to understand, okay, if the system predicts that this is 40% done, was it actually 40% done, or was it actually 50% done? That's actually a supervised learning signal that we can get after we

have data from recreations—like any food recreation from any chef with thermal and RGB footage, we're able to do that. The other part is motions. How fast is the robot able to do physical motions as compared to a human being?

Which, I said, we are not as good as human beings yet. It's basically a data problem. The more data we get, the better we—the better and faster we get at doing any individual task inside a kitchen. Does that answer your question?

Yeah, well, I was thinking of the end result, the meal that's produced. Are you guys benchmarking that versus what a human can do? How does that play into your—

Yeah, so for the end taste, so the thing that we realized is, as a professional—no professional chef is cooking to chemical—to consistency that can be measured in any chemical way. So our competition is not getting chemical-level consistency every single time.

It's about getting consistency to a degree that is better than a chef can do a second time. So a common benchmark that we do is we get a chef to cook a recipe once, and then we get our system—we get our robot recreate that recipe a couple of times, and then we do blind taste tests.

And so those are more unscalable evals that we do in-house, which act as a higher signal to, okay, actually the end product that we get is better than what chefs are able to do.

As far as what you're going to retail, what does it include? The arms for movement, the base that is motion, can move around any kitchen. Those screens in that video, are those also part of the unit?

No, it's basically just two hands on a mobile base with some cameras and stuff on it. It shows up at the kitchen. You basically interact with it like a human being, and that's the form factor. There's no additional screens, etc.

Those are just for video's sake.

Does the robot currently do measurements, or do humans have to prep for measurements?

It depends. Ideally, humans don't need to, but today in some deployments, humans do end up doing it. But our idea is that because the robot—because we have joint data from all the different motors from the robot, the robot itself is a weighing scale.

So when it picks up something, it already knows how heavy it is.

And then my last question would be, in a commercial kitchen, do the commercial appliances have to be altered to work with the robot?

So that is one thing that we've worked on a lot, wherein we are able to work on arbitrary unseen appliances because our sensing stack is so good. And the other thing is almost all appliances inside kitchens are controlled using knobs.

So the motion primitive that the robot needs is to know how to turn a knob, and then our control systems take care of it from there. Yeah, sorry.

Does this have wheels where it can move automatically?

Yes.

Yeah.

So I'm curious about speed, mostly because I think this is incredible stuff so far, but I'm imagining a kitchen with a lot of these devices. How fast do you think you guys can goright now versus like a year?

Soright now, for most motions, we are anywhere between like 80 to 95% the speed of a human being. And ideally, there's nothing stopping robots from being even faster than human beings. It's mostly just a data problem. Right now, the reason why it's not as fast as human beings is because the data that we collect on these robots are done by human beings who teleoperate the robot.

And because human beings teleoperating the robot are not as intuitive at teleoperating the robot as their own bodies, they're not as fast as—so the data is kind of slow. And then over time, we expect with RL and stuff, it'll be faster.

Follow-up to this graph, you don't have chopping on here, which I guess makes sense. What is your vision for the dice?

Yeah, so there's nothing stopping a robot from doing that either. It's just we don't have—we haven't gone out collected data for those tasks yet. So it's just something on our roadmap. We are very much planning to do that.

Where can we eat this?

Oh, you can eat this in Palo Alto. So if you're in San Francisco, if you order from Wingstar, that's a customer of ours who uses it, so you'll get it from there. If you're in Palo Alto, you can order from India's Top 20, and you can eat it from there as well.

Or if you're in Menlo Park, you can go to this high-end Indian restaurant called Elan, and some of the food there is also cooked by it.

We've asked questions around chopping and food preparation and whatnot, and speed of the robot. But in terms of throughput in the actual process, how much of that even matters? How much of the energy already goes in throughout the day into prep versus the 90% or 80%?

Does that matter? This is not a manufacturing facility. When it comes to servicing, how much of the economic value is already taken care of because you have the teleoperator in the back to make sure things are insured? Have you guys found that meaningful, or is that not a big deal at all and not a—is that trivial, essentially, at this point?

Great question. So basically what—the quick answer to that is about 50% of the labor costs inside any kitchen is line cooking labor, and that's where we are going at first. And the advantage there—I mean, speed does play a factor, but there's another variable that we have in our control, which is we are able to speed up recipes more than any human being is able to do because we know exactly—we've had several instances where we've recorded—we've observed a chef in motion and realized that, oh, this process that takes them 20 minutes to do can actually be done in 14 minutes.

So if the robot is even like 10% slower, it doesn't really matter. That's how it works.

And then a tiny follow-up to that is, I guess the robot can work longer in theory, at least? Do you prep overnight then, or how does that work and whatnot? Because I know you can—are there some recipes you work more than 40 hours a week?

Does this help with the throughput process, or is that just unit by unit when ready?

Yeah, unlike a human being who after working 40 hours a week goes into overtime to literally—a robot can work for like 168 hours. There's nothing stopping a robot from working 24/7. The practical constraint is most facilities don't operate 24 hours, so the robot will operate as long as the facility is operating, and then there are some tasks that you can do overnight.

So once we get into cutting, chopping, etc., the robot will just be doing that overnight before the actual stuff comes in.

Sorry, mine was kind of related to before. So did you find new bottlenecks in things like dishwashing or cross-contamination, stuff that you maybe weren't expecting to deal with this process?

So dishwashing, etc., not that much. And even for things like cross-contamination, we just put small gloves on the robot, and then our customers switch that out every day. These are washable small silicone pads. I can pull up a video on that, but basically that's how we take care of it.

And then for things like dishwashing, those are not tasks that we are envisioning doing in the short term. We want to do more of the tasks that actually add to the quality of the food that's being put out.

So that's why we are mostly focused on line cooking for now, maybe sometime later prepping, chopping, etc.

Last question.

Oh, yeah. Sorry. So it learns from chefs,right? The recipe is from chefs. Is it able to modify steps of a recipe to cook things faster?

So that is still in experimental phase. There are cuisines in which we are able to do this really well, but we aren't yet able to do this across all cuisines. So for cuisines where the thermodynamics modeling of what's happening in the process is straightforward, it is much more easier to basically speed recipes up, do minor variations, etc.

And there are some cases where it's not that easy. It is still experimental territory. We are still working on that. Yeah, last question. I'll just take her. She's been on.

Is the human responsible for making sure that all the ingredients and the control are available? What happens in the example if you are responsible? Does the robot solve all of this?

In the current version, it alerts somebody in the facility that the robot needs ingredients to work, and then they take care of it. Hopefully, once there are enough robots in the facility, they'll just talk to each other and—thank you so much.