Robotics: why now? - Quan Vuong and Jost Tobias Springberg, Physical Intelligence

So, good morning. Uh, thanks for being here with us. My name is Quan, this is Toby. Um, our mission is to make a model that can control any robot to do any task. Now, this is not something that's ready today.

We believe that there are multiple scientific breakthroughs that need to happen for us to get there. And so we're very open: we publish our research, we open-source our model, and we talk very publicly about what it is that we do.

And so, if you think about robotics before—not to say that it's not useful, you know, it's incredibly impactful on the world—but the scenario that you often see robotic in is either in a very constrained environment, such as, you know, a factory, very repetitive motion, very structured environment.

And then when you try to bring them into the real world, just kind of like semi-structured. I think this is a pretty kind of well-known video of a full-bodied humanoid, you know, struggling to perform a somewhat simple task.

Um, it was from some time ago. If you look at robotics today, what do you see? You see—

you see kind of humanoid dancing. I don't think I can do that dance move. Um, um, I'll try. And so, you see kind of very complex physical motion that robots are capable of. And you also see this video on theright, which is what we released late last year, of a robot operating with kind of somewhat semi-structured objects.

You know, this is clothing that just came out of a dryer that ran before, so it's very hard to control, you know, the exact initial scene of, you know, how the shirt should be. You know, it takes out all of the shirts, manages to put them in a basket.

And in the full video, you know, it goes on to bring it to a table and then complete the folding. So what really changed? You know, the obvious answer is that there is this AI wave that we're all riding on.

Robotic benefits a lot from kind of general AI development. But there is also this vision language action model that we're pioneering. And so what are they? I'll pass it off to Toby.

Cool, yeah. So vision language action models, what are they, as Quan asked? Well, you probably all know by now what a multimodal LLM, or we often refer to it as a vision language model, or VLM, is by now.

Essentially, a VLM generally takes texts and images as inputs. You have some sort of prompt for the model. And then it embeds them and passes them to a transformer model to autoregressively produce an answer in text,right? So you interact with these models probably as I do, every day now.

So, and then a VLA is essentially an adaptation of a VLM for the purpose of robotics. The model additionally gets inputs describing the robot's state, such as its joint positions. And instead of asking questions about what's going on in the scene, we ask the model to produce actions to control the robot directly,right?

So if VLMs and VLAs are so similar in kind of principle, then what are the additional engineering challenges that we face when we try to train such a VLA? To understand this, I want to contrast a little bit pipelines for VLM training to what we have to do when we train a VLA for robotics.

So when you, as kind of like a downstream customer of these, like, big models that have been trained, want to use a VLM, you generally kind of take a model and you can source data from the web. And maybe you have a little bit of extra data that you have for your specific task that you're interested in, that you supplement.

And then you probably train and use an off-the-shelf model, and you fine-tune it on a large cluster somewhere in the cloud. And then finally, you can use well-established libraries for inference and deployment in the cloud,right? So all of this is, like, probably bread and butter for a bunch of you.

Now, in contrast to that, if we want to do VLA training, you want to train a model to exhibit kind of dexterous frontier-level behavior. Then it's an open question what the analogy of data source for the web actually is,right?

And we believe that this is kind of a trillion-dollar question for the industry in some sense. And it's an entirely open resource question, as Quan already said. And then secondly, while we can use VLM backbones so we can reuse some pre-trained models, we typically have to adapt them and adapt the model architectures in order to allow for models that can control robots at somehow high-frequency controls that we need to actually make progress.

And then finally, we there is also no standard solution for deploying large robot policies in multiple locations, on-premise, on-device for robots,right? So this just doesn't exist. I won't really have time to talk about the third thing in detail here, but I want to delve into a little bit about data and model training for robotics that we do at PAI.

So first, how can we design a data engine that enables robust, highly dexterous policies for difficult tasks with robots? At PAI, we kind of believe that there currently is no standard solution for this at all that would enable us to do this.

And so we're building essentially a data engine from the ground up, from zero. We are designing this pipeline to get us very quickly to some sort of impressive capability. And I hope at the end of the talk you agree that it looks somewhat impressive.

But also to enable significant scaling in the next few years. So we've seen firsthand that operationalizing this pipeline is actually kind of one of the main ingredients. It's probably more than 50% of the work is getting the data pipelineright, getting theright data, getting it to be high quality.

So how does it work? Well, we typically start from a set of ever-expanding tasks that we pick to test what's possible in the moment,right? So these are tasks such as folding clothes, buying groceries, many other tasks that we're interested in.

And we then have human operators control our robots using a custom runtime and teleoperation system. You can see the system in this video here. So what happens there is there's human operators, and they are controlling what we call leader arms, where they basically trace out motions with their arms, with robot arms kind of strapped to their arms.

And the motion gets transferred via software to the actual robot end-effectors,right? And this way you can demonstrate fairly intricate, highly dexterous tasks and collect that data for training afterwards.

And then afterwards, we have a bunch of facilities to do kind of tracking of metrics of what's going on at the moment. I think there's another slide here, yep. So we basically schedule these data collection sessions all the time.

And each dash in this dashboard here shows from this week this is, I think, from Tuesday, an operator doing a specific episode for a specific task,right? So we collect a lot of this data all around the clock, basically.

And then we annotate that data in the cloud. We serve it in big buckets there and can filter it back down based on annotations and use it for model training. After we've trained, we then get policies that can actually solve the demonstrated tasks autonomously.

So here in this video, you see a model that is PAI-0. So this is the model that Quan referred to earlier already that we released late last year. And as you can see, it can kind of do these, like, fairly impressive, highly dexterous tasks such as shirt folding.

OK, so how far have we gotten by following this approach? Well, when we started, the biggest publicly available data set was the OpenCross Embodiment data set, which contained about 3,800 hours of data, largely from static scenes in different robot labs around the world.

After kind of running this data pipeline that I've described for six months, we had collected about 10,000 hours of successful episodes this is successful data in tens of environments, covering hundreds of different tasks, enabling, for example, the kind of shirt folding policies you saw before.

Oops.

Cool. Is that video actually playing? Or is that video not playing?

Can I get it to play? It can't.

Let's see what we do with the next video. And then after another six months oh, this please. We have collected many more hours of data in static scenes, but crucially, have also started to collect significant amounts of data using mobile manipulation setups, such as the ones you see here.

So the data now spans many, many more tasks and importantly has massively grown in diversity, covering hundreds of different scenes and environments. And as you can imagine, this scale and diversity enables new leaps, as you can kind of see here by this policy that is already running autonomously.

And I'll go into detail a little bit how this works, but also brings lots of additional engineering challenges. So now that we have kind of described how we get the data, then the question is, what kind of capabilities can we elicit with this data in VLAs,right?

And to understand where we are today, I think it's kind of useful to draw an analogy between industry trends for VLMs and VLAs, which have been kind of ongoing in the last three years. So first, for multimodal LLMs or VLMs, we have seen a constant stream of improvements over the last three years, kind of starting from initial conversational agents that you've all interacted with, all the way to the RL-trained, multimodal reasoning models and coding assistants that we all have today,right, and we all use.

For VLAs, they follow a similar but time-lagged trajectory, basically. So initial VLAs such as RT2, for example, done by some of my colleagues that are now at PAI, emerged in 2023 after LLMs had already been enhanced with vision encoders.

And in fact, actually, some of the earliest multimodal LLMs were trained for robotics purposes by some of my colleagues, Danny and others that are now working with us at PAI. These were kind of impressive as first proofs of concept and showed some generalization capabilities.

So you could kind of, like, ask them to pick different objects in the same kind of scene that kind of is in the training data. But they are generally held back by a lack of available robot data. But, you know, nonetheless, they sparked this big explosion of interest in the field, which is probably why you're here today.

Then in the mid-20 so mid-2024 towards ends 2024, the first kind of really dexterous multi-robot VLAs appeared,right? And the industry at large has now produced several of those. There are models, for example, such as Gemini for robotics or NVIDIA's Groot models that I think you'll hear a little bit about later as well.

And our entry in this category was PAI-0, which we believe is kind of perhaps the most dexterous multi-robot model that you can use. And in fact, it's open source as well. So these models generally adjust architectures to produce actions via diffusion to enable kind of very fast generation at high frequencies you need for robot control.

So if that's where we were then, what's next? Where are we now,right?

So for us, the next leap was to study just how exactly model capabilities change when we increase data collection diversity. And this led us to develop PAIO-5, which is basically a VLA with open-world generalization. And I want to talk a little bit more about this.

So what does this look like? In general, we have expanded massively, as I kind of said, the data we take in during training. It now consists of both static and mobile robot data on theright here, as well as an extended set of multimodal VLM data, such as data from the web, object detection data, and general language annotations for the robot data that we've collected.

So we have a huge annotation pipeline as well. And this is what's on the left here. We then feed this data into a specially designed VLM, which starts from a pre-trained transformer model and is expanded with an action expert transformer to theright.

This VLM part, so the big backbone of it, is trained to give predictions for both general questions about the scene now, but also to subdivide high-level requests that a human might have for the model, such as, for example, clean my bedroom,right?

So it is trained to subdivide these tasks into subtasks such as pick up a pillow in the case of cleaning the bedroom. And at the same time, the action expert transformer can attend to the internals of the large VLM and can run at much higher rate and produces the actual continuous output actions via diffusion flow matching objective, basically.

Cool. I hope this video plays. Can we get it to play?

Yes. So training this architecture on all our data then leads us to a VLA that can perform difficult long-horizon tasks of up to 10 minutes in each episode. And this is, I think, much, much longer than what we've seen before,right?

And it can do this in entirely unseen homes, showcasing perhaps the first sign, like, the first true sign that broad generalization can emerge from VLA training. So in this video here, you see my colleague Chelsea prompting the model in an, yeah, you know, an entirely new home that is not in the training data to essentially perform multiple cleaning tasks, such as cleaning a surface here in a kitchen.

Cool. And to understand this ability to generalize a little bit further, we tested how it emerges by training PAIO-5 on a fixed number of data, but varying the number of homes from which data was introduced during training, basically then testing it in a held-out location,right?

And as you can see, with increasing amounts of locations added so this is the yellow curve here. Performance generally increases in the test scene, as you would expect, until it surprisingly, perhaps, matches and even slightly surpasses training with the held-out scene specifically,right?

So this was a very cool result for us because we saw that we could kind of expand with more and more training data collections in different homes the capabilities of the model in new environments, which I think is pretty cool.

And then here we see the same model performing a bedroom cleanup task in a new home. And in this specific case, Laura, a colleague of mine, is prompting the model only to basically clean the bedroom,right? And you see here the power of these capabilities of subdividing into several tasks, such as, you know, throwing trash in the bin and then cleaning, making a bed, basically.

And as you can see at the bottom with the timer, you know, this policy is autonomously collecting controlling this robot for multiple minutes at a time, basically.

Cool. And with that, I give it back to Quan to talk to you a little bit about partnerships.

Thank you, Toby. So what you're seeing is a robot that we have never seen in person. We mean the team at PAI. We've never had access to it. This robot is running very, very far away from where our office is.

It's performing a somewhat interesting task of making a cup of coffee end to end. And it works pretty well. We didn't need to kind of iterate many times to be able to produce this video. Now, why is this important?

When you think about robotic, you know, oftentimes you would think about hardware. You would think about kind of real deployment challenges. Now, those are important, but it's our belief that actually one of the main bottlenecks is software and just model intelligence.

And if you think about what if successful would scale with maximum velocity in the sense that, you know, suddenly next year you have thousands and millions of robot deploy, it's really about demonstrating the hypothesis that our model can run across many different hardware platforms out there without us having to invest significant time and effort into trying to work with that hardware platform.

So I think the demonstration that we've never touched this robot before, we don't know how it works internally, and yet our model can control that robot to perform a fairly interesting task is a piece of evidence in that direction.

We have many more evidence here. And so that's why it's important. We are also very open when we work with other companies because we believe that, you know, the problem is far from being solved. So when we work with this company, for example, that you can ask, how do we run inference?

We literally sent them the model checkpoint for them to run, like, inference with. And, you know, we have very low-level technical discussion with them. This is to say that, you know, if you think there is a company that we should be talking to, please let us know.

You can let me and Toby know in person. And also, you know, just shoot us an email. Yep.

And if you ask, you know, what is our biggest bottleneckright now? Because we're after this mission of building a model that can work on any robot to perform any task, it's a scientific problem, engineering problem, operation problem that are far from being solved.

And so our biggest bottleneck really is we need the best people in the world in this area to help us accelerate progress. And so for a research organization, you know, any role that you might think we might need, we're hiring for it.

Even if there is a role that you think you're exceptional at that, you know, we don't have on our website, please feel free to also let us know that, you know, you should really be hiring for this role and happy to have a conversation with you about it.

Again, you can talk to me and Toby in person about, you know, what is how hiring needsright now. But you can also apply online and shoot us a DM on Twitter. Thank you for listening.