Creating Agents that Co-Create — Karina Nguyen, OpenAI

Hey everyone. My name is Karina, and I'm an AI researcher at OpenAI. Before that, I worked at Antarctic for about 2 years, working on Claude. So today I would love to chat more about what kind of scaling paradigms that has happened in the past 2 to 4 years in AI research, and how those paradigms unlocked new frontier product research.

I'm also going to share some of the vignettes from some of the lessons learned by developing Claude and ChatGPT products, some design challenges and lessons, and how do I think about the future of agents as they become from collaborators to co-innovators in the future.

I would also love to invite you to engage in the conversation, so I'd be more than happy to answer some of the questions at the end. Cool. So, not sure if probably the majority of you know this, but I think there are two scaling paradigms that have happened in AI research over the past few years.

The first paradigm is the next token prediction, and you might have heard this as called pretraining. And what's really amazing about the next token prediction is that it's a rule-building machine. The model learns to understand the world by predicting the next word.

And I think fundamentally, if you think about it, this is happening because certain sequence is caused by initial action, and this is irreversible. And so the model learns some of the physics of the world to understand. And the token can be anything,right?

The tokens that we pretrained are strings, words, pixels. It could be anything. And so to predict what will happen next, the model needs to understand how the world works. And this is why pretraining works. And so you can imagine the next token prediction is the massive multitask learning.

And what's amazing about this is that during pretraining, some tasks are really easy to learn, such as translation,right? Like the word "boarding" in French is? The model also learns a lot about the world. The capital of France is Paris.

And because some of the information is much more present on the internet and in some of the knowledge artifacts, the model has much easier time to learn this. But actually, the reason why compute is so important and scaling compute in the pretraining stage is so important is because there is a new class, there is a class of tasks that is really, really hard to learn.

And for example, the model learns a lot about the physics. It learns so much about the problem solving, generation, and the logical expressions. It learns some of the spatial reasoning, although it's not perfect.

But we are getting to the complexity of the tasks, such as math, when the model has to compute this number during the next token prediction. It's actually really hard. So that's why you need chain of thought or spend more compute on the chain of thought to help the model to reason through more computational such tasks.

Another class of tasks that I was thinking a lot about is creative writing. It's actually really, really hard. And the reason why it's so hard for the model is because, you know, you can predict very nicely the style of the writing, but a lot of the creative writing is actually role building and storytelling and the plot.

And it's much, much easier for the model to make a mistake for the next token prediction in such a way where it will completely deteriorate the plot coherence, which is really important for the stories. And this is an open-ended, like, research problem, creative writing in itself.

And the reason why is because it's really, really hard to measure what is good creative writing, what is not creative writing. And obviously, we would love for the models to invent new forms of writing and be extremely creative in their generations.

But this is actually one of the hardest AI research problems today is how do we make models to, like, write novels and have coherent stories over the long course of the period of time.

So I think the era of 2020 to 2021, that was an era of scaling, pretraining a lot, both at Antarctic and at OpenAI. And actually, the first, at that time, one of the first products was GitHub Copilot. And I thought it was a completely interesting product, the autocomplete, because it's so in pretraining, the model has learned so much about the code and the next token prediction for the code by the billions of code tokens using from GitHub, open source projects, et cetera.

And what has happened for the autocomplete tab, tab, tab in the cursor or GitHub Copilot is that the researchers pretrained via RLHF, reinforcement learning from human feedback, and then reinforcement learning from AI feedback to make it extremely, a little bit more useful to use.

And this is where the era of post-training has gone off. So in post-training, we teach the model how to complete function bodies, understanding doc strings, how to complete, generating multi-line completions, predicting the next diffs, apply the next diffs.

And I think we are still in that era where there is so much more to be explored in the post-training stage of RLHF, RLAF, to push the capabilities of models to reason through complex code bases.

The next paradigm in AI research, which has happened last year, and it was published by OpenAI with the new model O1, is scaling reinforcement learning on chain of thought. And this is why we call them it's highly complex reasoning.

And you can imagine you spend a lot more test time compute on training to scale reinforcement learning. And the reason why it works is because the model learns how to think during the training and learns from the feedback by having really good signals in RL.

So on the left, you can see the output of normal GPT-4.0, or GPT-4. And on theright, you can see the entire chain of thought that has been that the model has thought about to solve the complex problems. And as we think about harder and harder tasks, if you want the model to go from, you know, translation towards solving medical problems, you actually need to spend, you actually need the model to, like, spend a lot of time just thinking through the problem and completely creating more complex environments with tools and other tools and more complex environments to think through and verify its outputs during the chain of thought.

So as you can see, the chain of thought itself is very interesting, and the model has certain words that it does. But there is a lot of science to be done in terms of, you know, faithfulness in the chain of thought, how do we measure the faithfulness, what happens if the model goes into, like, the wrong direction, can it backtrack itself.

I think there is a lot of science around that. And we are only at the beginning of it. One of the first projects that I've done at OpenAI is actually how do we

the interaction paradigm is very different now. So the interaction paradigm is the model thinks a lot to solve the problem if the problem is hard. So, but how do we create this interaction, new interaction paradigm with humans such that it will be much easier so that humans don't have to wait for 15 seconds or 30 minutes for a model to come back.

And one of the things that we did as a simple approach is to have, like, a streaming model thoughts to a user. And that way, we had to communicate what exactly the summaries of the thoughts for the model and communicate very wisely to human.

But I think it's still one of the design challenges. Like, as the model's capabilities and interaction paradigms change, you have, like, new design challenges that you need to solve for these types of models.

So, and I guess, like, this year at OpenAI, it's the year of agents. And the way we think about it is highly complex reasoners, such as models trained on RL and chain of thought, using real-world tools such as browsing, search, computer use, over a long horizon period of time, over a long context.

But what's the next stage? In my view, the next stage level is co-innovators. And the way I'm thinking about it is it's agents that are built upon all the things that we've done with reasoning and tool use and long context, plus creativity.

And creativity is enabled only through human-AI collaboration. And I think this is where I'm really, really excited about in the future is to

create new affordances for humans to collaborate better with AI such that we both can co-create the future that we want. And so those two scaling paradigms in AI research have unlocked us new kind of product research. And, you know, you can imagine product research being, oh, we have API from the model, and now we have to integrate in the products.

But it's actually what's happening on the ground is we have, like, a very now we have, like, a very nice rapid iteration cycle of the product development. And the reason why is because we can use those highly reasoning models to distill back to smaller models or the models that we can iterate very, very fast.

And we can use those highly complex reasoning models to synthetically generate new data such that we can create new post-training, new data sets, new reinforcement learning environments. So,

so one of the things that we can do is creating new, completely new class of tasks. And, you know, if the task is how do we create a multiplayer collaboration with a human and AI, you might want to simulate different users.

And how do you do that? You might want to, like, synthetically generate data sets of different users conditioned on the different users and post-train on that. So it actually highly depends on, like, what kind of product experiences that you want to create and extrapolate that to a new class of tasks that you want to post-train the models.

I think we are moving towards more complex reinforcement learning environments, which means we can allow models to use search or browsing or much more collaborative tools like Canvas during RL such that they can learn how to be, how to become better at collaborating.

We can leverage things like in-context learning. I think models are extremely, extremely good. So you can essentially create something, a new tool, and then the model will learn just via a few short examples. And this is an extremely rapid iteration cycle for any developer.

As I mentioned before, synthetic data-wide distillation is another thing. I think we can also invent new model behavior and interactions to utilize user feedback. So now we're going to go through some of the vignettes that, that has happened from Antarctic to OpenAI.

I think the first concept that I've learned is how do we bring unfamiliar capability into a familiar form factor. And the reason why 100K Context was successful is because we found, you know, file uploads is extremely familiar form factor.

Everybody is working on documents. But you can imagine we could have deployed 100K Context via infinite chats such that it's like one huge long chat that you can interact with. But I think finding the simplest form factor sometimes for unfamiliar capability is one of the design challenges in this new era.

The next project that I worked, the second project that I worked on at OpenAI is called ChatGPT Tasks. And actually, I did not realize about this until it was shipped.

You know, reminders and tasks, scheduled tasks, is actually it's a very familiar thing that people do almost every day. But what's amazing about this product is that you can scale this with new kind of capabilities of the models.

So ChatGPT Task is not just scheduled reminders and to-do lists. It's actually you can create, you can ask the model to continue the story every day for you. Or you can ask the model to search everything that you are interested in every day or every other, every other day.

So in a way, you can also, like, help yourself to, like, learn new language by having, like, extremely multimodal and interactive visualizations that ChatGPT can create. And so I think this concept of your product feature should enable modular compositions that will scale very nicely in the future as the models will develop much higher capabilities is one of the, is something that I've learned by doing ChatGPT Tasks.

I think another design challenge that we have here is how do we bridge together real-time interaction with models to asynchronous task completion. And where we can ask the model to go off for, like, 10 hours to research or write code and then come back with a solution.

And the bottleneck here is trust. And I believe that giving trust can be solved by giving humans new collaborative affordances to verify added model outputs and having them to give models real-time feedback so that the model can self-improve.

And, you know, one of the first products from Antarctic was actually Claude and Slack. And it was the first attempt to have a virtual teammate in your organization. And it was an amazing concept because Slack had all the affordances with tools and image uploads and multiplayer collaboration that you can create.

And there is something in there that is still, I think there is still a lot of things that we can do here and take the lessons from Claude and Slack to the next generational products.

Again, the task was also, like, very much inspired by Claude and Slack prototypes when Claude could just summarize channels, Slack channels across the organization every Friday and have the summaries for everybody.

My first project at OpenAI was Canvas. And

I thought that human-AI collaborative affordances could scale and create new creative capabilities. And what I really loved about the Canvas and the way we operate in Canvas as a team is that it was an extremely flexible interface that we could come up with.

And here, here are some, like, the vignettes that we had. So the Canvas itself can become, like, a co-creator and co-editor. You can have, like, a very, very fine-grained editing interaction. The model can also do search in order to generate the report.

And then you can also ask a question back, "Hi, verify this output." And you can imagine this interface scales to multiplayer when other people can join your document or even multi-agents if I can create a model critic or an editor they can use, you know, have, like, a multi-agentic and multiplayer collaboration at the same time.

And so this is, like, a new design challenge that we need to navigate. How do we do that? I'm also excited for personalized tutors. I think the models are becoming extremely multimodal, extremely flexible, that you can learn new things in a new way, the way you like.

If I'm a visual learner and you are more auditory learner, the model can adapt to your personalization. Sorry. One thing that I did yesterday is that

I was on the plane and actually I used Canvas to create me a game. And so I really like this character entertainment on the fly.

Anyone can create their own tools and web apps now. And I'm not sure what the future will look like, but I think it will be extremely amazing if a non, a person who never had to touch code ever before in their life for the first time can create the tool that they really wanted and deploy that for themselves or to start a business from scratch.

And I think there's something around pair programming and co-creators that we can use in order to create the future that we want. And so Canvas also becomes more of a pair programmer. So the reason why Canvas is so flexible is because it was adapt, it was trained both to become collaborative for writing and coding.

Because it has tools such as search and can search for API documentation, it can become a data scientist too. And especially if you upload the entire CSV docs, it can generate a real-time analysis.

And finally, what I'm really, really excited and what everybody is excited in AI is how do we actually help models to become better at research and creating new knowledge. And here you can see that the model, that the model and the human can co-create a document or co-create a new artifact that has never been happened before.

So here's a demo of the paper that I've co-published. And then I'm asking the model to kind of reproduce it. And you can imagine this is, like, one of the, maybe one of the most common tasks in research is to reproduce, or you can imagine to reproduce, like, open-source GitHub repo.

And you, you have, like, this very nice interactive paradigm where because the model can also leverage its own internal knowledge, you and a, you and an AI can work together to come up with new research hypotheses and verify certain, like, research directions together.

And you can also handle, delegate the tasks to an AI assistant to do that.

Finally, what I'm really excited about the future is that there will be this layer of invisible software creation for all. And especially what I'm really excited is, like, from the mobile itself, people can just create their own software tools.

I think the way you interact with AI fundamentally changes in the way that the way you access the Internet will also change. My prediction is that you will click less and less on the Internet links. And the way you will access the Internet will be via a model's lens, which will be much cleaner and in a much more personalized way.

And you can imagine how, like, very personalized multimodal outputs. Let's say if I say, "I want to learn more about the solar system," instead of it giving me a text output, it should give you a 3GS interactive visualization of the solar system.

And you can have, like, highly richly interactive features to learn more. And I think there is, there will be this, like, kind of cool future of, like, generative entertainment on the fly for the people to learn and share new games with other people.

I think the way I'm thinking about it is the kind of interface to AGI is, is blank canvas that kind of self-morphs into your intent. So, for example, you come to the work today and your intention is to just write code.

Then the canvas becomes more of an IDE, like a cursor or like a coding IDE, although the future of programming might change. Or if you're a writer and you decided to write a novel together, the model can start creating tools on the fly for you such that it will be much easier for you to brainstorm or edit the writing or create character plots and visualize the structure of the plot itself.

And finally, I think the co-innovation is actually going to happen with co-direction, creative co-direction with the models itself. And it's through collaboration with highly reasoning agentic systems that will be extremely capable of superhuman tasks to create new novels, films, games, and essentially new science, new knowledge creation.

Cool. Thank you so much. I think that's the end of my talk.