See, Hear, Speak, Draw: Logan Kilpatrick & Simón Fishman

So we can think of OpenAI as a product and research company.

We build awesome models, and we think about what are some of the best ways to apply them to solve the biggest problems that humanity faces. And so there's this deployment pipeline. Logan and I sit at the end of this deployment pipeline.

We work with people in the real world that are using OpenAI's models. We spend our time thinking about what are some of the best ways to use our models, what are some of the hardest problems that haven't been solved yet, and how can we apply OpenAI technology to solve these.

I'm on the Applied team, and I'm an engineer.

Yeah. And my name's Logan Kilpatrick, and I do developer relations stuff. So helping people build fun and exciting products and services using our API.

So yeah, folks, all from the title of the talk, we'll talk about multimodal stuff. But I think it's important to start off with where we are today. And I think, as we all know, people who have been building in the AI space for the last 6, 12, 18 months, 2023 has really been the year of chatbots.

And I think it's been incredible to see how much people have actually been able to do, how much value you can create in the world with just a simple chatbot. And it still blows my mind to think about how rudimentary these systems are and how much more value there's going to be created in the next year, in the next decade.

And that's why I'm excited for 2024, which I think is really going to be the I don't know if I can trademark this, but the year of multimodal models. It's a tongue twister, but also, hopefully, the domain is available, yearofmultimodals.com.

Don't buy it if it's available. Yeah, so I'm excited. OpenAI has a ton of multimodal capabilities that are in the works. Some folks might have already tried some of these in ChatGPT and the iOS app or the web app today.

Things like vision, taking in images, describing them. We'll show that later on. Also, the ability to generate images. We've had this historically with DALL-E 2, but DALL-E 3, really, if folks have tried it, it takes things to the next level.

So excited to show some of that today as well.

Cool. So if you think of the way that multimodal capabilities are workingright now, it's a little bit of a setup of islands where we have DALL-E that takes text and generates images. We have Whisper that takes in audio and generates text, transcripts.

We have GPTV with vision capabilities, Jupiter with vision capabilities that takes images and text and can reason over both at the same time. Butright now, these are all very disparate things. However, you can think of text as a connective tissue between all of these models.

And there's a lot of interesting things that we can buildright now using that paradigm. But what we're actually really excited for is a future in which there's unity between all these modalities. And this is where we're going. This is not where we're today.

But you can think of models in the same way that GPT can consume images and text simultaneously. Maybe in the future, we'll consume even more modalities, and we'll output even more modalities, and we'll be able to reason about them at the same time.

However, we're not there yet. And so today, Logan and I are going to show you just some architecture patterns and some ways in which you can mimic this kind of situation with what we have available today and some of the patterns that you can start to think about as we move towards this future in which models can reason way beyond text.

As Simón and I were making these demos today, waiting till the last minute, as always, it was really interesting to see that really much of the work of making multimodal systems today is like, how do you hook everything up together and connect the different modalities?

And again, as Simón said, using text as sort of the bridge between different modalities. But it's going to be super interesting to see how much developer efficiency gains there are when you no longer have to do that, and you really just have a single model that can do text in, text out, video at some point, speech in, speech out at some point.

So it'll be super cool to see when that's possible and make making demos even easier and simpler.

Allright. Well, we'll show you guys two demos today. And we'll talk about some high-level ideas and some high-level concepts. And hopefully, at the end of it, you'll be inspired to think about what are some of the things that maybe you're not able to build today, but you'll be able to build six months, a year from now, and how you should start thinking about your products

as they are able to incorporate more modalities. Cool. So on to demo number one.

It's a very simple DALL-E vision loop.

Yeah. So yeah, sorry. Excited to look at this demo. So Simón, we'll pull up the demo, and I'll sort of just walk through it. But the basic idea is, let's take a real image. Let's use GPTV or GPT-4 with image inputs to essentially create a nice, human, readable, understandable description of that image.

And then we'll put that into DALL-E 3 and actually go and generate a synthetic version of that image. So this whole pipeline takes a little bit to run because it's not a production system at the moment. But the nice part is, we've got a couple of examples ready, and we can if you want to kick one off live as well, we can let it run in the background.

So this is a fun, simple idea. But this is a photo that I took in the lobby downstairs, just when you walk into the hotel. There are these kind of Halloween-themed painted ladies. And so what we did here is that we asked GPT-4 with vision to describe this image in detail.

And then we asked it to generate a description for DALL-E to generate a new image based on this.

You can see it does an OK job. Here's the description of the image. Here's the prompt it uses. It picks up on a lot of details, like the RIP in the tombstone and the old dog's welcome thing here.

And then it generates a whole new image. But there's a lot of details that are off.

The marble is black, and the spiders are white. And so what we do next is that we pass the yeah, it's close enough.

But we give the two images to GPT with vision again. And we ask it to compare them and see what are some of the differences. And it picks up on a lot of the different details. And then we ask it to create a new image based on these differences.

And it goes ahead, new image. You see?

All the black marble is gone. The spider is now larger and black. But it matches something closely. And I think this is just to illustrate. I think there's a long way to go. But this is to illustrate the idea that there are plenty of tasks that we doright now in AI where we need the human in the loop to be able to evaluate a visual output that a model produces, compare it with something else, then iterate on the instructions, pass that again to another model.

And so that's a pipeline where we thought that humans were very essential and that we're probably going to continue to be essential for some time. And now, that's something that the models can do by themselves. And there's a couple of interesting

patterns here. I think one of them is describing images. That's powerful because now you have an image. Now you have text, and you can reason about that text. You can do a lot of things with that text. But another really powerful element is comparing images

and spotting differences, like having a final destination that you want to get to and a current destination. And that pattern of comparing things, you can apply it to a lot of things. So imagine talking now, Logan and I were just chatting about some other ways that you can apply this.

And Logan's idea was, imagine you are curating your room, and you just moved to a new place. And you're on Instagram, and you find some images of a vibe that you like and maybe some object. And then you can grab that image.

You can give that to GPT-4 with vision, and you can tell it, OK, now crawl through Amazon and find all the lamps that match this vibe that I want for my room.

I want this so badly.

Yeah.

I can't do interior design. So it's like, I would love to be able to just be like, get me all the stuff that matches this specific vibe. It's a hard problemright now.

Yeah. And a couple.

Simón, can I make one other quick comment?

For sure.

I think also, folks were laughing in good jest when this third image came up. I think it's important to know that there's no prompt engineering or anything like that that's happening. This is the rawest output that you can get.

This is a one-hour demo version. So people will hopefully go wild with this once it's available through the API and ideally get much better results than we're seeing today. Yeah.

Yeah.

Probably using a bunch of techniques that other people talked about at the conference so far. So this is the very basic version of this demo.

Yeah. And we wanted to keep it simple and minimal just to illustrate the power of the models. This is as raw as you can get when it comes to the models. There is almost all the completion output. It's going straight into the model.

And I think there's like 50 lines of code. So the majority of the power lifting is being done by the models here.

Another quick example that I'll show you guys, and then I'll try to do one live, which will probably be tragic.

So this is the backstageright here. I just took this photoright before walking on stage. You can see that GPT with vision does a really good job, actually, of describing that. There's the monitors, and there's boxes, and there's cables, and there's whatnot.

And then this is the image that DALL-E generates, DALL-E 3. So you can see blue carpet, cables, boxes, lots of the elements. And then it goes on to spot the differences. And it notices, for example, that in this image, there are all these vertical lights that are not present in the first image.

It says that here, lighting, all these vertical lights on the walls and ceilings, which adds. But then it rewrites the prompt, and it gets rid of all the vertical lights. And it adds the curtain in the back, which wasn't present here, but is present in the black curtain here.

So just little interesting things. It's still a long way to go, but this whole new this opens a whole new box of interaction patterns, the fact that now you can reason visually.

Cool. And let's give a shot to a live example. So this was a trail run that I did over the weekend up in Purisima Woods. And so I'm just going to do it from scratch.

Hope that it works.

You want to go to another?

There you go. Cool. So the image depicts a serene and picturesque woodland setting. The focus of the images are a wooden boardwalk or a footbridge that winds through the dense forest.

Very detailed description, light filters through the trees. And I'm just passing that raw just straight to DALL-E.

Yeah. And if folks have seen what happens in the DALL-E mode in the ChatGPT iOS app, for example, it's actually doing a little bit I don't know off the top of my head what the prompt is for that.

But it's doing some amount of prompt engineering. If folks have actually tried to use our Labs product before to make DALL-E images, you have to do that prompt engineering yourself. And I think that's been one of the limitations.

If people used Midjourney or other image models in the past, it's just kind of hard to make good prompts that work well for these systems. So it's nice that the model can take a stab at doing it for you.

It's telling. That's a lot of how the second image is a lot more beautiful and more detailed, which checks out.

It's also interesting to see, just for folks to think about. It's interesting to see that it's still of these image models. The main limitation, as we're seeing this demo in real time, is actually.

No.

Of course.

We're going back to the slides. Next to the back slides. We're going to run out of time.

I'm going to leave it running. And then at the time, if we have time, it'll probably work the second run.

It worked three times before this.

Cool. OK. For the second demo,

we're going to take it a little bit further, and we're going to do something with video. And the idea here is that there's a lot of video summarization demos out there that we've seen. The majority of them just take a transcript and then ask GPT-4 to summarize this transcript.

However, videos have a lot of information in them that is conveyed visually. And so what we're doing here is that we're taking frames from the video. And then we're asking GPT-4 with vision to describe all the frames. And then we are asking Whisper to transcribe the video.

And now we have this long textual representation of the video that not only includes all the audio information, but also includes visual information from the video. And then we're doing some exciting mixes on that that Logan will tell you about.

Yeah. I'm ready for the next slide.

Yeah. So for this demo, we're literally just taking the GPT-4 introduction video folks have seen on YouTube. It's a good video if you haven't seen it before. So taking the video raw from YouTube.

Next slide.

Taking the video raw from YouTube. Again, like Simón said, cutting up the different frames from the video, putting those into GPT-4 with image input, getting the summaries, which you can see, and I know it's really hard. But literally just actually saying what's these are simple images, so it's easy to capture the depth of what's shown here.

Taking those images and then going to the next piece, which is essentially another wonderful DALL-E image, but a big description of the transcript and then all the image essentially image embeddings is the easiest way of thinking about it.

So if you want to actually see the results of this, QR code bottomright-hand corner is real. You can scan it and see the resulting article.

It's pretty good. It does a good job. And I think for me, why this is exciting is because you can sort of, again, capture the depth of

what happens in a video. So a DALL-E image to start and then a bunch of actual frames that match up with the contextual representation of what's being talked about in the blog post. And again, there's no I couldn't open source the code because it has a bunch of unreleased APIs, but no sort of magic behind-the-scenes stuff that's happening.

This is like a raw, crappy prompt to generate this blog post, which I think is again, I think it's really cool and takes videos and makes them more accessible in the text form. So I like it.

Cool. Let's see if this is finished.

Oh, well.

Cool.

Can you read the URL for that?

Oh, yeah. Sure.

Cool.

OK. So some concluding takeaways.

Start thinking multimodal. That's something net new that's happening these days. And if you have any crazy ideas that you think, wow, it would be really cool if technology could do this, we'll probably be able to get there. And the products that you'll be able to build six months from now, a year from now, are going to be incredible.

So start having this in mind as people who are building AI products and people who are building companies.

Think of text as a connecting tissueright now. And I think this is a very powerful concept. And that's going to continue to be the case for the near future. And there are many powerful patterns that are yet to be explored when it comes to multimodal stuff, especially when it comes to doing things with images.

So really excited to soon get this in the hands of all of you guys and to see what you all build with this. I think it's really exciting to see AI start to venture into the visual world.

Yeah. Agents with image input is going to be sick. I can't wait. I feel like so much of the internet requires that. Yeah. And we're excited. I think there's a lot of stuff that's going to happen in the near future.

And I think it's cool to be able to hopefully get a glimpse of what some of those use cases look like. So anything else you want to say, Simón?

That's good.

Allright. This is wonderful. Thank you all.

Thank you all.