The era of unbounded products: Designing for Multimodal IO: Ben Hylak

I'm so excited to be here with you guys today at, I think, what is probably the coolest AI conference in the world. It's such an exciting, exciting time in history, I think, especially for AI products.

If you don't already know me from either demos on Twitter or sometimes, probably, ill-advised, spicy takes on Twitter, my name is Ben Hylak, and I'm the founder of Dawn. So

at Dawn, we help some of the best companies in the world, everyone from GitHub to can of soup, build better, more predictable AI products.

My entire life, I've been, like, really obsessed with building and designing unbounded products. So unbounded products are products that transcend

the mouse, the monitor in some way,right? So for me, that started with robotics when I was, I think the first one was when I was in eighth grade. Eventually, rockets at SpaceX, so these are very unbounded products,right? And then most recently, I was on the design team for the Apple Vision Pro for four years.

So we designed the first version of Vision OS.

I think that AI makes products less bounded than they've ever been,right? You can type, you can talk, you can show images or show video, just like we just saw. You can also sort of plead, you can bargain, you can confide,right?

These are very interesting sort of input modalities. And this unboundedness often makes products unpredictable,right? Confusing, hard to understand. Users assume your product can do things that it can't. They try to do those things, doesn't work. And they walk away thinking that it can't even do the things that it can.

When you talk to people, and specifically people that are not in this room, how they use ChatGPT, how they learn how to use it, it's often word of mouth,right? So they hear one of their friends say that they use it for travel planning, and then they go use it for travel planning.

A lot of us, a lot of us in this room, especially people that are more technical, we often learn through trial and error,right? So we just keep trying and keep trying. We keep trying because we know that these models are good,right?

We know that it's impressive. But a lot of people are not. They don't do the trial and error thing,right? So they try it once, it doesn't work. They don't try again. And so this talk is about making good AI products.

And to that end, I'm going to cover just three things. So those three things are the past,right? So how have products become more unbounded, and what has worked for unbounded products in the past? The present, which is AI products today, what are sort of good design patterns and bad design patterns?

And then the third point is going to be the future,right? So again, just three things, just the past, the present, and the future. Easy. So we're going to start with the past.

So

most software that we use lives on a screen,right? And you use it just by typing. Sorry, you use it primarily by swiping, clicking, and tapping,right? When you click something, whatever the developer expected to happen is what happens, depending on how good of a developer you are.

It's easier for users to understand what your app can do. They look, they see the buttons, they get it. It's also very easy for you to understand what your users are doing. You just add an Amplitude or Mixpanel call on a button press, you see what they did.

So

if you think about one of the biggest changes to this previous to the last two years was multi-touch,right? And this is just like, instead of one pointer, you have two. But just by adding that second pointer, you get relative distance, you get rotation,right?

And just this one little change largely made the smartphone possible,right? Largely made it easy to use a screen that is that small. And now it's just getting crazy,right? It's like we have unbounded products everywhere. Products are so unbounded, you have software freely running in the streets of San Francisco getting attacked by fiery mobs.

Crazy.

And so I want to talk a little bit about just one unbounded product that I got to work on, which is the Vision Pro. And what I want to talk about is just three lessons that we learned while we were designing it.

Lessons that I think aren't as intuitive looking from the outside in.

So I think that unbounded products are often defined by this what-if question. Like when we were starting, it's like users get themselves into the craziest situations. So something as simple as, oh, well, what if someone's in the living room and then they move to the bedroom and they lay down on their bed,right?

What should happen to your apps? If you're designing macOS, it's on a laptop, you don't have to worry about that. But that's something we had to think about. And there's hundreds of more questions like this,right? What if someone's on a plane?

What if someone's next to their friend? What if someone has a disability of some sort,right? Like they cannot move. They can't move their neck. They're bedridden. So all of these what-ifs. And I think this is, again, what defines unbounded products,right?

All of us that are building AI products, we're constantly thinking, oh, what if someone puts in this? What if someone puts in that? And there's evals, et cetera, et cetera.

And so without structure, you just have chaos,right? You have a blank slate. You have all these what-ifs, infinite world of possibilities. And so it's really on us as product designers to add structure. And structure is what creates clarity.

So again, I want to talk about three ways we added structure. The first was highlighting what matters and doing it really fast. So the first thing you see in Vision OS is a home screen. It has apps, it has people, and it has environments.

So those are the things that we think matter when you're using Vision OS. So they're the first thing you see. It might not sound that novel, and it's not. In a lot of ways, it's the same thing that happens on your iPhone.

But when you compare it to VR products that came before, it's very hard to understand how you're actually, what is this thing good for when you look at this menu?

The second point is hierarchy.

Hierarchy is what gives unbounded products a shape and a purpose,right? It's what helps users understand what it's good for, what they should use it for. So again, we have the home menu. That's kind of where everything starts and ends for Vision OS.

We have windows. They have bounds. You can resize them and move them.

And any individual window can go full screen,right? So that was our hierarchy. The last point, which is really important, and I think the easiest way to make unbounded products feel familiar is, sorry, intuitive. I got ahead of myself.

It's familiar. Familiarity. It was something we hit when we were building Dawn. Our first kind of prototype was this star cluster thing that you could explore. It was really fun. Nowadays, it looks a lot more like this, which are tables, and we have graphs and examples.

Again, it's just structure, clarity. And I think that it's no accident that the TV app on Vision OS looks a lot like the TV app on TV OS,right? It's not an accident. It wasn't laziness. When people are sort of in uncharted territory, you want to give them as many signs of home as possible.

Same thing for Control Center,right? When people see Vision OS, they already know how to use it.

So again, these three points. Highlighting what matters, bringing that to the forefront, establishing hierarchy, and then leveraging familiarity. Allright. And so that was the past. Now we're going to talk about the present. And specifically, we're going to talk about AI products.

We're going to talk about ways, both good and bad, that products have been incorporating structure into their AI features.

It's really important to note theright structure is very unique to your app,right? That's the whole point, is that it gives your app a shape. It helps your user understand what it's actually for. So let's take something like Dot,right?

Dot is a companion. Dot is sort of a journal, at least for me. And so the structure they added was that if you pinch out, you can see each day separated,right? It feels a lot like a journal. And if you tap a person or two people, in this case, my co-founders, I can see this, again, structured information about them and a timeline every time I mention them to Dot.

And so again, you're pulling that structure out of the chat.

Perplexity does a really good job of using structure to make their experience feel more like a search engine and less like ChatGPT, less like a chat you're having a conversation with,right? And they do this by really pulling your title, your query up top as like a title, highlighting the sources it came from, and then having the answer below that,right?

And then having that take up the full page kind of regardless. So it makes it feel, again, more like one shot, less something you're having a back and forth with.

Now I want to talk about sort of an anti-pattern I've seen, which is this is in the Vercel chatbot demo. I think Vercel does some of the coolest design work in the entire world. I didn't like this one.

So this is like having this idea of almost ephemeral UI, but inside the flow of chat,right? And I get the appeal,right? So actually, if we go back here, sorry, this was a video I meant to show. You have a slider,right?

So instead of having to, like, you inquire about, you want to buy Doge, and it shows you this UI so you can adjust exactly how many, instead of having to do it over text. It could be good. The problem is that when it's stuck inside this sort of unstructured thing, it starts floating away as I try to ask follow-ups,right?

And then at some point, I even have two of them,right? So I go back up to the first one, I press purchase, and now I'm interacting with something that's completely different. So it reminds me a lot of sort of the house and up,right?

It's just kind of up, up, away. So instead of trying to put structure, stuff it into this unstructured thing, I think the answer is you pull it out,right? You pull it off to the side. And what that means is that as the conversation continues, you can just sort of update that structure without disrupting where the user is.

And that's exactly what Claude did with artifacts,right? And I think why it's so successful is that they pulled out the structure, which is the app you're working on and iterating on, from the actual conversation. And then so as you make changes, you can even go between the versions here without even having to scroll in the conversation,right?

So it's beautiful. And it actually brings us to another thing that I think has been really effective for AI apps, which is this almost concept of version control. So this was actually one of the shipping original ChatGPT features, which is kind of crazy.

But if you edit a message, you can go between the versions,right? And it actually maintains this entire tree. It's very complicated, but super powerful. With v0, Vercel did something, again, amazing, where it feels extremely familiar, almost like you're working on Google Slides or something.

But you can go back and iterate, keep iterating on UI without having to be afraid that you're losing something,right? So again, versions. Again, I think familiarity is really one of the most important things for unbounded products. I think Claude did an excellent job with this.

Again, I'm hyping them up here. But ChatGPT introduced memory across all of your chats,right? Completely unbounded. So when I tell it something about some sort of medical problem as I'm working on a JavaScript, it's like it knows that, which is very weird to me.

I think this idea of projects and that structure of a project is very familiar. So sharing context across a project makes more sense. Agents are something that are extremely unfamiliar to most people. And this idea of having all these different tasks and you're feeding data between steps, whatever.

But you know what is familiar are spreadsheets,right? Spreadsheets are extremely familiar to, not to me actually, but to a lot of people. And I think the only real uses of agents I've seen in the world, in the real world, are spreadsheets.

So this is clay,right? And each column is essentially a step that an agent is taking, the user is defining. So it's going across, it's building up kind of context across the spreadsheet. Each row, it's often almost you do it like an eval,right?

So it's like you run the first 10 rows, and then you run the next 50,000, 100,000,right? So you get itright. And you can see here, eventually, you end up with a personalized email as the last column, but with all these steps in between.

The next thing that I think is extremely effective in helping people understand what your app is for and skipping all the sort of noise of prompt hacking, prompt engineering, are examples and presets. So ChatGPT, I think, was the first for this, where they had these message to comfort a friend, plan a relaxing day, and so on.

v0 does an awesome job with this,right? We're not just having those suggestions below, but they also have an explore page where you can see what other users are doing, what's actually working,right? Again, try to shortcut this prompt, blank canvas problem.

Notion as well,right? They have a simple menu where you can change tone for text instead of having to be like, you are a very concise GPT, whatever, whatever, whatever,right? So you're just using these tried and proven things that Notion can validate.

And that last point brings us to the future,right? So where are interfaces going in the future?

Linus gave an awesome talk last year where he described prompt engineering as this almost trying to drive a car, a llama trying to drive a car with a pool noodle from the back seat was, I think, his metaphor.

And there's some real truth to this,right? And so I think, first of all, the future has a lot less prompt engineering. And we're already seeing this,right? We're already seeing this with

generative images, the way that Apple designed it, where you're mixing and matching these different concepts.

There's a ton of demos on Twitter of people. Essentially, you're going between emotions here in a more intuitive way. And then just yesterday, Figma released this way of adjusting the tone of text,right? Where you're going between professional, casual, expanded, concise.

The problem with this is that casual means a lot of different things,right? Casual for a Fortune 500 company and a direct-to-consumer cosmetics brand with ads on TikTok,right? These are very different things. Casual when talking to your best friend or a coworker, these are different.

So how do we avoid being reductive when trying to offer these sorts of presets?

And the answer is you just, like, I don't know exactly how many zeros I put here, but you just, like, million X or billion X the number of presets,right? So you have enough presets for everything. And I think sparse autoencoders show a really promising path towards that.

So if you guys have tried GoldenGate Claude, where you can kind of identify the one feature of GoldenGate bridgeness and amplify it, and it makes Claude obsessed with GoldenGate bridges specifically, or the GoldenGate bridge. My friend Giddis has an amazing demo towards this, but for manipulating images,right?

So you can see here, he's increasing the amount of play of light and shadows, increasing the amount of serene forest streams or Venetian canals in, again, a very controllable and predictable way.

OK, but so now we have a million, billion options, whatever. How do we avoid too many options? I think this gets to point three, which is ranked presets. So these are presets that are personalized, searchable, and even invoked through natural language.

They might not even be directly visible to the user. So the user types in something like "more friendly," and you pull up the corresponding presets, like kindness, how close you are, how confrontational it is. Again, maybe they're directly editing it, maybe they're not.

And this gets to the last point, or second last point, which is developer-defined personalization. So as soon as you're able to define those sort of features, you can start tuning them per user. So each user, in a way that you can't do with just text prompts today,right?

Because text prompts are sort of this fragile house of cards, where if you remove one word, the whole output changes. So you're able to tune it per user.

And the last point is especially true. As you start, your app is going to become increasingly different per user, is shifting from evals to analytics. I don't think there's going to be some objectively correct, for a lot of domains, answer to things.

Like, who was the first president? Yes, but theright sort of tone for a summary for a specific user, I don't think so. And so I think that increasingly, it's going to be about how do you understand if you're meeting the needs of your users and what they're asking for.

So that's it. Thank you so much. Oh, yeah, we'll skip this one. And thank you so much for coming.