Understanding is the new bottleneck — Geoffrey Litt, Notion

What's up? Yeah, thank you for coming to the Design Engineering track at AIEs. Everyone having fun?

Yeah.

Yeah? I think this is going to be a great track, so get excited. Allright, let's get going. My name's Geoffrey Litt. I'm a design engineer at Notion currently, and I'm here to drop a hot take for this room, maybe.

I think it is still important for people to understand how code works.

Now, some of you might agree, some of you might disagree. Let's actually, let's do it, let's try a poll. I'm curious for this room. Raise your hand if you agree with that opinion. OK, maybe some selection bias. Any brave OK, raise your hand if you disagree with this opinion.

Yeah.

Wow. OK, we have maybe we'll do a debate later. Yeah, I was hoping we'd be at the AI Engineer Conference so we'd have more bold OK, I might be preaching to the choir here. You know, I think we the reality is, though, we are entering an era where this is a legitimate question that people are debating,right?

Agents are writing tons of code for us. They're landing 50,000-line PRs, and it is getting harder to keep up. We all feel this. Now, I think the good news is there are lots of ways to understand. The, you know, the days of just reading code line by line, that's not the only way anymore.

And what the point of this talk is about is I want to share with you a bunch of the practices that I use to understand the code that my agents are writing for me. This includes things like explainer docs teaching me about how my code works.

My agents write quizzes for me to test my understanding. Am I still really in the loop? Am I keeping up? I have agents build me micro-worlds that I can inhabit to get this intuitive sense of how my code works that's deeper and richer than just a written document.

And I think all of these are really exciting new possibilities that are opening up for AI to help us understand better, not worse. And so that's what the point of this talk is going to be about. And I hope I can leave you with some techniques that you can take home and use yourself.

By the way, my timer isn't running. If you could get that running, that'd be great so I know if I'm blabbering. OK, but let's start let's back up for a second. Before we talk about how, let's talk about why.

Why bother understanding? This is, again, it's a question now,right? And I think a lot of people get this subtly wrong. So what a lot of people think of, why do humans still have to understand? They think we understand to verify.

The agents do dumb stuff. We've all seen it. And your job as the human is to keep them in line,right? Make sure they don't screw up.

When people say things like code review is the new bottleneck, I think that's the first thing that pops into people's heads is correctness, checking. There's this mental model that's like, hey, the agent's going to give you something. And what's your job?

It's to ask, is this correct? Now, correctness can have lots of definitions. Does it match the spec doc you gave it? Does it take down production? Is it well architected? But fundamentally, those are all kind of thumbs-up, thumbs-down decisions,right?

And the thing is, over time, we've all seen it, the agents are also able to ask these questions, and they're getting better at it. You give it theright verification loop. And over time, this is the reality. The role of humans in correctness checking is decreasing.

And you know what? I actually don't hate that. If I have a clear idea of what I want to do and the agent does it correctly, instead of coming back to me with an incorrect thing, that's great. I'm into it.

So then I think people extend this and say, you know what? That means as the agents get smarter and smarter and smarter, we don't have to understand it all,right? Get out of the loop, man. Run the loop. And that's where I think people miss something really important.

There is a deeper reason to understand what's going on, and that's understanding to participate. Because here's the thing. It's not just one loop. When you review what's happening and get in the loop, you come away changed. You understand something.

And that understanding is what you take to the next loop and the next and the next.

Your understanding of what's going on is the foundation for you having that next idea and being an active, creative participant in a project. I think probably you've all felt even before AI, you know, the difference of the kinds of ideas that someone can have when they really understand what's going on versus when they're a few layers removed are different.

Because when you have rich conceptual structures in your head that you can fluently recombine really fast without going out to, like, ask some agent or some human how it works, that gives you the ability to fluidly take creative leaps.

And that's the human part of the work, coming up with the next idea and the next idea. So this is actually the real reason I think understanding matters. And this is not something that we can just wash away with better agents.

Because if we want to be active participants, you still got to do this.

There's a great term maybe some of you have heard called cognitive depth that I think really captures this spirit well. It's an analogy to technical depth, popularized by the scholar Margaret Story. Simon Willis and also blogged about it.

And I love this idea because similarly to tech debt, you might get away with it for a little bit, but at some point you get burned if your understanding degrades. And maybe you felt this. I know I felt it.

You're vibe coding. Things are going well. And then at some point you realize, wait, I have no idea what's going on. I basically can't participate anymore,right? You've built up too much cognitive debt.

OK, so maybe it sounds like all of you were already convinced. We agree. We need to understand. But how,right? We don't want to live in 2023. We are using agents to move fast, and it is harder and harder to keep up.

How do we do it? I think to answer this question, we should actually take a step back and ask a more fundamental question, which is, how do we understand stuff in general?

Plot twist, this is not the first time that any human has asked this question. There is a field. It's called education. Now, when you think education, you might think of bad memories from sitting in lectures or whatever. But I think we can do better.

We can take inspiration from the best ideas that have ever been invented in education and use them to stay in the loop and understand. So that's what this talk is about. We're going to talk about three techniques.

First, explanations. So when an agent writes some code, it's an opportunity for it to explain the work to you,right? And the most naive explanation is, hey, here's the code diff. That's the raw change, the material of what happened.

But we can do, I think, much, much better. What would the best possible explanation be? Like, if you sent a team away for a year to come up with a personalized curriculum just to explain this one code change to you, what would that look like?

I think this is a very generative question to ask. So I've done a bunch of attempts at this. One is this skill I wrote called explain diff, which I use every day. And a lot of my coworkers do as well.

And I want to walk you through it. So we're going to go through a little example here. I'm working on a video game where you draw zen gardens, kind of de-stress, you know? Could all use it these days.

And we made a code change to change the perspective of the game from top down to isometric. And when I run my skill, it produces a code explainer doc. This can be an HTML file. It can be Markdown.

I like to put them in Notion because I work there, but also because it's then collaborative so my team can comment on it and talk about it. And here's how it looks.

We start with background. We do not start with what happened in this change. It starts by teaching me, hey, here's how this system works. Here's the game engine we're using. Here's the coordinate system,right? Here's the subsystems. It makes sure that I'm sort of being led up to the point where I can even begin to understand what's going on here.

Obviously, you can skip this if you already know. You can personalize it to what you already know.

Second important principle is intuition before details. So before we start looking at code and stuff, it says, hey, the goal of this commit is to make the garden feel three-dimensional using only 2D drawing tricks. You can think of this sort of as like a well-written commit message, a little deeper.

Give me examples. Give me a feel for the essence before you throw a bunch of code at me,right? This, by the way, this is good teaching. This is what, like, good math teachers do.

Third, interactive figures. So where it makes sense, give me things to fiddle with and try. So with this change, it was like changing how we draw rocks. So I can drag around rocks in this little simulation, and it shows me the coordinates that are happening, how the Z layers of the painting are changing.

This, by the way, is actually using a new feature that Notion literally launched this morning of HTML blocks in Notion pages. So agents can put interactive simulations into your Notion pages. Pretty cool. I think you have to be careful with interactivity.

It can just be a crutch, and it can be kind of slop, to be honest. But used tastefully, it can provide an understanding that's hard to achieve with just static pictures.

OK, then we finally get to the code,right? Show me the code. But we don't just throw a list of files in order. We do what we what I call literate code diffs. Give me prose. Explain it to me in theright order.

Tell me before each file what's going on. And when you accumulate all this stuff, it's much, much easier to follow than just a raw diff. Whoops. In fact, I print these out and take them to the coffee shop sometimes and just read them.

I find it really beautifully ironic that AI is actually taking this process where I used to be, like, glued to my computer in my IDE, and now I can go to the cafe, and it's like I'm reading a textbook about this PR.

It's really cool. OK, so there is one problem, which is that reading is hard. And I am lazy. People are lazy. You know, there was this one time when I sent a PR to my coworker that I thought I had read the thing.

I thought I understood. And she asked me the most basic question. And I was like, oh, no. I don't know. I clearly hadn't understood,right? I had fooled myself. So I thought, how can I create a system where that never happens again?

For inspiration, I looked to the work of the researcher Andy Matusiak who has this great line, "Books don't work." What he means by that is it's really easy to read a book and not realize you didn't understand it.

So he and his collaborator, Michael Nielsen, tried this thing where in an essay, there are interactive spaced repetition quizzes that test whether you actually remember what you just read. And this is cool. They actually keep emailing you the quiz to make sure you remember it forever.

But this is nice because you cannot get through this essay without understanding it, or at least without remembering it. That's what I do with my code explainers. At the very bottom, there's a quiz, five questions, medium difficulty. And my rule is I don't send code to others on my team to review unless I can pass the quiz about what my agents wrote.

And it might sound kind of silly, but you should try it. It really is shocking the number of times this has caught me and made me realize I didn't understand. I think of it as sort of a speed regulator.

Everything AI is speed up, speed up, speed up. There's all these incentives to go faster. How do we make sure we're not just moving at the speed of correctness, but also of understanding? And the quiz is that speed regulator.

It's a system I can use for that.

I did just put the skill on the internet. So yeah, photo moment. If you want the explain diff skill, take that QR code, try it out. Make it your own. It's really simple, actually. There's two versions of that QR code, one that outputs HTML, one that outputs Notion.

OK, second technique, micro worlds. What does that mean? So this takes inspiration from the educator Seymour Papert, real visionary, who had this idea of living in mathland. And what that meant was, hey, kids learn French from living in France.

Where do they go to learn math? Is there a mathland where you can learn intuitively math just by being there? So he did these great things with this is a robot called the Turtle that kids program to draw stuff.

But the point isn't making robots. The point is they actually learn math by doing that programming. But the point isn't the robot. It's the kids that are changed. So how could we apply this to understanding code? Here's one example.

Last year, I was trying to implement for my own learning this interpreter for a programming language Prologue, which is think of it a little bit like a database query language. And there's all these parts of it that look like this where when you read them on Wikipedia, they seem really complicated.

And then when you actually get what's going on, it's like, wait a second. That wasn't that hard to understand. It just felt hard when I read it that way,right? How could we make it click more for my brain?

So I had Claude make me a micro world. This is a debugger ephemeral UI that was built specifically to visualize the internal implementation of my programming language. What's happening here is that I'm scrubbing through a timeline that's running step by step.

What's my interpreter doing? It's visualizing all the state at every step. So I can kind of open the hood and see what's going on and start feeling it, you know? And yes, I used this to fix bugs. I even added a little hard to see here, but there's a commenting feature where I can leave comments for myself on the timeline so I remember what I was thinking.

And I used this to fix narrow bugs. But also, as I was fixing the bugs, I was getting a feel for the machine,right? That's something that if you just have an agent go fix the bug, you don't get that peripheral vision.

If you live in a micro world, you do.

Another example. I was migrating my personal website from one framework to another. And first thing I did, I said, Claude, write me a script to do it. It did this. It seemed like it worked. And I read the script, and I was like, I don't know.

Like, I just don't have a feel for what it's doing. A bunch of files went a bunch of places. It seemsright. So what I did is I said, hey, Claude, make me essentially a video game where I do the port myself.

And the way this works is old website on the left, new website on theright. I just click a button, next, next, next. And at each step, it says, here's the commands I'm running. Your new website's coming to life step by step.

You see it. There's actually file trees down there where you can see files moving. And the result is it's kind of like if I did it manually, but I'm just clicking a button. So I'm getting some of the benefit of doing it iteratively without the pain.

And I think the big takeaway here is agents can write code to help us understand code. Where the point isn't building software to ship, it's building these little micro worlds for us. It's the mathland,right? It's a simulation of just this thing.

OK, quickly, the last topic, shared spaces. So far, this has all been about me solo understanding. But a lot of the time, I think the challenge is actually you're working on a team. And your whole team needs to understand together so you can actually jam and have creative ideas together.

We think a ton about this at Notion. We believe that the shared understanding that exists between you and someone else is what lets you communicate effectively, whether that's names for parts of a system. It could be names for UI elements or concepts,right?

So we think a lot at Notion about how do you make tools that enable collective understanding. Some things we're exploring. Can you have multiplayer chat threads between multiple humans and agents together? So here, I might ask a product manager on my team, hey,

what are users asking for with this feature? And she might say, hey, I don't know. Let's ask a different agent. And that agent comes in and talks to us. What's happening here is that instead of me and my PM both talking to our own agents, we're in a shared space.

We can see each other's communication. It's kind of like going from one-on-one conversations to Slack channels. You see more of the behavior happening together, and you understand together.

Also, having documents that you can talk about together is a really powerful permanent. Here, Claude made us a plan. What if I have a question about that that I want to discuss with my team? I can just leave a comment because this is in a collaborative space, not on my computer locally.

And then I can ask, hey, what do you think about this? And my teammate can chime in, and we can talk about itright there,right? I think having these spaces for shared discussion around ideas with our agents is really powerful for building up that collective understanding.

This, by the way, we just launched last week the ability to bring coding agents into Notion. So Claude and Cursor can now live in Notion. And our team actually builds a lot of our code in Notion itself, mainly because of these benefits, because having it in a shared space is just so valuable.

OK, so we've talked about these three techniques. And I want to bring it back to the beginning. You know, I think at the beginning, I said it's important for humans to still understand how the code works,right? But I actually think it's much bigger than that.

I kind of think it's just important for humans to still understand how everything works. And maybe you all agree, it sounds like. But this, I think, is being called into question now and is something we actually have to actively fight for.

The thing is, this is not a new battle. It actually harkens back to the very origins of our field. Alan Kay is one of the pioneers of personal computing, co-inventor of the modern GUI. And literally 50 years ago, he wrote this essay that I find very prescient called "A Personal Computer for Children of All Ages."

It looks like two kids on iPads watching YouTube or something,right? It's kind of crazy. This is 50 years ago, his vision. But that's not YouTube on the iPads. What he envisioned is, hey, these kids, they're playing a video game, and they're modifying the code as they play it to learn physics.

So the point isn't the computer. It's the kids. The point of computers was to level us up as humans,right? And Alan has talked a lot about how it kind of feels like at some point, computers detoured a bit from that vision.

But I think the exciting thing is maybe now is the time to bring that back.

Here's kind of the meme version of that. I think with AI, a lot of people are waking up to, oh my gosh, code is free. We can make ephemeral UIs, dynamic simulations to understand concepts. We can make debuggers, playgrounds.

And it's like, yes, that's great. And it's actually not a new idea. Like, this was the goal all along. And so I think the optimistic thing that I find really exciting is, hey, it's still really important to understand how things work.

And with theright tools and theright mindset and theright creativity, we can actually understand better than ever before, not less. With AI, we can kind of empower ourselves more, not just taking ourselves out of loops, but actually putting ourselves more deeply in loops than we ever have before.

And I think that's a really exciting prospect. And I hope it's something that we all, together as an industry, figure out. That's all I have for you today. Thank you so much.