Excalidraw: AI and Human Whiteboarding Partnership - Christopher Chedeau

Thank you so much for the intro. I'm so excited to be here, uh, talking about, like, figuring out how do we let AI and humans work in the world of whiteboarding. And I built Excalidraw, and if you don't know about it, you'll see many things about it.

And one of the expectations you probably have about a speaker at the AI Engineer conference is that I talk about AI in every single sentence for the entire talk. So I'm just going to give you a warning: I'm only going to do it for the second half of the talk.

But the first half is going to be a good setup, and you should, like, pay close attention because it's going to be really interesting content. So let's get started. So we are five years ago. I'm working at Facebook, January 1st, and as you—if you've been at any, like, big company—there's the performance review process.

And so you have to give feedback around all the people you've been working with, how good or bad you've been doing, and same for your manager. And one thing that's, like, I'm probably not unique in this room, but there's this little devil in me that is like, nope, you're not going to do it until the very, very last minute.

And this is called procrastination, and I'm really bad at this, and I've tried many techniques over the years. And the one at that time that I worked on is what I call active procrastination. So I know I'm not going to be able to write the performance reviews until the last minute, so may as well do something actually useful in the meantime.

Otherwise I would be doom-scrolling on my phone and not be productive. And so what I ended up doing is, I wanted to write a blog post for a long time, and so I was like, okay, I'm going to write a blog post.

And so I used a tool called Swibler in order to draw these kind of hand-written-like illustrations. And unfortunately, at the time, the website Swibler was down. And I was like, no, come on. I've wanted to not procrastinate and everything, and then it's still down.

And so what did I do? I'm like, okay, we're going to go one level deeper. What would it take to actually re-implement Swibler? How hard would it be? And so I opened Code Sandbox and I drew, like, I wrote some code to use ROFJS, and now, hey, I got a demo.

And the thing I didn't expect is I posted about it on Twitter and it exploded. Like, 200 likes on this first tweet, and so many people replied and commented and were excited about it. And I was like, oh, this is so cool.

And so this was the beginning of Excalidraw. Now, one other thing is, if I were to do this again in the future, likeright now, I would not start writing code myself. I would probably use any of the coding agents that we've seen.

And the good thing is I'm not the only one. So if you look at two weeks ago, during the Anthropic keynote, when they wanted to demo Claude's AI agent, they actually used Excalidraw for this, and they actually had the agent re-implement the most popular request, which is a table layout.

Unfortunately, they haven't sent the PR yet, so if anybody's working at Claude here, we welcome pull requests. Anyway, so one thing you may have noticed is I've said AI multiple times, so I knew you were not going to last the entire half of the talk without it.

So now let's get back to the non-AI part of this. So this was, like, in the story we were, like, in January 1st, 2020, and three months later, you probably all know what happened: Covid. And so everybody had to go home and now had to figure out, like, how are you working from home, like, what's life like.

And a lot of people, one of the big things that they tried to figure out is whiteboarding. And so when you're in the office, you have a physical whiteboard and you can draw stuff and you can collaborate with people, but when you're at home, you don't have any of this.

And I was very confused because a lot of people started coming to me around, like, oh, actually Excalidraw is a really good whiteboarding solution. And I was, like, very confused because I built Excalidraw for drawing illustrations for my blog post.

And why? What does it have to do with whiteboarding? And then I started being interested around, like, okay, what are the other people doing whiteboarding, like, other tools doing? And one thing I quickly realized is that all of the tools at the time basically took the physical limitation of whiteboarding and mapped them one-to-one with the online versions.

So for example,

a whiteboard, like a physical whiteboard, has a fixed size. And so what everybody did, they created a fixed size, hey, create a new whiteboard, and then if you run out of space, then, like, you're out of luck. Now with Excalidraw, you can just, like, scroll around and, like, start writing somewhere else, and it just works.

Then the next one is drawing text. And so on a physical whiteboard, you have to trace every single letter, and on a mouse it's actually very, very hard. And so on Excalidraw, I was like, eh, like, we have a text box, like, you can select, and this is fine.

And when I did the dry run for this talk, like, the people were telling me, like, there's no way, like, even MS Paint, like, 30 years ago had, like, keyboard support, but this was the state of the art at the time.

And one thing to remember is nobody, like, thought about online whiteboarding at the time, so, like, all the projects were pretty small and, like, not very used. And now, all of a sudden, there's been a huge influx. So this is why it was so bad.

And then it, like, the list keeps going on, so if you want to start erasing, they would actually have you, like, erase every single pixel. But on Excalidraw, you can select and delete and it just works. And then, like, I don't know if you've seen, like, the reels TikTok video where, like, people draw perfect circles like this on a blackboard, and this is, like, what you had to do for, like, doing this online.

But with Excalidraw, you can just, like, select a shape, like a circle, and, like, you just draw it and it just works fine. And finally, there are things you cannot do on a physical whiteboard. So for example, like, copy-pasting is, like, just doesn't work, but if you're working online, like, this is the baseline expectation that you're able to, like, select and copy-paste.

And so this is, like, all of this, like, really was the big breakthrough in my mind around the idea of the physical versus virtual whiteboard. And this is, I feel, like, the same place and time we're in with AI integrationsright now.

So in practice, what a lot of the AI integrations today, they're basically taking, like, whatever we're doingright now and then, like, sprinkling AI or re-implementing with AI. And the end result is, like, pretty bad because, like, hey, you don't know how it works.

But the thing is, unlike Excalidraw and the online, like, where we had the experience,right now we have AI like nobody has experience. And so this is why I'm super excited about this conference, where, like, all of the people, like, trying things out are actually sharing the lessons to the other with each other.

And so hopefully, like, as we keep doing these conferences, now we're going to bring the state of the world with AI to be much more, much better. So now I talked a lot about, like, Excalidraw the early days and everything.

And so a lot of people have been asking us, like, how do you integrate AI with Excalidraw? And we've tried a bunch of things. So now, for the second part, I'm going to walk through what are the things we've tried, what are the things that have worked, what are the things that haven't worked.

So the first one is actually not us, but Gorkem, who I think is in the room somewhere, working at Fal AI. And so he discovered this really cool machine learning model that takes an Excalidraw diagram with, like, a palm tree or, like, a sun, and then he's able to generate real-time live picture, like, bit of the image in reaction to this.

And so this is, like, what's mind-blowing to me, where, like, oh, now, like, machines can actually generate images and the model is working really well and, like, the, I think the demo is still working, so if you want, like, you should try it.

But one of the issues is, when we started, like, integrating with Excalidraw, we realized that people don't actually draw realistic images on Excalidraw. And so this is one, like, the first, like, key lesson that we found, is that just because there's a model that exists doesn't mean that it should be within your product.

And I knowright now, like, there's a huge pressure for all of us to, like, just add whatever kind of AI into our product that, like, in order to be, like, AI first and whatever. But I really, like, want you to think about, like, what are the actual AI integrations that actually make sense rather than, like, hey, I just put in AI and that's it.

And in practice, if you do that, then actually, like, the whole product experience is going to be, like, way worse than if you didn't have any AI at all. So this was, like, the first learning. Now, people do not draw realistic images on Excalidraw, but they do draw diagrams.

And so now the question is, okay, so LLMs are magical and, like, they're going to generate diagrams, so let's try it. And so we tried and that didn't work out. And so this is an illustration, but in practice it was even worse, like, there was no resemblance of a house or anything, it was just, like, bad.

And this is where, like, we started realizing that, like, just doing this wasn't going to cut it. So we need to, like, be a bit better. But one thing is, yesterday I was chatting with somebody working at Gemini, and he showed me a demo of, like, Gemini actually outputting Excalidraw files.

So my talk is already outdated, and we probably need to get back and, like, start coding on this. But anyway, let's continue. So the thing, though, that we found out is, LLMs were actually able to generate something called Momages.

And so the name of the project is super weird, but in practice it's a way to describe a diagram. And so in this case, you're just writing flowchart LRA AI to be engineer, and then it draws this specific diagram.

And so LLMs were able to, like, realistically generate this kind of, like, Momage JS syntax. And this was really exciting because now what we're able to do is to, like, encode the knowledge of, like, the diagram and Momage supports, like, many different types of diagram.

And so now we're able to, like, encode all of this and, like, all of the prompts can actually generate the diagrams. Now the challenge is, like, this output SVG or canvas and not Excalidraw files. So what do we do?

So in practice, I can check it to work. And so she implemented an Excalidraw backend for Momage JS, and now we finally have our very first very, very successful integration of AI within Excalidraw. So now what you can do is to, like, set up prompts and, like, put in anything that you want.

So in this case, the flowchart starts with AI and an engineer, but we've seen people, like, draw, like, org chart, diagram, or TCP, or network request kind of diagram, or architecture diagram where you can see, like, classes and inheritance and those kind of things.

And then you get a result which is an actual Excalidraw component that you can play with. And one of the things that is the, like, the title of my talk is the Human and AI Partnership. One of the things that you've probably been very frustrated is, like, you start and you generate something that, like, looks 90% like what you want.

But then you're like, hey, this is an image, I can't do anything with it. And this is, like, the magic of this integration with Excalidraw is now this is just Excalidraw and you can use Excalidraw to, like, make it however you want.

And so, for example, you can get to something like this, where you start modifying the layout, start moving things around, you start adding colors, and you really make it to where and to what you want to be. And so I think this is the mindset that we're going to see more and more with successful AI integrations is no longer going to be, like, one-shot, hey, I'm asking the AI something and the AI gives me, like, something perfect.

But how do you get something that's close enough and then you as a human can start modifying? And soright now this is, like, just one way, like, one run trip operation, but I foresee a future where, like, now you're going to be able to go back to the AI and say, hey, modify this and then go to the much further.

We don't have thisright now, but this is what I think the future is going to look like. So now this is one integration, so I'm going to go through a few other AI integrations that we've been working on within Excalidraw.

So the next one is one of the things you can do with the SaaS offering of Excalidraw is managing files, creating a bunch of different files and syncing in the cloud. And as if you've been using this, most of your, like, documents are going to look like entitle one, entitle two, entitle three, entitle four.

And one of the things that LLM can actually be really good at is generating names for these documents. And so now, whenever you're creating a document, we wait until you stop working on this so we actually have content, and then we're going to prompt you, hey, these are, like, all of the names that we think would match, and then you just click and then you're done.

And so this way, like, it actually removes the burden on you to do it, and AIs are really good at this. Next one is while people are not using Excalidraw to generate, like, good-looking, like, painting kind of things, they people tend to, like, want to have, like, a lot of infographics, a lot of illustrations within their slides or, like, their diagrams.

And one thing that we are doing, which is in, like, alpha, like, closedright now, but we're going to open source and to ship in a few weeks, is the ability to generate illustrations. And so we are really hyped about this because I think it's going to help a lot of people.

And finally, the thing that, like, kills me the most, I'm using Excalidraw every day, and I usually want to integrate logos within my diagram or my slide or those kind of things. And I cannot find a reliable way to remove the background from all of those pesky logos.

So I just took a screenshot of the AI Engineer World Fair from the website, and I'm like, hey, please, like, how do I remove the background? And the thing that I wanted to do is to challenge all of you, like, you're all AI engineers, and probably someone from all of you can create a model that removes the background for logos.

And if you're able to make it work within the browser, send it to me and we'll make it happen and we'll, like, ship it within Excalidraw. So I'm really excited that some of you are going to be able to help us with this and help all of the people using Excalidraw.

So this is the end of my talk, and I have a few conclusions for you. So the first one is,right now, as I mentioned, we're in the, like, physical whiteboard to virtual whiteboard, like, kind of moment happening within, like, all of the products, but specifically, like, for the design products.

And now we need to figure out, like, what is the AI native way to implement those interactions. And so really excited about this conference to, like, start showing, like, all of the potential. The second one is, please, please, please resist the urge to actually ship all of the random models that exist within your app and really only ship the one that, like, you feel like are actually adding to the product and helping your customers getting, like, doing what they actually want to do.

And the last one is LLMs love structure. So if you're able to take your problem domain, problem space, and find a way to represent it in some kind of domain-specific language, now it's going to be most likely going to work way better to, like, have the LLM target this domain-specific language, and then you take this language and without AI you actually output it to your own system.

So this is the end of my talk. I'm really excited that we have this event and conference. And I'm also excited that, like, Steve is going to be talking next about TLDraw. Like, we've been working together for a long time and, like, he's probably going to blow your mind with all his examples.

So thank you.