Books reimagined: AI to create new experiences for things you know — Lukasz Gandecki, TheBrain.pro

So, my name is Łukasz Gandecki, and I've been programming since I was a little kid, and I want to tell you about my newest project, um, "Books Reimagined," so how to use AI to create new experiences for things you already know.

So, how it all started: I was reading a book about Donald Trump reelection, and since— as you can hear, I'm not from the United States—

there was a bit... a few too many characters to me. I didn't follow everyone, so I decided to vibe-code my way through the understanding. I built a little bit of an AI companion application. Looked terrible, but it gave me context for the people that were on the page, with a little bit...

it found images for them and gave me a little bit of a summary in the context of the... of the page that I was at. And a month later, it turned into, um,

it turned into something different. So this is going to be the snow queen... this is one of the first experiences we've built. This is the snow queen book, and this is the part where the sorcerer's apprentices are flying away with the mirror that distorts the reality.

Um,

so, allright.

Wpadłem tu odruchem, a jeszcze z tak ciężkim zwarciadłem.

So it tells a story about the air flying and flying, and the heaven is so far away. There's music, and it reads, but you can't hear it, I'm sorry. But then the crash happens, and the mirror shatters, and it distorts everything all around.

So this is one of the first experiences we've built, and— but it's all in Polish, so I want to actually demonstrate one that we built just for this conference that's in English. This is 1984, and

what's interesting here, which I don't think I'll be able to show you, is that you can send a quick voice note to the book and ask, "What's going on in this sceneright now?"

I don't really have audio, but the point is that there are many different, uh, AI voice assistants, but they are all— almost always— just terrible. If not all of them, to be honest. Serious terrible. We had a demo from Google yesterday.

They were saying up front that it works 50%, it's usually there's a delay, they start talking in the wrong position, I mean, at the wrong time, they interrupt you. So we've built here a system where you hold it as to just specify when you are speaking, and then you let it go, and it immediately, 100 milliseconds, responds to you.

And then you could scroll further and then ask a question, like, "What happened between last time I asked you a question and now?" and it can summarize what's going on. Um, so you have to believe me that. You can check later on bookgenius.net.

Another thing that we were talking— thinking about— is the search. That's a very common thing, searching. So the most normal search would be just exact search. But if you want to— the way our brains work, they don't memorize the pages.

So if you want to find a scene where Winston met O'Brien, then exact search is not going to work, but embeddings work. So you can quickly find the scene you were thinking about this way, and then you can go to that— go to that spot, read a bit more, and you can go back to the place where you were reading.

But there's also

one step forward— I mean, one more thing you can do. You can basically say, "Talk about all the way the party

propaganda works," and you can do deep research, and it's going to actually read the whole book till the— to the point that you finish that, to give you the answer. So that's very useful. It's going to take a couple minutes, so I'm going to go back to presentation.

So I started with vibe-coding, vanilla JS, very confusing code, but it gave me the freedom to iterate very quickly. You basically don't know what you don't know, and if you start to— especiallyright now, the time it takes to plan everything up front is often wasted, because you can much quicker just tell your thinking to the AI and generate something that works, and then you see, "Oh, that's actually not that great, let's try this and that."

And I realized that throwing away code that you poured your heart into often feels terrible, like you're vested, you'd spent so much time. But throwing away code written by AI actually feels great. Um, so I would describe this as waves of changes.

So basically, once I start feeling that I don't rewrite the whole codebase day after day, like the amplitude of the waves is getting lower and lower, and there comes a time where I can start old-school engineering. I can start adding tests and refactor.

But there are traps to refactoring. Do I refactor the worst piece of code? I would suggest that it's better to focus on low-hanging fruits. So, for example, I had a piece of code from OpenAI Audio Processing, and it's like JavaScript, very quickly written, no types, very confusing, but I never have to touch it, so I'm not refactoring it, although it was very tempting.

So we often think about refactoring by adding this: how bad, how painful, how easy. But if something is very bad and very easy to change but it's not painful at all, then it's probably not a good idea to change it.

So I would suggest that it's better to look at how bad the code is multiplied by how painful, multiplied by how easy. And when all those factors are taken into consideration, then it starts making sense to make a decision.

So a lot of the AI experiences that we see and talk about are basically either ChatGPT wrappers or image generators, or half-working useless voice assistants, including Siri. So our approach was to hide the AI from the user. So when we produce the books, the AI does the initial draft, and we do the rest.

And I would argue that the human touch is invaluable in situations like this. Like, AI cannot tell if the music that it generated is good. It cannot say if the graphics are good-looking or if the avatar is actually matching the vibe of the person that the book is talking about.

So you want to make the AI disappear. And multiple things connected together. Simple things. Simple building blocks make for the magical experience for the reader. There's nothing really new here. You could already ask a friend about a question about the book, but is your friend available 24/7 and all-knowing?

Probably not. You can already search, but is the search the spoiler-free search? Is it natural language search or exact match? Um, so I think that beautiful graphics help you get into the mood of the book and help you with the character recall.

And music that matches the scene makes the experience like watching a movie, and we know that music influences the emotions hugely, and it's very nice when you're reading the book and the music just flows with the book and gives you this great experience.

So nothing new, but at the same time completely new, which is what AI allows us to do nowadays. And I would encourage everyone to think about those tiny little niches where we can create some completely new experiences on top of something that we have known for such a long time.

So in thousands of years, it was never possible to read books like this, nor even really produce books like this, because if I had to do all those graphics and music per every single book, it would cost me, I don't know, hundred thousands of dollars per book, so it never made sense to do this.

So how do we do this? The process is we use a combination of LLMs to do scene analysis, book character detection. We give the AI an overall music theme. So we say, for Sherlock Holmes books, for example, that it's like Victorial London and all that, and it should be gnorr music and kind of on a sad node.

So with scene analysis plus mood detection, we do music generation, and we also do structured XML with metadata. So, for example, we have a text like this, and we read— AI is very good at doing this kind of a mapping, which then is very easy for us to use in the book when we say, like, we can display the avatars that are in the scene.

But it would be very time-consuming for a person to go through the whole book and map every single thing like this. So today we are open-sourcing the player, so anyone can create the Netflix-style experiences for books. And if you want AI that feels like magic, not like chatbots, come talk to me.

We build AI experiences that ship in the light and not slides, although I hope the slides were nice. So thank you, and you can find me at those places.