Voice In, Visuals Out: The Agony and the Ecstasy - Allen Pike, Forestwalk Labs

Allright, I'm Allen Pike, and today I'm going to be sharing some of what we've learned building Voice In, Visuals Out experiences using AI. Here we go. So this is Andrej Karpathy, and he made an argument last month that voice is the human-preferred input for AIs, but that we prefer visuals as the output.

And he knows a thing or two about this stuff, but that's not how we have been, for the most part, building or using AI. We've been typing to it, it's been typing back, maybe with some markdown. But there's been breakthroughs over the last few months where both this Visuals In and Audio—or the Audio In and Visuals Out experience—is now really feasible, and we can create really delightful experiences with it.

The visuals outside is pretty intuitive,right? Of course, a third of our brain is dedicated to processing visual information; we love looking at things. And models have recently got to the point where they can generate rich HTML, tool calling, so we can have these experiences where there's visualizations that come back, explain things, help us understand, communicate the responses from these models.

They can give us interactive controls that allow us to explore and understand and modify and change and direct the models. And they can even respond with beautiful illustrations and images,right? And so the ceiling on the visuals out piece has really lifted, and we have a lot more capability of what we can do in terms of responding from these models.

The more controversial half of Karpathy's argument, though, is this idea of voice as the preferred input. We've long sort of idealized and fantasized about speaking to an AI and having this real-time conversation where it understands what we need and it reacts appropriately in real time.

But the experiences that most people have had so far with voice interfaces have been more like trying to get Siri to turn the lights on and it's not working, or like this guy that is trying to get ChatGPT voice mode to do things but it keeps being awkward and confused,right?

The models that we have so far, the experiences that most people have seen, are both slow and dumb, which is not a great combination. And so a lot of people are down on voice as an input. But speaking is the ultimate way that, as humans, we communicate.

When we're speaking, we have more words per minute than when we're typing, but we also convey more with each word. There's a huge difference in between if you say to me, "Okay," versus if you say, "Okay." And that's why when we have something that is really important that we need to communicate or get through to somebody, we will jump on a call and we will—or we'll speak in person—so that we can get through that high-bandwidth communication.

And so what we've done at Forestwalk is we have built an agent that is in our calls that can help us out in real time. And so the other day I mentioned to my co-founder on a call that I'd seen a bug with this Slack integration, or at least I thought I had.

And she responded that she had seen the same bug. And so I just said, "Okay, well, let's file that as a linear issue," and the voice agent within a second responded that it had done so. And that feels perfectly natural when you get it really dialed in, that you are speaking either incidentally or with a purpose, intentionally speaking to the AI, and it responds in a way that is not interruptive of you.

It doesn't need to be voice; it is just taking action on your intent. And so we're going to see this experience more and more in the coming months and years, but there's a huge barrier, a huge challenge to making this actually work and feel good.

And that is the tyranny

of latency.

It's really difficult to get a response through that whole chain fast enough. We've known since the '60s that to have a computer react to us fast enough that it feels instant, it needs to react in about 100 milliseconds, a tenth of a second.

And so, of course, we are always aspiring to get our products to react that fast, and sometimes we can achieve that, but with networking and everything, it can be challenging depending on what work needs to get done. And so sometimes we might flex up to 1,000 milliseconds,right?

You get a full second. It's about the limit before people start to lose their train of thought. They ask a computer to do something, takes more than a second, we're off to something else mentally,right? And so we're always in that play in between those two kind of human limits of trying to respond within a second, ideally within 100 milliseconds.

But if we want to have a seamless voice conversation with an AI or another human, the limit is much more aggressive. We need to have a 200-millisecond latency or less if we want to have a fully conversive. People are verbalizing something and they are interrupting or interjecting or agreeing and forming that sort of connection, full voice in, voice out conversation.

And so imagine trying to do that where you have a network request, maybe you turn speech to text, and then you do some sort of model inference on that text and then send it back on the network. It's like a ridiculous amount of work to do in 200 milliseconds.

There are some clever approaches of trying to work around that. The Thinking Machines in NeoLab has a really thoughtful architecture where they were demoing recently, just a few weeks ago, they've time-sliced into 200-millisecond chunks, like that 200-millisecond goal, and have a model that's doing continuous inference in and out on 200-millisecond slices.

So there are ways around it for voice in, voice out, but there's also just we don't need to wait for novel architectures. We can just switch to having voice in, visuals out, and then we benefit from the more forgiving visual response envelope that people have.

That if you have something that appears on screen within a second of what you've said, it feels like it has met that. It's within your attention span. It's reacting to you in a way that feels seamless. And so we've been building for that over the last few months, and it's a lot of fun, and we've learned a lot doing that.

And so I want to finish up just by sharing three of the things that we found to be really important for getting within that latency envelope and making the experience feel really delightful for people who are using it.

So the first one is that in order to respond in a way that feels seamless, you have to use a really fast model. So that's not just a model that is so small that it can respond fast enough in a few hundred milliseconds, though of course it does need to be, but it also has to be on an inference platform that prioritizes latency.

So when GPT-5 mini came out, we were excited

that, okay, well, yeah, we get this more intelligent model and presumably it'll be fast, but then in practice we are seeing latencies of 5,000 milliseconds, 7,000, p95 sometimes 10,000 millisecond latency for this small model. It was cheaper than the bigger model, but we weren't consistently or ever seeing it respond fast enough.

Haiku is much better in terms of that p95 latency. So you really need kind of a haiku-class model as the one that's responding in this real time or one of the smaller open-source ones. And then if there is a larger chunk of work that needs to get done, then that model then hands off or sends off an asynchronous message to a larger model that can think, and then the continuous real-time model or soft real-time model that's responding quickly then can interleave in those responses if it's doing something heavier.

So that's number one, you need to have a fast model, obviously with a short enough context that you're providing it that it can respond in a few hundred milliseconds. The second thing that's really critical if we're going to have this feeling of it being really responsive is to have short intervals of how quickly we're sending out for inference.

And so traditionally for a voice application you might listen to the user for a few seconds and then they stop, you listen for a second of silence, and then you have some sort of inference goes on, and then you've blown your budget by a pretty wide margin just for waiting for silence.

So if we're going to have something up on screen within a second, then when we want to have our inference pretty eagerly responding as the person is talking, even if we're not entirely sure they've stopped talking, being willing to send inference every one or two seconds as they speak, because we sometimes have a sentence where we ask, "Oh, hey, we're going to want to change this and actually also listen to this other thing," it feels much more seamless if you're doing those things as people talk.

So you have to have a model and an infrastructure where you're able to get these fast turns going. And then finally, in order to actually make all of that work, you need to have a stable caching regimen. So we have this huge improvement in the way that we work with LLMs over the last year where we have on the different platforms different ideas of this prefix caching where if the beginning of the context you send to the model is the same each time, then you can get up to 90% cheaper, faster inference depending on the conditions.

And so you need to lean heavily into this architecture really. I think for most applications we're moving towards this, whether it's like a long-running agent or it's a frequently running agent, it's the same principle applies that we want the first 90% of the context window if we can to be the same from request to request, and then just use that final 10%, and then also, of course, minimize the number of output tokens so that you can get these really fast and relatively affordable inference turns to create that delightful experience.

So these are some of the techniques we've found really useful. I would love to hear from anybody who has been exploring and experimenting, whether it's with real-time or any way that people are pushing the boundaries of creating delightful experiences with these models.

I'd love to share what we've been learning, and I hope some of this inspires some of you to go out and build something great. Thanks.