The Voice-First AI Overlay: Designing Conversational Co-Pilots - Gregory Bruss

Hi, my name is Gregory, and today I'm going to be talking about the Voice-First AI Overlay designing conversational co-pilots.

So, conversation is the oldest interface; you can think of voice as our original API. It's the modality that we mastered even before FHIR. But when you're actually talking to someone in live interaction, AI can't really help; it's essentially locked out of the conversation.

So this talk really was just a way to explore this question, which is: as AI systems grow more powerful, can we find ways of keeping humans in the loop with that progress, and on track with that progress, through what is our most natural interface, the interface of voice?

There's a few reasons that this seems like it's on the horizon. The first is that we are developing highly specialized agents that can perform these incredible tasks over longer and longer time horizons. And we also have the entire voice AI wave, where we have conversational agents that make AI very accessible.

You can have a call with a conversational voice AI; it has access to tools, it can search for something, it can come back to you with that information. So we have those two explosions ongoing. But it seems like the user experience for ambient agents that respond not to a text chat or to a message, but rather to an event, is still being defined.

So there was a great talk on this by Harrison Chase last month, called "Ambient Agents and the New Agent Interface," and this talk is just a way of exploring what might be some future options there.

So there are currently these two waves ongoing. The first is the agent capability wave; agents are getting more powerful over time, we have better ways of designing RAG systems, multi-step tool calling acting over longer and longer time horizons.

There's also this entire agent orchestration piece to it as well. And then there's the voice technology wave. So time-to-first token is reducing, latency has improved a lot, and it also looks like full-duplex speech-to-speech models are on the horizon.

So if we try and combine these two, is there a way of offering real-time assistance via agents but in an ambient setting, conversational setting?

So there's a demo, which is real-time conversational assistance in the context of a live foreign language call, in the case where you do not necessarily speak the language fluently. And what you'll see in the demo is caption scraping, smart debouncing, managing the context so that the suggestions, foreign language suggestions that the LLM is giving, are in line with what's happening in the call.

There's an entire LLM pipeline around that, around suggestion endpoint, translation endpoint, and all of this comes together within this voice-first AI overlay, which is then rendered.

C'était amusant, j'imagine.

C'était très amusant.

Quelâge avais-tu?

J'ai été à la ferme de mes 5 à mes 15 ans.

Qu'est-ce que tu faisais à la ferme?

Je conduisais les tracteurs. J'organisais les vaches pour la traite, et je faisais toutes les petites choses avec les petits animaux, donc les poules, les lapins, etc.

C'était un travail difficile? J'adore la vie à la ferme.

Moi aussi. Ce n'était pas si difficile, car je faisais ça sur mon temps libre pour aider mes parents, donc c'était mon choix.

Sehr schön. Berlin ist sehr grün. Wir haben hier viele große Parks, zum Beispiel Tempelhofer Feld, das war ein ehemaliger Flughafen, der wurde verwandelt in einen Park. Man kann dort auf der Flugzeuglandebahn mit den Rollschuhen laufen oder Fahrrad fahren, das ist echt toll.

Kennst du das? Hast du das mal gesehen?

Ja, ich liebe den Tempelhofer Feld. Ja.

Ja, das ist wirklich toll.

Hast du dort schon Rollschuh gelaufen?

Ähm, nein. Ich kann das nicht. Ich glaube, ich würde hinfallen. Aber Fahrrad bin ich gefahren, und Skateboard auch einmal. Aber ehrlich gesagt, ich gehe da nicht so oft hin, weil es ist einfach zu groß und zu weit.

Wie oft gehst du da hin?

Vielleicht einmal im Jahr, ehrlich gesagt.

So I'll just define the overlay paradigm. A voice-first AI overlay sits alongside human-to-human calls and adds real-time assistance into that call, but without becoming a third speaker. So it is native to voice in that these two speakers are, of course, speaking with each other, but it does not try to actually participate in the call itself by having a sort of group setting.

So the typical voice AI interaction is: you have a human speaking with an AI, that AI can then have access to tools, it can look up some information, get back to you. But the overlay paradigm is different. You have a human and a human speaking with each other, and an AI operating in between to enhance and augment the dialogue.

So in the case of the demo, the overlay is listening passively to the natural dialogue and then is surfacing relevant help under a specific context in that call. So language suggestions, phrase suggestions, definitions, things like that. And otherwise it stays out of the way until it is needed.

So you can think of it like the overlay enables an ambient agent, but that is conversationally aware because it exists only within that conversational moment.

Where would overlays fit in? So there are many layers of the stack, and in the landscape currently. We have core speech models to go from, you know, speech recognition, text-to-speech. We have intent and agent frameworks, so agent orchestration, for example.

The overlay wouldn't necessarily be the agent itself, but it can decide when and where an agent surfaces, or where the help from an agent surfaces. It doesn't necessarily need to be concerned with the internals of that agent. There are also meeting bots or note-takers, but these happen after the fact.

They don't really happen in flow during a live interaction. There are voice avatars and full AI callers, which is a very interesting fieldright now. But with overlays, it's not that they participate in the dialogue directly. They simply try to give a way of amplifying the humans that are in the room.

And part of that is, in order to do that, there needs to be UX research on cognitive load, overlay design, timing, and this whole intersection of human-computer interaction plus AI UX research.

Besides the design challenges, there are also engineering challenges to be aware of. So with normal voice AI systems, latency is the most important thing. But with overlays, it's a little bit different because you're not constrained necessarily by, you know, a voice agent getting back to you within a 200 to 400 millisecond window, where it will feel off if it's 400 milliseconds too late.

The thing is, though, if an overlay is assisting a live conversation and that help comes early, it's an interruption. It will interrupt the conversation. If it comes late, then you've missed the opportunity for the help to be of highest value, so it's useless.

If the help arrives but it's loaded with the wrong context, then it's kind of spam. And even if all of that goes correctly, if that help arrives but derails the ongoing conversation, then it's not usable because it hasn't really respected the conversational flow, the conversational rhythm of the two people who are speaking with each other.

And through all of that, the latency still has to be well managed throughout. So if I could summarize it, it's timing, relevance, attention, and latency.

Now, for this design landscape, there's some principles that we can think about. I'd say the first is that for any overlay, we need to provide transparency and control. So you should be able to decide how much the overlay is actually getting involved in the conversation.

It should enforce minimum cognitive load. So it could be the most intelligent system in the world, but if it's entering the conversation in such a way that it overloads the speakers and derails their conversation, then I think that's difficult.

I don't think it would be very usable in this case. And the third is an interesting one, which is allowing progressive autonomy. So in some cases, you know, if you're new to a field or something, you need a lot of help at the beginning.

But over time, you also want to moderate how much help you're getting so that you can also make sure that you are learning in theright way. So finding ways to allow progressive autonomy, I think, is a key design principle.

If you try to build one of these systems, you will almost certainly encounter these four challenges. So I've listed them here as the four horsemen of overlay engineering. The first is jitterbug input. So when someone is speaking, they might pause to take a breath, and there'll be a moment where the speech-to-text is no longer running.

So yeah, debouncing is quite important there. Context repair is also very important. If you aim to give live assistance, then you're probably working with a sub-second speed limit, so the entire pipeline has to be optimized. The third is when help arrives, it could come too early, so premature interrupt, or it could come way too late or not at all, so no-show.

And the way to try and think about that is, if you have very good conversational awareness, you know at what point it is theright time to step in and help. The fourth is glanceable ghost. So you can think of it like attention is a kind of currency, and every time a hint arrives, it taxes the attention of the person that's seeing it.

So it shouldn't be, for example, if it's an overlay in a video call, like, you know, completely obstructing the field of view. It needs to be flexible, it needs to be dismissible. There's a lot of stuff around user interfaces that can be helpful here.

What excites me about this space? So I would say that I find it really exciting that latency is now within striking distance. We can thread round-trip calls to a fast-provisioned LLM provider in, yeah, 500, 700 milliseconds. Time-to-first token is very low in some cases.

I've seen people get, yeah, very, very quick responses back. Another angle is how to make the entire thing private by design. So we're seeing models become increasingly capable, even though they're smaller. If you do run this entirely on-device, how would that work?

You know, how can we make sure that we can get the same level of intelligence but keep it private by default, on-device inference? I think that's an interesting angle. The third is how to inject a strong user experience ethos into the entire concept.

So a UX stance that values human conversation and really respects it, like it's something that is very native to human beings and, yeah, in many ways kind of something that should be protected. And I think the fourth angle is more speculative, but voice as a linkable surface.

So if you have ambient agents in the context of calls or

live human conversations, can they be linked in some way? How would you orchestrate that? I think that's an interesting piece.

Where I'm curious? So one thing that I'm curious about is that ASR errors can cascade. So if you have a word error rate of 5%, and in some of those cases you transcribe "don't" to "do," for example, so "I do want this" versus "I don't want this," the overlay might give the wrong advice.

So how to deal with that? Pairing it with the amount of conversational context that you have would be one way. But yeah, this is something that I'm quite curious about how to think about. Another thing is prosody and timing complexity.

So because we've had voice for so long, and we've really evolved with voice, we are hardwired to detect even micro intonation signals, and those are all lost when we just convert straight from speech to text. So I think that's also something that I'm wondering about.

How much? What is the quantity of that information loss? And can you still get relevant assistance even if everything is flattened from speech to text? And then, of course, the security surface. So if you have agents actually interacting in live conversations, what security risks does that pose?

It feels like a completely new security surface to think about. And yeah, it would be interesting to look into that further as well.

Extensions and future directions for voice-first overlays. So one is, it does look like full duplex speech models are on the horizon, where you do not convert from speech to text in order to do anything. You take just the raw audio through a speech model.

It's not get converted into text. You have audio features. Can you provide contextual suggestions in an overlay paradigm just with that? I think that's interesting. Another thing is multimodal understanding. So being able to see the live call or the live video might give information that makes the AI interaction more helpful.

And then speculative execution and caching, that's also quite interesting. So yeah, I think overall it's a very interesting space to be in. And I think in many ways, just with how voice AI has exploded, it does seem like the future is conversational.

It seems like the technology is ready, but the interfaces are not. And yeah, thanks for listening. I've dropped my email there. Thanks.