Beyond Transcription: Building Voice AI That Understands Conversations — Hervé Bredin, pyannoteAI

Uh, good morning, everyone. Thanks for, uh, being here to the voice and vision session. So I'm Hervé Bredin, Chief Science Officer and co-founder at pyannoteAI. So I'm going to talk to you today about conversations, understanding conversations, and what you can do on top of transcription.

So a quick words about myself: I've been an academic researcher all my life, until 2 years ago when I started this company. Oops, sorry. What happened? Yeah. So basically, yeah, I worked on this topic called speaker diarization, which I'll introduce a bit later.

For those of you who don't know this weird word, that is tricky for me to pronounce. But basically, over the years I built an open source toolkit called pyannote, which focuses on speaker diarization, and that became quite popular over the years, in particular since OpenAI released Whisper speech-to-text open source models.

Basically, Whisper was some kind of revolution in terms of STT, the fact that it was free, that it was very good, but it didn't provide the actual names and tags of the speaker. So people naturally turned to pyannote to combine the two.

And you can see that on the inflection points on the GitHub star history. And just a few words: I happen to turn 45 in just 1 week from now, and we're almost at 10k stars on GitHub, so please give me a birthday gift by just going to the website, to the GitHub website, and add your star.

That will make my birthday.

So,

um, let's go to the core of the presentation. So you all know what transcription, or speech-to-text, is. Basically, that's answering the question: what was said from the recording or streaming of a conversation. Basically, you get from the audio, and you get a sequence of words as output.

That's, for instance, what Whisper does. But without the actual names of the speaker, usually the conversations and the transcripts are not really understandable. So the next step after transcription is basically to attribute a speaker tag to each of the words.

So that's what I call here speaker-attributed transcription. And so that answers the question: who said what. But for some applications, that's enough. Like, for instance, maybe for meeting note-takers, to make a summary of a meeting, that might be enough to assign an action point to theright people, to know exactly that Hervé said that and John said this other thing.

But to understand the conversation a bit better, we might want to go slightly better. Sorry, I forgot about this slide. So basically, there are many cases where knowing who said what is as important as what was said, actually.

So I have here a few examples. Like, for instance, video dubbing, automatic video dubbing.

Knowing who said what is actually really important to put the actual

good, correct voice to theright speaker. So if you want to automatically translate a video from one language to another, you want the voices to be consistent as well. So for that, you need to know

who spoke when. Same for meeting note-takers, and there are other applications, like, for instance, podcast, what I call podcast intelligence, like being able to track a speaker across different episodes of a podcast, or across different podcasts, finding the same guest across multiple podcasts.

But yeah, so I was a bit over just before, but just knowing who said what is sometimes not enough to really understand the conversation, how it goes. So knowing who said what and when actually brings even more information.

For instance, in this example, without knowing exactly when each words were pronounced, you can't detect that actually the black speaker interrupted the green one. You can't really detect that maybe this small backchannel that the green, sorry, the black speaker does during the speech of the green speaker.

If you miss it, basically, you don't really understand that maybe the black speaker is actually agreeing, nodding at what the green speaker says. So knowing exactly when this small word has been pronounced is very important. And same, without precise timestamps, you can't really know whether I made a, I make a pause in between two speech turns, which might convey additional information about my state of mind, my, I don't know, the point that I'm trying to make in the conversation.

And we can go even a step further. Like, we might want to know who said what and when, and how they said it. Like, I have a few examples here. If some of you are laughing at some point, is it because, I don't know, I said something stupid or I was actually funny?

Same for coughing. Maybe there's something in the conversation that makes me uncomfortable and I'm coughing for some reason to hide behind that. And yeah, so I also mentioned stress, disfluency, prosody, the fact that I stress on particular words in a sentence might actually change completely the meaning of the sentence.

I have this example in mind, like, I don't know, in the sentence, "The dog ate the cake." If, I don't know, you're talking to a child and the child tells you, "The dog ate the cake," or "The dog ate the cake," or "The dog ate the cake," the meaning might be slightly different.

So stressing particular words, being able to have a voice AI actually understand this stress and this kind of low-level details might bring some more information to any downstream LLMs or whatever tools that you use after this first enriched transcription step.

And if we take even a step back,

knowing who's talking to who. Am I here in this example? I'm probably can be seen as this gray speaker addressing to all of you. But at some point, if at the end of this talk you do have questions, then I will be probably the green one talking to one particular person, answering the particular question.

So knowing that, having a higher-level view of the conversation also opens up a new bunch of applications.

And all of this also happens in an acoustic environment. That gives you information about the context, whether it's in this quiet room for now, or if I'm in the street. This brings more context to the game, and you can actually take a better decision about what's going on, why a person is actually saying what they are saying, et cetera.

So all of this is basically what we are trying to work on at pyannoteAI, but we started with a smaller problem called speaker diarization, and that's really what I want to talk about today.

So speaker diarization is the, yeah, is usually, what I wanted to say about this slide is that who said what is usually just as important as what was said. And you can tell, by instance, by going through to the Hugging Face model repository and filter all the models there with the ones that have the audio tag.

Where is my mouse? Yeah, here. So this is what I did, and then you sort them by number of downloads. And basically, among the top, here, 7 models that are at the top of this list, 3 of them, the first 3, are actually related to speaker identity and speaker diarization, and the other ones are STT.

So that's one way of showing that speaker diarization is actually quite important in the community.

So going into a bit more detail about what speaker diarization is. So as I said, it's answering the question who speaks when. So starting from the recording of a conversation, basically the first step that people usually do is basic voice activity detection.

So you basically tell whether someone is speaking at one time or not, anyone is speaking at one time or not. And then you can go a step further. You can actually segment those speech regions into smaller speech turns.

So this is what I call segmentation, and it includes finding speaker change points between in those long speech regions you might find, you know, a speaker change point, and also find regions where people are actually interrupting each other.

Like in the first example here, there's definitely here someone interrupting someone else, and it's possibly here some kind of backchannel where someone is actually nodding or saying, "Hmm, hmm, okay," this kind of stuff. And this is the kind of small speech turn that

you don't want to miss, because sometimes they actually convey the most important information of the conversation. If it's a small yes here, and if you miss it, then you have no idea what's the state of mind of this speaker.

But that's not yet speaker diarization. Speaker diarization goes all the way to actually assigning a speaker identity to each speech turn. And so here in this example, the speaker diarization system automatically detected that there are two speakers, the green one and the black one.

But we usually don't have any prior knowledge on the actual number of speakers in the conversation. You might have some kind of guidance on, I don't know, if you are a meeting note-taker, for instance, you have the list of attendees, so you might have an idea of the number of people that were invited.

But that doesn't mean that someone, two people joined from the same channel, or that an attendee that was not invited actually joined in finally. And so that makes the problem kind of difficult in the sense that we don't know in advance the number of classes that we're supposed to detect, contrary to other, let's say, classical machine learning problems.

And also, we don't know the identity of the speaker that we're supposed to diarize. In the sense that speaker diarization does not really output John, Hervé, and Jack, but basically it's more like speaker 1, speaker 2, speaker 3, and they can be permutated and it's still correct.

For example, in this example, if I just change green into black and black into green, the diarization is still exactly the same from the point of view of the evaluation metrics that I'm going to talk about, and from the point of view of the actual speaker diarization task.

So that makes this problem kind of difficult, and that's why even though the community has been working on this topic for a long time now, it's still not solved. There are other reasons, like the fact that we have to detect overlapping speech, to handle overlapping speech, that we have to take into account very short speech turns that.

Have to take into account imbalance between the speech time of multiple speakers in the conversation. All of that makes the problem difficult. On top of the usual acoustic conditions, problems, and everything that any speech processing

task has to deal with.

So I'm going to switch to a demo to show you a bit more how diarization systems are evaluated. Maybe when you stumble into benchmark, you find this DER acronym, which stands for diarization error rate. And so hopefully this will work.

So I have here a Python notebook that I've prepared for this talk. It's actually already available on GitHub. I'll share you the QR code at the end of the presentation so you can play with it. But so in this example, I have a conversation between two women talking over the phone.

Hello?

Hello.

Oh, hello. I didn't know you were there.

Neither did I.

Okay. I thought, you know, I heard a beep. This is Diane in New Jersey.

And I'm Sheila in Texas, originally from Chicago.

Oh, I'm originally from Chicago also. I'm in New Jersey now, though.

Okay, so you get the idea. A 30-second conversation between two women. So this is the expected output of a perfect speaker-attributed transcription. This was manually labeled. And now what I'm going to do this here is actually run a pyannote open source community one model on this same file.

So it's actually runningright now on my Mac. So that's why here I'm using this MPS PyTorch backend. So how it works is that you first download the model from Hugging Face with this line of code, then simply apply it on the audio file.

So now it's finished, and there's this prediction variable that contains,

sorry, I should have done that on the other tab because this was the one that was already processed. I wanted to do it live, so let's do it live now. So it's running. And so the next step is to actually visualize the errors that this system made.

So at the top here, you have the reference annotation, so the expected output of diarization. And here is the output of a community one speaker diarization pipeline that is open source and free to use. So it makes three kinds of mistakes, as I was saying.

Like confusion, here it got the speaker wrong. It can also make false alarms. So false alarm is when it detects speech when there's actually no speech in the ground truth. And misdetection is the other way around. And misdetection can actually happen, for instance, here, during overlapping speech when we actually detect only one of the two speakers here.

And then once we have these basically errors of false alarm, confusion, and misdetection, we can actually compute the diarization error rate. So this is with this library called pyannoteMetrics. And basically it gives us, in this example, a 5% diarization error rate, which is basically the sum of the confusion, false alarm, and misdetection divided by the total duration of speech in this file.

And so we have this other model that I'm currently running. Hopefully it works. So it's runningright now on our cloud API. Basically it's a better model than community one that we called Precision 2 that is running here. Hopefully it will work, but as for every demo, it might fail.

If, no, it worked. And this is the kind of output that you can get. So you see that it got the speakerright here and makes slightly less mistakes in terms of false alarm and misdetection. And overall, in this example, I think we got a 3% diarization error rate here.

So let me switch back to the presentation. So this was about benchmarking speaker diarization model, and I'm often asked how well does state-of-the-art speaker diarization work today. And it's a difficult question to answer because really depending on the use case, it might vary completely.

For instance, in this example, we have here conversation telephone speech, like the one we just listened to, like two persons over the phone. And we can go down to 8% diarization error rate. The best system does that. But if you are now in a restaurant with many friends, with lots of background noise, even the best system reaches like 41% diarization error rate.

So it's far from being a solved problem, but we are working on that.

And so once you have speaker diarization and you have transcription, it shouldn't be that hard to actually find the speaker-attributed transcription,right? It's just a matter of, you know, assigning a word to a speaker, a speaker to a word.

But actually that's not that easy. And there are many reasons why that's not that easy. The first reason is that most speech-to-text models are actually trained on single speaker data. As soon as you apply them on multi-speaker data with overlap, with speaker change, with all kinds of mess, they completely, they fail miserably.

So for instance, when you look at the open ASR leaderboard from Hugging Face, if we look, for instance, at Nvidia Parakeet that we are using, that I will be using in a demoright after that, they report 11.4% word error rate.

When we apply the very same model on the same AMI

on our side, we get 26%. So the question is, why is there a difference? Actually, it's in the way that those benchmarks are set up here.

For example, in this AMI dataset, that's a dataset with meetings between four to five people, I think, in meeting rooms. And there are basically microphones like the one I'm having here, so a headset microphone, as well as one microphone in the middle of the table.

And those numbers here on the open ASR leaderboard are based on the headset microphone. Those numbers here are based on the microphone that is in the middle. And so on one side, you have a single speaker speech. On the other side, you have multi-speaker and even distant microphone speech.

So that's why it degrades a lot. And so that was my point of this slide. So when doing speaker-attributed transcription,

the reason why it might go wrong is either because STT doesn't work great, and it's usually the case that they don't generalize very well to multi-speaker recordings. Four, as I was saying, distant microphone, speaker change, cross-talk, interruptions, you name it.

Maybe code-switching as well when you change language in the middle of a sentence. But also it might be because of the actual reconciliation between diarization and STT timestamps. Though it may sound obvious how to do that, as I was saying, I will hopefully show in this live demo once again that it's not such an easy problem.

Because STT does not transcribe overlapping speech well, because the timestamps disagree between STT and diarization, and because sometimes diarization will detect speech that the transcription will not transcribe and the other way around.

So second part of the demo about speaker-attributed transcription. So first I'm going to apply this Parakeet model from Nvidia that does, by the way, a great job at transcribing

this sample. And so Parakeet gives us this kind of output. So we have this sequence of words, and for each word, we actually have the corresponding timestamps. Okay? So I can play it quickly.

Hello.

Oh, hello. I didn't know you were there.

Neither did I.

Okay. I thought, you know, I heard a beep. This is Diane in New Jersey.

Well, you get the idea. And then the question is, now we have our best diarization. This is Precision 2 output here. This is Parakeet output here. And the question is, now we need to assign a speaker to each word.

So let's do it step by step. For instance, so in this example, even though the timestamps of the words are a bit shifted on the left here, it's obvious that it's probably the yellow speaker who spoke the gray word.

But if you go just the third word. So now we have this. So we have this word, the word O here, that is in between those two speech turns according to diarization. Which one do you assign it to?

So I can listen to it.

Oh, hello. Oh, hello.

It's not quite sure. And there are many other positions where it happens. Here, same. There's a word here, okay, that apparently according to diarization, there are two speakers speaking and only one word from the transcription. So that's why at pyannoteAI we came up with this new STT orchestration thing that basically does its magic and does the job for you.

So here I just submitted to our cloud API

a job to transcribe and use both Precision 2 diarization model and Parakeet transcription model and take care of what I call this reconciliation between the two. And you end up with this kind of output. What's nice is that so now it did the job for you, but in particular, I like to go to this particular example.

Even where there was overlapping speech before, it actually managed to interleave the words from two speakers. Let me play just this part and focus on this area maybe.

In New Jersey.

And I'm Sheila in tech.

And there's actually an end that, or this is the um here that is actually

the yellow lady actually has this hum at the end of her speech and the blue one actually interrupts her. But they're actually overlapping and we managed to get themright.

Yeah, so this is the end of my talk. I wanted to give you a few minutes. I have two minutes left for questions, but I just wanted to say that this demo that I did is already available in the pyannoteai/tutorials GitHub repo.

So in there, you'll be able to play with pyannoteAudio, open source toolkit for diarization with pyannoteMetrics for evaluation, with iPyannote for this nice visualization widget that we've implemented. I stands for not iOS, iMac, or whatever, but really for interactive.

And also the SDK to play with our premium models. And I'll stop here and I'm happy to take a few questions. We have one more minute left.

Yes, please.

So what's the trick that you use to resolve the conflicts? Is it part of the model training or is it some sort of like heuristics that you have around to identify who to attribute the word to?

Yeah. So the question is, what's the trick to actually solve this problem of reconciliation? So that's a proprietary trick. But what I can say is that there's already part of this trick that is available in the community one model, which we call exclusive diarization.

Basically, what we do is that we find a way when there is overlap to actually select the most likely of the two speakers that will be transcribed by the STT model. So that simplifies actually the reconciliation between the two.

So that's one major part of the approach.

The model training is like a couple of heuristics you have to.

Yeah. So the question is about, I'm repeating because I've been told I need to repeat questions. So it's not part of the model training in the sense that we really plan to support any kind of STT without having to change the STT model itself.

So really it's supposed to work with any STT, even fine-tuned ones that you might have internally because you fine-tune them for your particular use case and nobody has it. You can't combine it like that.

18 seconds. So I guess we need to stop. I'm sorry, maybe we can talk offline. Otherwise, they'll beat me, I guess. Thank you very much. I'm sorry.