The End of Awkward AI Transcriptions - Travis Bartley and Myungjong Kim

Hi everyone, I'm Travis.

Hello, I'm Myungjong.

I'm Jae-hun.

And we're here to talk to you today about the end-of-awkward AI transcripts from NVIDIA. So what we're going to go over today is basically how we develop models at NVIDIA Speech, um, for NVIDIA's speech AI. We're going to go over the model architectures, our main approach to how we develop models for our customer base, the entire development process, and our focus on deployment and customization.

So, who are we? Well, we're with NVIDIA Repo. We focus on enterprise-level speech AI model deployment. We cover speech translation, text-to-speech development, speech recognition, speech translation—basically everything so that we can, at an enterprise level, allow customers to provide the best possible conversational AI that is possible.

And our focus is generally on low-latency, highly efficient models that can be used on embedded devices. So, at NVIDIA, there's generally like 4 categories we focus on when we're thinking about model development. First is robustness. Are our models going to work in noisy environments as well as work in clean environments?

What type of sound quality are we trying to meet? Are we going to worry about telephony? Are we going to worry about a bunch of environmental contamination factors? We also consider ourselves with coverage. What are the domains that our customer base wants?

Is it going to be medical? Is it going to be entertainment? Is it going to be just straight-up call center-based? What is the language demands that we need? Do we want to focus on monolingual development or multilingual development?

How will dialect play into our model cases? And is there a problem of code-switching that we should accommodate for in development? We also focus on personalization. We want to make sure that our customers are able to have the exact model that they need for their needs.

Sometimes we've got to focus on target speaker AI. Sometimes we've got to focus on word boosting for uncommon vocabulary. Sometimes we just get down and dirty with old-fashioned text normalization and FST models just to make sure that you're getting the exact output you want.

And then finally, we focus on deployment cases. How does speed and accuracy trade off here, which is more important, or do we want a middle ground of the two? Do we want our models to be have high variety, or are we going to focus on efficiency and kind of lead like the Swiss Army knives in our back pocket?

So, what type of models do we use to accomplish this? Well, if you're from a speech AI background, a lot of this is really familiar stuff. First off, we still rely on the old-fashioned CTC models because we keep finding that non-out-of-regression decoding cases is really optimal if you need high-speed inference, especially in streaming environments where you can just run CTC over a chunk dataset, use that as a streaming model, and you can move on with your next project.

But of course, we still like high accuracy. So when we find that just simply non-out-of-regression is not ideal, then we rely on the old-fashioned RNNT, or you use TDT models, which is a variant developed at NVIDIA. Here, we've used the audio output of an encoder with an internal LM so that we can have auto-regressive streaming setups for deployment.

Then sometimes you want even more accuracy. Sometimes you don't want to worry about streaming, in which case we also offer attention encoder-decoder setups. We are all familiar with them at this point: Whisper, ChatGPT, LLMs. And yeah, we offer this here.

We find that they're really good if you want a highly accurate model with less focus on alignment and, more importantly, accommodate a lot of tasks within a single model. We find auto- we find transformer decoders really great when you need to incorporate speech translation, timestamp prediction, language identification, and speech recognition, all within a single model with simple prompt changes.

But across all of our different decoding platforms, we have one unifying tool, which is that our fundamental architecture is the fast conformer. Through a lot of empirical trials, we found that the original conformer model can be even greatly subsampled so that instead of just the conventional 40-millisecond times-step compression, you can switch, you can add in another step of subsampling and get 80-millisecond compression.

And we find that along with allowing you to have very small audio inputs and thus really lighten your memory load on training, it also makes training a lot more efficient because you can get way quicker convergence with not as much data as needed.

Further, it allows really fast inference because now you're chunking all your data into just 80-millisecond timestamps. And so we found that this is a really strong architecture that we have at the backbone of all of our previous decoding offerings.

And this allows us to split our model offerings into two options. On the left, we have Riva Parakeet, which is focused on our streaming speech recognition cases. And this is where we put our CTC and our TDT models.

And the focus is on streaming. If you want speech recognition, speech translation, target speaker ASR, you're going to run it through a Parakeet model so that you can have very fast

and efficient recognition. Meanwhile, on our other option, we have Riva Canary, which is where we put our fast conformer models. This is where we focus on accuracy and multitask modeling. This is where we're going to get the best accuracy as possible while not really caring that much about speed, though we do push for strong speed.

And so being able to cover all this, we get a really, really comprehensive toolkit in which we can offer to our customers a mixture of fast, multitasking, or high-accuracy models. And most important is that our focus on we have the model to meet the need, as opposed to an idea that one model fits all.

We focus on variety and coverage rather than unifying and keeping everything under one hood.

Given the success of the Parakeet ASR model, it can be easily extended to the multi-speaker and target speaker scenario with the integration of the diarization model, Subformer. This Subformer is an end-to-end neuron diarizer that follows the rival-time-storing principle.

In simpler words, it's standing for who comes first. So the Subformer behaves as a bridge between the speaker timestamp from the diarization and the speaker token that can be recognized by the ASR model. In our model architecture development principle, we fuse the ASR encoder embedding and the Subformer embedding through the speaker kernel in providing the full-fledged who-spoke-what-and-when problem.

And this objective this model can be fine-tuned with a really simple objective similar to the NASR model training perspective. And also, by feeding the model feeding the model with the option no query audio or not, we can make the model conduct the target speaker ASR task or the single multi-speaker ASR task.

And the model our unified model architecture consists of ASR encoder and Subformer, but it can not only be applied in the parallel joint manner, but can also be used as a cascade system just like conventional multi-speaker system.

Yeah, we have additional models we offer to improve further accuracy, customization, and readability. The voice activity detection, VAD, detects speech segments for better noise robustness. And we offer this little VAD and the MarbleNet-based VAD models. The external language model rescores ASR transcription for better accuracy and customization.

We have N-GRAN-based language models in our the Riva ASR pipelines. Text normalization and the inverse text normalization convert spoken terms to written forms in text for better readability. And we have the WFST-based ITN models. Punctuation capitalization, PNC, adds punctuation and capitalization to the transcription for better readability.

And BERT-based PNC models are supported. Finally, the speaker diarization identifies multiple speakers in conversation. And we have the Subformer-based speaker diarization models in cascade model as well as the upcoming for the end-to-end models.

And so, given this level of customization, it doesn't like really surprise that like it pays off when it comes to rankings. On the Hugging Face Open ASR leaderboard, the majority of the top 5 models do come from NVIDIA.

And all it does is come down to this approach on a focus on customization and variety.

But, you know, rankings aren't everything. So let's try out a demo. Let's say you're trying to run a recognition system, but you have a but it's a song you're going over. Can our models work?

Living for the now, long as time allows. I'ma keep on switching different styles, keep creative on a cloud. Sweat is on my brow because I'm running all these tracks just to keep them running back, you know, to drill the chord and back.

And I've been practicing my craft, dedicate this play to Kobe, what could be a bigger legacy than making it as.

Yeah, seems to work pretty well. Accurate transcription even in a noisy setting.

Okay, so that's our architectures. That's our approach to development. So how does it actually go into training? Is there anything special we do with training to

meet this demand? And the answer is honestly not really. We really focus on to meet the fundamentals when it comes to data development. Like many labs, our focus is on robustness when it comes to sourcing data, multilingual coverage, and an ear for dialect sensitivity.

We try and get as much language documentation as possible before so that we know exactly what type of data spans we want to get. We incorporate both open-source and proprietary data, with the former allowing us to focus on variety and domain shift, while the latter allows us to focus on high-quality and souped data.

We incorporate pseudo-labeling, where we use the top-of-the-line models that are available for commercial use, and we take transcripts from them, allowing us to benefit from further developments in the community and further developments from our own releases.

And then for training, again, a lot of it is just standard available stuff. The NIMO research toolkit that is used for our model training is an open-source available library available for anyone in the community to use. It's tools for GPU maximization, data bucketing, high-speed data loading through the Lhotse backend, all available for just general use.

And so we just focus on this approach where we can maximize our data, we can maximize the speed in which we can ingest data across different settings. Most of our data is then stored on a object store infrastructure in which we can quickly migrate between different cluster settings.

And then for validation, just as much as with training, we focus on a nice mixture of coverage in both open-source and proprietary data. We make sure that by the time data by the time our models actually reach end users, we've gone through as much bias and domain testing as possible across all possible language categories just to make sure that our models are as robust as possible.

The trained model is now deployed to NVIDIA Riva through NVIDIA NIM for low latency and high throughput inference. The high-performance inference is powered by NVIDIA TensorRT optimizations and the NVIDIA Triton inference server. And it is available for gRPC-based microservices for low-latency streaming as well as the high-throughput offline use cases.

The NVIDIA Riva is fully containerized and can easily scale to hundreds of the parallel streams, and it can be run on-prem in any cloud at the edge or embedded platforms to support a variety of the applications, including contact centers, consumer applications, and video conferencing.

NVIDIA NIM offers prebuilt container industry-standard API support for custom models and optimized inference engines.

So one of the pain points in real scenarios from customers is customization because every use case every application requires domain knowledge, including the medical terms in medical use case, menu names in food ordering, and some of the acoustic conditions, telephony, and noisy environment in contact centers.

So NVIDIA Riva offers the customization features at every stages so we can fine-tune our acoustic model from the Parakeet-based model, Canary-based models, and we can fine-tune the N-GRAN, external language model, punctuation model, and the inverse text normalization models.

And also, we offer the word boosting to recognize better on some of the product names, jargon, and context-specific knowledges.

So we are offering our the Riva models in NVIDIA NIM. So you can visit our website, build.nvidia.com, explore speech. Then you can find more the available Riva models in the website.

Here is the more thing how to get started with NVIDIA Riva. Quick start guide, developers developers forum, and the fine-tuning guide how to fine-tune our models in the NIMO frameworks.

Yeah, so that's it. Thank you.

Thank you, everyone.

Thank you.