Full Workshop: Realtime Voice AI — Mark Backman, Daily

I'm Mark, with Daily. This is Aleś.

Aleś.

And then we have a few other Daily folks: Quinn, Nina, Varun, and then I'm not sure where he went, but Philip fromright there, from the Google team, Google DeepMind team. So this session we're going to spend just a few minutes getting everyone started.

The idea here is it's going to be a hands-on workshop where all the folks I just called out are going to be available to help out. We'll walk you through a quick start to get you up and running.

And then the idea is to build something. So build a voice bot in the next 78 minutes and 12 seconds, or whatever time we have left. The one, I guess there's one consideration is the Wi-Fi. If you don't have good Wi-Fi, you might want to try to tether.

I was able to tether, and it worked fairly well, but the conference Wi-Fi was a little shaky. This is real time, so you will be streaming data. It does require a viable connection and not just sending a few bits over.

So just a heads up if you hit that as a snag. So I guess before I get started, who here knows about Pipecat or has built anything with Voice AI? OK, a smaller audience. Has anyone built any real-time applications with LLMs or AI?

Maybe slightly bigger? OK. Great. So Pipecat is

an open source repo. It's a Python framework for building voice and AI multimodal agents. And it's built by the team at Daily, but we're an open source. It's an open source project that anyone can contribute to. It's been around for, I don't know, just over a year now.

Yeah, like, I would say officially Pipecat was March 2024, something like that.

OK, so 13 months. There we go. So just a quick walk through, maybe just to kind of ground everyone in the thinking around Voice AI. These slides weren't built for this talk, but I'm going to use them. So the Voice AI or real-time applications are tough because there's just, you know, we as humans communicate all the time with each other, thousands, tens of thousands of years of evolution baked into our brains.

So it's pretty tough to make a machine work on the same level. So we have great expectations being the user in it. So you need a good listener, some that is smart and conversational. You need to be connected to data stores.

It has to sound normal or natural. Think back to even just maybe two, three years ago, what your voice bots sound like. And many of them, if you call them on the phone, still sound like. It needs to sound natural.

And actually, kudos to the Google team. The latest Gemini Live native audio dialogue is quite good in that regard. It has to be fast. So the whole end-to-end communication needs to happen. And roughly, kind of the benchmark is around 800 milliseconds.

You could strive for more. I think we see maybe on the human level, it might be 500 milliseconds or somewhere on that order. So it is pretty fast. So there's a lot to kind of get all the way there.

And this is something that we at Daily, everyone building Pipecat, has been working very, very hard on getting all the way to meeting all these expectations. So just to kind of ground you in some of this, since we're going to be working in Pipecat, Pipecat has a pipeline.

I don't know if maybe, Aleś, you want to talk a little bit about the origin of that quickly.

Sure, sure. You can think about it as a multimedia pipeline. And you would think, what is the multimedia pipeline? It's basically just think about, like, boxes that receive input, and input could be audio or video. And then those boxes just will stream those same data or modified data or new data

to the following elements or processors. In Pipecat world, we call them processors. So in Pipecat, you would have a pipeline where you have a transport, which is the transport of your data or the input of your data. For example, when you're talking, you could be talking in a meeting.

So that would be the audio of the user. Then you would have another box following that, which is the speech-to-text service. So the speech-to-text service would grab audio from the user. It would transcribe it. Then you would get text.

That would be the following data that goes through the pipeline. And then the next one would be the LLM. So now the LLM has what the user has said. And then it generates output, whatever the LLM. That would be tokens.

Then those tokens are converted into text-to-speech. And then the text-to-speech outputs audio. And then the audio goes back to the transport. So you could hear what the LLM has said. Today, what we're going to do today with Gemini Live, a lot of those boxes go away because

the LLM will do a bunch of these things. It will do transcription. It will do the LLM. It will do the text-to-speech in one of these boxes. But you might still require, for example, if you want to save the audio or record the audio into a file, you need a bunch of utilities to do that.

And Pipecat has all that built for you. So basically, that's it.

I mean, a lot of this is really just, it's orchestration. So if you think about what Pipecat offers, it's orchestration. It also offers a lot of abstractions for a lot of common utilities, like Aleś had said. So recording, transcript outputs, artifacts you might want to produce, or even ways that you might manipulate the information in the pipeline itself.

So this image here, which is what Aleś actually just talked through, is what you would call, I guess, a cascaded model where you have this flow through of information. So we're going to be, you can build with Google and many different services in this way.

In the last year, there's been an emergence of speech-to-speech models that now take audio in natively and audio out natively. And those models also allow for audio in and then optionally text and/or audio out. So you can actually, for example, take a raw microphone input or audio input.

And then the model would run all of its logic. And you can actually opt to have it output text if you want to, say, parse the text output before speaking. So there are a few different demos we'll look at that offer that.

And in Pipecat, we show kind of all the ways to do things because that's what we offer, or at least what Pipecat offers as a value proposition.

Just one thing. I don't think you mentioned it in the slides. But all these boxes, you can plug and play the service you want in Pipecat. So the speech-to-speech, speech-to-text, it could be, I don't know, Deepgram, for example.

The LLM could be Google or OpenAI or whatever. You can just plug and play any service you want.

Right, yeah. The modularity, I guess, is the other big strength. So there's no, you can change out a service without changing out your underlying application code, which makes it easy. And we see with this, a lot of companies that are building for Voice AI might have maybe even a more complex thing.

A pipeline here runs straight down, but you can actually have split branches where you might have one leg that's running some logic and the other running a different. We call that a parallel pipeline. So if you wanted to have, say, a failover, if vendor A goes down, you can move to vendor B dynamically, even within the same conversation.

That's something that Pipecat affords as well. And that can allow you to transfer contexts over. So a lot of really cool stuff. The goal, again, today being to get you familiar with just building a voice agent and building one to get started.

So one of the cool things, like Aleś had pointed out, that with the cascaded models, there's a lot of complexity. But with your speech-to-speech model, things get dramatically simplified. Your code may have looked something like this. This is like an old example of some, like, a ton of orchestration in the pipeline.

But with a speech-to-speech model, you may be able to simplify it down to this. But then you have to remember you actually need orchestration around it. So it does get simpler to some regard. So it's more about the services you interface with.

I think with that, why don't we transition now, because I'm realizing we have only about maybe 70, 75 minutes left to looking at the actual activity for today. Allright, so there's a public repo. I don't know how big or small this is, but it's under Daily Co.

So on GitHub, Daily Co, daily-co./gemini-pipecat-workshop. I'll give everyone a chance to make sure your internet's working.

Can everyone see the repo?

Yeah, it's really tiny.

Yeah, OK.

I should have it in big text somewhere. I don't know. Not really.

No. Here we go.

Allright.

Let's take a look at this. So what I want to do, so I spent a little bit of time, Aleś and I spent time writing up this repo. This is meant to be just a jumping-off point. I'm going to get you oriented.

And then I want to look through one of the bot files, which is kind of the main Pipecat code, with you. And then we'll break and make this an interactive session where we can answer a bunch of questions.

So in the repo, you could either start doing it now or maybe take a pause. But this will give you the steps to walk through getting the quick start running. Before we do that, I want to take a moment here.

Let's see how the Wi-Fi is doing. That's not a good sign.

It's down? Ah.

Well, hey, you know, it is tough to do Wi-Fi for this many people. Very, very tough.

Instead of real time, it's going to be real slow. Real slow.

Yeah, real slow voice communication. Allright. Yes, yeah, there will be some. OK. So here is this Gemini bot.py file. This is all Python again. So everything will be in Python. There will be some client code options, which we'll look at in a second.

Just to orient you, I'll just jumpright into the meat of the pipeline. We have this main function that runs your bot. Everything is going to run kind of encapsulated within an AIO HTTP session. We're going to pass that session around.

That's more of just kind of the mechanics of things. In our pipeline, let's just jump to the simple part here. We'll have Daily as the transport. Daily, we are a WebRTC provider as well as we build Pipecat. There will be context aggregation.

So one important note, when you speak, every turn of the bot is like a discrete point in time. And this is maybe less so the case for a speech-to-speech model. But for basic LLMs, they get discrete inputs. Everything is like a REST API call.

So you're going to get a snapshot of the conversation. The context aggregator is going to collect all the bits of the conversation, both from the user and the assistant. And we'll put that into the form of what the LLMs can handle.

So this, in this case, is more for function calling and kind of logistics and management. Gemini is amazing because it offers a lot of this for you. But if you're to build with, say, just build with Gemini not live, the actual kind of just the text-based LLM, you'd have to have this context aggregation.

That will then go to your LLM, which is going to be Gemini Live, Gemini Multimodal Live. And then it's going to be outputted through Daily again on that side of the transport. So you have Daily. We'd configure our service, which takes a number of arguments.

Like, you set up a room with a token, give it a name, and then have some properties. There are docs, which I'll link to. It's linked in the quick start. There's also a Gemini Multimodal Live LLM service, which is a Pipecat class that is a wrapper around the Gemini Live API.

So this, again, you just initialize and run. With the LLM, you see we do a few special things. We're going to define tools. This one has just two really basic kind of canned functions. Fortunately, we're not calling out to the internet because it's not working very well.

Our connection's not working well. So this one just has the dummy, like, fetch weather. And we'll give you a restaurant recommendation. So these are two handlers that when your function is called, we'll just return this result information. So we have the actual functions themselves that are defined in this function schema, which is you can use just native function definitions using whatever LLM format.

We also created this function schema, which is a universal schema that lets you define and move between any LLM without having to kind of transform your LLM calls from OpenAI to Anthropic to Gemini to Bedrock. Because they're all a little bit different, or Grok.

They all have slightly different formats. So this is more of kind of a universal transform for that. And then they're collected and translated into the native format in this tool schema. So we'll pass the tools then to the Gemini service.

And that's how it gets access to use and run those tools. There's also a prompt above, which I think in this simple example, we just say, hey, you're a chatbot. You have these tools available. And that's that. We're also setting up our context aggregation, which, for better or worse, we use OpenAI as kind of the default, like the lingua franca for context.

So everything gets kind of folded back into OpenAI at a certain level.

And we define the pipeline. So the pipeline is, again, like Aleś had said, just a list of all of your different, or I guess a tuple of all of your different services that are running in the pipeline. And you can write your own.

So if you want to instead make the LLM output text, and you want to either extract information from them, maybe you have it in code, some XML, or some type of information, you can actually extract it, store it.

Maybe your application does something with it. Or maybe you want to inject a text-to-speech type of frame. You can actually do that by separating the LLM, that audio output from audio output to just be text. And then you would add like a text-to-speech service here.

Or you could write your own processor, which maybe not within the next hour. And then lastly, all of these, or not all, but many of them have events they emit. The transport emits handlers for when the client connects and disconnects.

So in this case, we use this line here to actually inject a context frame into Gemini to kick off the conversation. So when your client application connects, it's going to queue a frame. So again, frames being kind of the base format for information.

Think of it as like an object for your pipeline. You're going to queue one of those frames. And what this function does is it just grabs the latest context. So when we set this one above, I think that just says hello.

That's going to pass that into, basically push that into the pipeline, which then it will make its way to Gemini to initialize the conversation. And the rest of this, you could think of as boilerplate to run it. You create a runner, which then is the thing that actually runs your task.

And the task is what runs the pipeline. So maybe beyond today, but just know that that's something that's required to run your code. OK, I'm going to pause here just for any questions, because I do want to get to developing soon.

So you said it was a web, you kind of provide a web RPC, so do you provide web sockets as well?

Mhm, we provide both. That's a whole topic that is a talk in itself. The short answer is, if you're building a client-server app, you should, with like strong emphasis, use WebRTC. It has a whole bunch of properties that are relevant, like error correction, better audio quality, et cetera, et cetera.

If you're building server-to-server, so one of the options today is to build a phone chatbot. You can use, and it's probably the best option to use web sockets. So you can bring your own transport. We actually, in Pipecat, there's a fast API version of that.

That's a server that you can use to exchange messages with a web socket. So yeah, it's really up to you. So I guess maybe the takeaway is, if you're building client-server, you really want WebRTC. You could technically use web sockets, but you'll hit like a long tail of errors when you get to production.

And then server-to-server, totally fine. You're going to be fine with web sockets.

Could anyone download the repo by any chance?

Did anyone not?

Just one person?

Download the repo?

Two persons? Three, four?

Wi-Fi's dead.

Wi-Fi's dead.

Can anyone, can you tether? Is that an option? Do folks have tether hotspots on your phone? Or that's even shaky. Oh, jeez. OK.

You have a hotspot.

I can dance. I could do something. I don't know. You know, I'm not really, you know, it's not my thing. But I could try. True. Yeah, I guess I could just make this code walkthrough.

Or we could write it.

Yeah.

Yeah.

Gemini does have its own VAD.

Yeah.

Gemini does have its own VAD. In fact, if you're using a speech-to-text service, that likely also brings its own VAD to the equation. What we found is that, so maybe to use some of our extra time, because we're having internet issues, the VAD serves a really important purpose of detecting when a user starts speaking.

So in the whole kind of lifecycle of a turn, that user speaking kind of ushers in the user's turn for the conversation. So Pipecat will emit a user started speaking frame. And that will also push through an interruption.

So the user will interrupt like anything that's talking. So if the bot was speaking or whatever, it basically clears the way because the user has expressed that they want to speak. The idea with the VAD is that we want it to be extremely accurate and extremely fast.

So running something on-device, we recommend Solero, which is an open-source option. It works incredibly well. I don't know what the inference time is like. I don't know. Millisecond. Extremely fast. So you're going to get an event back fast.

In fact, you have the ability to tune how long to hear human speech before that event gets emitted. And the defaults are pretty good in Pipecat. And there are maybe scenarios where you want to change that. But the VAD is a really important consideration.

It's extremely low CPU consumption. Quinn has a great spreadsheet of breaking down the full cost analysis of an agent. And really, the CPU is going to be extremely low. Interestingly, the TTS tokens or characters are the most expensive by far.

So when you think about it, running that local VAD gives you superior performance. And it allows you, and there's not much of a cost hit. I mean, it's maybe like a fraction of 1% to run a local VAD.

But yeah, you have all sorts of choices. But we find that to work really well.

Is Pipecat integrated with phone carriers and things like that? Or just web integrated?

It's integrated with phone carriers too. So there are, I found out, I didn't work much with phone. Varun's been our phone expert, though. Varun is like a figure in the WebRTC community. He's like an author of many things.

But he's also a phone expert, which I don't know if that crosses over too much. Maybe that happened before.

Phones are super complicated, how you actually make calls. Pipecat supports all of them. So maybe a very quick list. You can make a web socket connection with a phone provider like Twilio, or Telnex, or Plivo, or Exotel, and exchange media streams.

So that's a way to have just a native web socket connection from Pipecat to Twilio. You'd call Twilio. It's going to emit the web socket. And there'll be a handshake to get connected. You can also use PSTN, which is a public switch telephone network, which lets you dial in.

And that's going to be kind of a different mechanism. There's also SIP, which is its own separate thing. Again, and all of the telephony providers would also support this as well. With SIP, you would call something, say like Twilio.

And you would call into like, say a server, like or a SIP provider like Daily, which offers SIP-provided rooms. And you then have the ability to kind of bring the two together via that SIP connection. The nice thing about SIP is that you have the ability to have like superior call control.

It is slightly more complicated, whereas that web socket connection is instantaneous. Your bot needs to be up and running. So there's a whole, we're not going to talk about it today. But like cold starts for agents, they need to start immediately.

So if you don't have resources provisioned, you don't want your users waiting like 20 seconds while the bot comes online. So a long-ish, medium-ish answer for a very complicated question. Maybe you didn't know that.

Do you have a question?

Yeah, yeah, yeah.

If you compare with something like Carpathia, Pipecat plus Daily, Pipecat and Daily versus something like Carthysia, you know the AI voice?

Can't think of that. Yeah.

Yeah, I'll let Aleś answer that.

Yeah, yeah. Carthysia of.

What was the question?

Yeah, can you, I didn't hear it.

Well, I think you were asking if Pipecat compares to, or how it compares to Carthysia. Is that?

Yeah, Carthysia is, well, I think they're going to do more stuff. But as of today, it's a text-to-speech service. And you can clone your voice, or you can, you know, and then they provide a real-time API. You can just via web sockets, you pass the text, and they reply with audio, basically, with audio frames.

And then Pipecat integrates with Carthysia, as any other text-to-speech service, like 11 Labs or anything. Like think about Pipecat as a, it's just a framework for developers, where they can plug and play the service they want. So they can plug and, they can take Carthysia, they can put Carthysia, they can take it out, put 11 Labs.

Oops.

Even closer. OK. Now I can hear myself.

Yeah, you can plug and play any service you want. Like you can change LLM, you can use Llama, you can use Anthropic, you can use Carthysia or 11 Labs. Then for speech-to-text, you can use Deepgram. Or you can use one box, like Gemini Live, which has all that built for you.

So is that clear? Yeah? Yeah, OK.

What about.

About what, sir?

What rails?

I mean, so we are talking about, what is product,right? The range of production. And for instance, if you are applying to use,right? You can make it live. So what about the response to AI? How are you, like focusing on quick, sorry about my bad English, quick rails,right?

Instead for guarantee that the model is going to respond, something that is correct for the professional and, of course, and the user, say something that could not be, I don't know, for instance, like bursting or whatever,right?

Mhm.

I don't know if I get myself clear, but.

I'm not sure if I understood. Are you talking like how to ensure that the LLM says theright thing or not theright thing? Yeah. Well, Pipecat doesn't have control about that. It's up to you to define the prompt or define how the LLM

will reply. You can put, if you want, you can put, you will be able to write your own processors, that we call them, those little boxes, and check what the LLM has said, for example, before it's, like put some kind of real-time eval, like to make sure that the LLM has, you could do that, yeah.

All that pipeline is very flexible. So you can put whatever you want there, like in parallel, not in parallel. For example, Mark was saying about parallel pipelines. Like if you have video and audio at the same time, with Gemini Live, you can do everything in one box.

But let's say you don't have Gemini Live. You want to use other services, that one does video and the other one does audio. You can have a parallel pipeline, which, you know, it's like a tree,right? You have your transfer input.

And then if it's audio, it goes this way. If it's video, it goes that way. And

you can do things dynamically like that, yeah.

This next question? Yeah.

Yeah, I have a related question. Is that common that people put some kind of check in place? And you see like what is the extra latency of sort that produces? And then for models like Gemini Live, that produces audio to audio, it seems like, do they also produce the text with it?

Or do you have to kind of do speech to text again, and then back to speech? And you're losing all the metadata.

OK. So the question was, are guardrails requirements, and how do people use them? And how does that apply for speech-to-speech models? So the answer is, they're not required. There is a challenge here. And actually, I talked about this on like one of the first slides.

Latency is absolutely critical. So you want to avoid our unnecessary turns. You know, obviously, LLMs are amazing language processors. So if you had all the time in the world, you could do hallucination checking against the LLM. There are other strategies to handle this.

One of the big things is that we see, because of the aggregate nature of the context, it grows over the course of the conversation. You actually, you can find

better accuracy with the responses if you have more control over how you prompt the LLM. So this is a whole topic and talk in itself. What we found is, there are two ways to handle this. Well, at least two.

One, if you, for a lot of conversations, they're going to be task-oriented. So let's say something simple like a restaurant reservation bot. It may have to take your name, get your time, log the time to a database. You can chunk that out, even that small conversation, into just discrete tasks.

And LLMs are really good at following the most recent input. So if you kind of feed it task by task, that helps. Also, if you control the context window, like the size, it can really be beneficial to kind of manage that really judiciously.

So you could either reset. One example might be, let's say you're building a patient intake bot at a doctor's office. The very first thing it may do is verify the date of birth, which serves no utility beyond just the very first checkpoint.

So you may actually remove that from the context, like completely get it out of there. Because otherwise, it's just cruft that hangs on. And instead, you kind of reset and then maybe roll through the tasks. You could also, for really, really long conversations, summarize the context.

So you may want to do an out-of-band LLM call. This is something, actually, Quinn, just we talked internally about this, that we're going to see more and more of this mixture of LLMs, where even in the context of real time, you may have an out-of-band, like REST call to the text-based LLM just to do a summary and then return it back so that you can kind of compress that context window.

And just to give a call out to Google, the live APIs, so maybe transitioning there, they offer context management through a bunch of different strategies, like a rolling, they have a rolling window or sliding window. I think they offer like token caps for that, so you can have some control.

Or if you want, you can output text and then kind of do whatever you want with it. They also take text input. So there's a lot of flexibility with speech-to-speech models. They do offer, or they do pose some other maybe development challenges, but offer like tremendous benefits in terms of the features they offer.

I'm going to, there's a question.

I'm going to hold questions just for a sec, because I've heard the Wi-Fi is back. Can folks try downloading the repo again?

This is Slack channel.

Slack channel, yes. Quinn, can you maybe come to the mic? And I can't remember what you told me.

Workshop Voice Gemini Pipecat, with dashes in between, on AI engineer Slack.

OK. Does everyone know where that is? I know this is like day one, hour three.

Workshop-voice-gemini-pipecat is the channel name.

So if you go to the AI engineer Slack and search for Workshop Gemini, it should come up. And Quinn and I will be posting links as we go along, if the Wi-Fi stays up. If anyone can get on that channel, can you raise your hand?

Can people, like do they have, do you guys have access to the Slack, AI engineer Slack?

There's two channels. Well, Workshop Gemini Pipecat and Workshop.

Oh, yeah. There's like, yeah.

So we'll tell you what that happened when the Wi-Fi was down and we were trying to create it.

Allright. We'll take a question. Should be the join Slack group.

Allright. I'd say, I'm going to take that one and then I'll come over. If I could, let's just try, maybe listen and also try to get the repo. I'd like to walk through the quick start, and then we can look at some examples.

You mentioned moving to fast for local processing. Do you find that there's an easy advantage to moving more to Pipecat locally? Or is there kind of offsets on the lack of processing power?

You know, it's something, actually, Quinn has done a ton of work with this. I have not personally. I mean, there's obviously massive latency benefits, because you cut out network round trips all over the place. So you save a ton.

And depending on where you are in the world, that can save a lot. If you're US-based, your network latency is going to be relatively low. But a lot of the developers in the community are in Europe. And a lot of these AI services are relatively new, with data centers only in the US.

So there are different challenges when doing that, though, when running it. Actually, there are great hosting providers like Modal that offer really good options for running like your own local LLM, which then you're not, you know, you're buying or I guess leasing like the GPU time instead of running everything on a GPU, which would probably be cost prohibitive, because of the way processes run.

But that is also a talk in itself. Next question.

Just want to state that.

Sure.

Have you seen specific use cases where you have to enforce things or instead of just pushing knowledge to the cloud?

Oh, yeah. Definitely.

But that can also, from your experience, does that cause more latency to your local?

Well, the large context windows definitely cause LLMs to process slower.

The question.

Yeah, I'm sorry. The question was around state management with LLMs and whether it's better to, I guess, chunk or have more kind of deterministic input versus just a large context where you just dump everything in. This is actually an extension of what the other gentleman was asking.

The idea being that, and actually, so daily, we built the chat widget that's on the homepage of the Voice AI World's Fair page. I personally built that. What was interesting there is that, and Alesh and I were just talking about this, function calls in the context of real time are still slow, unfortunately, like too slow.

Actually, Gemini, I'll give more props to Gemini, being like maybe one of the fastest. If you run like a basic local, like one of these demos and you ask it what the weather is, it will return back with time to first byte in, I don't know, less than 500 milliseconds.

Whereas other vendors not trying to throw OpenAI under the bus, but it has gotten slower. Like you might see upwards of 1.5 to 2 seconds of waiting just to get that first token back. And we dug into that, actually, this is just something recently this morning.

The issue is, when you get the normal streamed response for the conversation, you can start playing that audio out once you get the first sentence. The issue being, when you get a tool, you need the entire JSON response before you can actually do anything with it.

So that's slow. That's one part of it. Separately, chunking the prompts is absolutely the way to go. And building that World's Fair bot, I was kind of balancing between the two worlds, because you can talk to it and ask it about speakers for the session.

Route one would have said, let's use like a MAG approach and put all of the speaker JSON in something that could be a tool that could be accessed. And I tried that. And unfortunately, it's just a little too slow.

It's a giant context. It takes a while to come back. What's interesting is, if you instead move that all just directly into the context with Gemini Live, it's a little bit variable. But under good conditions, you'll get a response back on that like 800 millisecond latency.

So it actually has access to like the full context. The one trade-off, though, is what this gentleman over here was asking is accuracy. It's going to get confused, because especially when you get a JSON with a lot of speaker, and this isn't, this is all LLMs.

It's not specifically Gemini. With that type of even structured data, it becomes very hard to kind of discern what's what when a lot of it looks the same. So it's all, I mean, a lot of this is emerging like other things in AI, trying to just do it as fast as possible with voice.

Before I take one more question, I want to check in on, have folks had any luck with the repo? I've got a thumbs up. Allright. I'm going to pause on questions for the time being. We can take them at the tail end.

I do want to try to go through the quick start, if we could.

That would be great. And again, so now we've consolidated channels. There's one channel in the Slack. It's Workshop hyphen Voice hyphen Gemini hyphen Pipecat. Quinn is in there, answering questions. Room will join. And we're going to share links there.

So Mark, that's great.

OK. Great. So I would recommend, maybe we'll just take like a few minutes of independent getting set up. If you all go to the README in the Gemini Pipecat workshop, and if you roll through the first few steps, maybe get a, I don't know, a hand up.

You could just flash it up real quick, so I could see when people start to get through it. You don't have to hold it up. If you don't mind, if you brought a device. OK. We've got one, two.

Ah. So for this workshop, I have leaked my key, a key from one of my accounts, which I'll cycle after this. So you don't need to sign up for a Daily account. You do need to sign up for a Gemini account.

I don't have a key I can just give out. So in the environment.example, you'll see there's already a Daily key, which you can use.

Do people know how to sign up for a Gemini?

I have it in the README.

You have it in the README.

Is through AI Studio, is that a good spot to do it? OK. So.

You can do that on your own computer.

Allright. I'll take a question while we're waiting.

Have you standardized the events from different real-time APIs in Pipecat, or do you still need to do it based on a particular?

So in terms of the, that's a good question. In terms of the Pipecat interface, you interface with the Pipecat class. So there's a service class. And within Pipecat, so how you write your application code is going to be uniform across all of the services.

The individual providers haven't really, unlike the text-based LLMs, haven't really settled on kind of a standard. So Pipecat handles all that translation on your behalf. And I think there are other frameworks that do similar things. The idea is to provide like a uniform, simple interface so that you could take, and this is part of the modularity.

If you wanted to, you could swap this bot out for OpenAI real-time or a text-based model with a TTS and STT paired with it. That's kind of the whole idea. There is maybe a little bit in terms of the one thing LLM providers maybe, I don't know if anybody here can nudge anyone you know, the system instruction or system prompt is not uniformly dealt with.

I don't know if that's well understood. But OpenAI has this like user that is system that you can inject anywhere at any time, which is really fantastic. But Anthropic and Google require like a named system instruction that's a special one-time, like at constructor time instruction.

So there's some differences. As best we can, we unify.

Allright. Any, how are folks doing on getting quick start going? I see a hand up. Is that a question? OK. I'll take, maybe it's related to the quick start.

Oh, yeah.

Great topic.

Are there any ways you can move in terms of rapidly?

Yeah. So there's a question about noisy environments, which is like Voice AI's kryptonite. So with a VAD, no, not at the moment. But there are, again, this is where like Pipecat is the assembler of all things. You want to plug in?

That you can run separately from the VAD. We found like Crisp is one that's a partner of ours. They have fantastic noise cancellation. You can run something outside of the loop that would actually clean up. Like in Pipecat, it would be in the transport itself.

So in that audio input, it would take the audio input and remove any ambient noise, so like chip bags opening or dogs barking. But maybe more impressively, human background voice. So it will remove that from the feed. So you could be in this conference, and it picks up the primary speaker for the device like incredibly well.

At the moment, they're the only ones that I'm aware that know how to do that and do it that well. But it's, I mean, it's phenomenal. But you'reright. The VAD is, I mean, it was one of the biggest problems that we saw until we found Crisp.

And they're fantastic.

Is that C-R-I-S-P?

K-R-I-S-P.

OK.

That's big props to the Crisp team.

Which feature of the speech model are you guys usingright now?

Well, we use all of them. Pipecat's open source. So we bring, we have, I think options are Gemini, Multimodal Live, OpenAI Real-Time, and then AWS just launched a new one called Novasonic. So we have those three within Pipecat.

What's the, is the latest feature?

They're all pretty, they're all very much on, actually, well, they're all very similar. I'm not going to, I don't want to like nitpick on all of the vendors here. But they're, I mean, they have strengths and weaknesses each, because it's still an emerging field.

But latency-wise, that is not an issue. Latency is fantastic for all the providers. Yeah.

Allright.

OK. Maybe Alesh will walk through a quick start. Why don't we, he's going to just do some live coding here. And maybe this will help to understand how this all works. And then perhaps we I stop chatting. And maybe you could grab me if you have questions and just do some heads down working time.

I have a few.

What's that?

Yeah, I cannot type. And.

OK. Teamwork.

Teamwork. Yeah. I'll try from the very, very, from nothing, from scratch. So this is a Python project. Actually, let me try again. Yeah. This is a Python project. The first thing we'll do is create an environment, like a virtual environment that is called, I like to call it .env.

That's how you create a virtual environment in Python. A virtual environment will have.

Can we turn off my subtitles? It's so hard to see them.

Oh, yeah.

What's that?

Your screen is hard to see.

Oh, it's hard to see. How do I do that?

Can we go faster? Try that. Can we use this like VS Code?

Better?

Now you're talking.

Can you change your theme to like white for the time being?

I guess.

I only have one theme.

Maybe.

The only issue is I only have one theme.

Is it, can you see it better?

Yeah.

Oh, actually, yeah.

Better?

Nice.

OK. Yay. OK. So we'll start with, yes, it's Emacs. You can judge me. I'm not going to change. I have a few years only coding. So that's what it is. So I'm going to create a requirements file. Sorry.

I loaded first thing. I did the virtual environment. We did this. Then we're going to load it. Now we can start installing packages in this environment, like Python packages. So the ones we have to do is Pipecat, of course.

So I'm just creating a new file, which has Pipecat AI. And Pipecat has a few options. So for this example, we're going to use Daily. We're going to use Google. And we're going to use Cylero. Right? Cylero is the VAD that we've been talking about.

The other thing we'll do is we will load the, there's a package called Python.env, just to load the environment variables that we'll have in this, in here. OK. That's it. One file. Now we're going to, I'm going to create

an environment variable. I'm going to just copy it

from the quick start. It's just a .env. And we can take a look at it. I will, these are the API keys. We'll get rid of them after this workshop. So, but yeah. These are the, you just need two keys, Google API key.

And just go to AI Studio and create your own one. And a Daily API key. This one we already, this one you can copy. It's a bit long. But I don't know. It doesn't.

It's in the project.

Yeah. It's in the project. Yeah. Yeah. If you are able to go to the repo on GitHub, you can just copy it from there. Allright. Next, we're going to create a Python file. Let's call it bot.py. Again, empty file.

First thing we'll do is typical, I think it's called name equals main. No help from an LLM. And then we do asyncio.run. A little bit of help. asyncio.run. And we're going to write a main function. Function. And then let's write the main function.

Allright. So now we start. So the first thing we said is we're going to need a transport. The transport input is like we're going to, like the Daily transport. We're going to say, that's, I'm going to speak to the bot through the Daily transport.

That's what I'm going to create. And then we want, it's a transport for incoming audio from me talking. And then a transport for outputting what the LLM will say. So I'm just going to create a transport. And this is a Daily transport.

And it has a few arguments. It has a room URL. I'm just going to create mine for now.

This is, again, a public room that I will have to delete. I don't need a token. And params.

OK. And then the Daily params is like, what do we want this transport to do? So we're going to say that we want input enable, which means get audio from the transport. And we also want

to send audio to the transport. OK.

Allright. What else? It's complaining for something. Oh, yeah. And we also need VAD analyzer, which is going to be Cylero VAD analyzer. So the transport is going to be able to use this VAD analyzer to detect if the user has spoken or not.

What's that?

Oh, yeah. And this is going to be AI engineer. There we go. OK. Now we have the transport. Now we're going to create the LLM. In this case, it's Gemini Live. So I don't need to create a speech-to-text or text-to-speech.

We just create the Gemini Live. And for that, I'll need to copy it, because I don't know that from memory. But,

oops.

That's in this file, Gemini bot. I just want to copy these lines here. There we go. Allright. So this is my LLM. And again, it uses this Gemini Multimodal Live LLM service.

Gemini Multimodal Live LLM service. That's going to add my import. OK. And now it needs a couple of things. The system instruction, which is like what the agent is going to do, and some tools. The tools, I'm going to skip them for now.

So let's do the system instruction. Again, I'm going to copy it from somewhere. There it is. This is like the prompt, like the main prompt of the, well, yeah.

Allright. So the system instruction is, you're a helpful assistant who can answer questions and use tools. For now, we're not going to use any tools. You know what? Let me get rid of the tools. Just copy here for later.

I'm going to command this out. And you are just a helpful assistant. OK. So that's, allright. So no complaints here. Allright. And now we just create the pipeline.

I'm going to avoid storing the context, because I don't think we need it for now. And this is the pipeline. The pipeline just receives a list of

processors or elements. And the first one is a transport.input. That's the input transport. So how we get audio from the Daily room in this case, the LLM, and the transport.output.

Allright. Now we need, this just defines the pipeline also is another processor. So you could build a pipeline of pipelines of pipelines of pipelines. So you can build, or you can plug and play

the way you like that. So how do you run a pipeline? You need a task, what we call a pipeline task,

that receives a pipeline. And the pipeline task also has some params, which are called pipeline params.

And we're going to say, oops, that we allow interruptions. And I think that's enough.

And how do you run a task? You can create more than one pipeline task if you wanted. In this case, we just have one. Usually, you just have one. You're going to create a runner. And guess what? It's called pipeline runner.

It's a pipeline runner. And then we just do await runner run task. And some completions, please. Pipeline

runner. Allright. And I think that's it. We'll try it. Oh, OS, import OS, I think. I think there's no more warnings. Oh, and I need to load the environment variables, which is this line here. Load.env. I'm just copying it from another file.

OK.

And where do we get load.env? This is just a function

that imports the environment variable. Allright. And yeah, let's try. I'm just going to open the, oops.

I'm just going to open the terminal here. And I'm just going to run it. I think we call it bot.py.

No model. Oh, maybe I need to install the requirements. I forgot this step.

There it is. OK. So at the beginning, I wrote that file requirements.text, which had a bunch of, well, again, just a few requirements. But I forgot to install them.

In the meantime, I'm just going to go to the Daily room that I just pointed the bot to.

No video. OK. So that's,right now, it's just me in that room. And now we just have to wait for this to finish. And hopefully, the bot will join the room and we'll be able to talk to it, hopefully.

Yeah. How important is Daily as part of the explorer? Like that room component, like is it interchangeable?

It is, yeah. This is because we're using the Daily transport. And the Daily transport just connects to a Daily room. But you could have a WebSocket transport and then use Twilio with a phone number and Twilio being connected to that.

If we have time, we can even try that. So I think.

About small web RTC transport. Can we talk about small web RTC transport and why you didn't use it?

You want to talk about that? Oh.

Yeah.

Yeah. I'll wait.

So in Pipecat, we also have added, based off of the AIO RTC Python package, which is how web RTC package in Python, we've added a new transport called small web RTC transport. It is a peer-to-peer web RTC communication that's free.

So it's separate from any vendor. Though, the one downside is that it requires turn server, which you bring your own. So we weren't prepared for that for the conference. And also, just the conference Wi-Fi makes that a little challenging.

But normally, if you're running any of the, we call them foundational examples in Pipecat, think of them as the essential examples that show how to do very specific functions. There's probably about 100 of them in Pipecat. But one by one, it shows you how to record, or add an STT, or push frames, or show images, or sync images and sound.

Those all use the peer-to-peer web RTC transport. So we would have loved to have used that. You wouldn't need a key. But unfortunately, firewall rules have trumped.

Allright. So I'm just running the bot and see how it fails, because it has to fail the first time.

Yeah. This is the, because there's a bunch of the Python packages. And Python just decides to take a time to load them. But you see how easy it was to write

like a voice agent with Gemini Live, if it worked. It's just a few lines of code that we wrote in, I don't know how long it took me, but maybe like 5, 10 minutes.

Yeah. Are there any questions on the example or? What's that?

It worked?

What worked? The bot worked?

Yes.

OK. Allrighty.

Allright. Nice.

You need to print with good Wi-Fi.

Yeah.

I didn't see my phone last time. There was a couple of things that popped up there with the words later. And where are you at in terms of production ready and stuff like that for commercial use?

I can take that.

We have, I don't know the actual number of customers. But I mean, Pipecat probably serves hundreds of thousands of calls a day. I don't know. A lot. Quinn, you probably have a better idea.

Oh, yeah. I mean, Pipecat is used by some very large companies in production. And people are contributing to it from NVIDIA, AWS, OpenAI, Google, lots of big companies.

Yeah. There's one thing we didn't mention about Pipecat is that what you see now in the screen, this runs on the server side. But we do have client SDKs for Android, iOS, JavaScript, and React. And I think that's about it.

But Ragnarok, even a C++ client, if you want. So yeah. So that's the server side. But you can plug your client and connect to the agent on your phone.

Neural web RTC connections or like what?

That would be, that depends on the transport you use. But yeah, you could have your client connect to a Daily, or we support LiveKit as well. But to a Daily room, we like Daily, because we work at Daily.

But you can connect to a Daily room. And then the bot would connect, or the agent would connect to the Daily room as well. And then that's the transport, the web RTC transport, yeah.

I think there were questions there. I can't see.

One second. One second.

Say it again?

So are there other ways to test the bot without other devices talking to it directly?

Yes. Yes, there is. Actually, for the previous version, I just hacked together a thing that I call release evals, which is a bot talking to a bot. And what it does is, I put this bot up. And then it joins a Daily room.

And then I have an eval bot. And the eval bot, what it's going to do, it has a prompt, which is, ask a simple addition. OK. And then that eval bot is going to connect to the room. It's going to add, what is 2 plus 2?

And then the other bot is going to reply, 2 plus 2 is 4. And the eval bot, the LLM, it checks if the answer of the user is correct. And the user, in this case, is another LLM. So it verifies.

It's like an end-to-end. The good thing is, we used to run more than 100 examples every release just to make sure they work. So I just got tired of it, because it's very painful and very slow. So we have this eval bot or release evals that are going to test each service.

Like we test Gemini Live. We test Cartesia, Deepgram, like all the services like end-to-end. And then the bots basically talk to each other with voice. That's the nice thing. So yeah.

OK. Oh, is this on? Maybe real quick, just to show, for those that didn't see it, it is working. Hey, can you tell me a joke?

Why don't scientists trust atoms? Because they make up everything.

If you build Pipecat, I've heard that joke probably, I mean, not like 5,000 times. You can even try to seed it with something different. But it still doesn't work.

I will try to come up with a new joke. Yeah. Why do you always say the same joke?

That's an interesting question. Why do you think people keep telling the same jokes?

I don't know. You tell me.

From my search, there appear to be several reasons why people repeat jokes. Enjoyment and reinforcement. People repeat things they find funny, because they want to re-experience the good feeling associated with laughter. It feels good. And repeating it is a way to try and recreate that sensation.

Memory aid.

OK. OK. That's enough.

Repeating something can help you remember.

Is there anything else I can help you with today?

No, thank you. That was it. Thank you so much.

Well, just to show there was a question about interruptions, we should just have it. My favorite is to ask it to tell you a really long story and then interrupt it. So can you tell me a really long story?

OK. I can do that. Here's a story. And feel free to interrupt whenever you like. Once upon a time.

OK. Actually, tell me that new joke.

Following the clues.

Hey, tell me that new joke.

Oh, you'd like to. Why don't scientists trust atoms? Because they make up everything. OK. Here's one.

That was the same joke.

Because they make up the same joke.

Allright. Allright.

That was it.

Well, I find a lot of conversational agents chunk based on when you pause talking. That's the term.

Oh.

Is there any

does RTC, is there a way to have it do something after talking?

So I think the questions asked during this workshop could map out years of work. So this is another one of those fantastic cutting-edge things. So again, back to human evolution. We all know. And when we talk, actually, it's even hard for humans to talk to not speak over each other.

So the way that it works mechanically is when the user stops speaking, the VAD has a timeout. You tell it and program it, wait, let's say, one second, 0.8 seconds, half a second, whatever feels natural. And you're trying to balance low latency response with giving the user enough time to speak.

It's a really hard thing. And one of the biggest complaints is that agents will speak over the human. So if you're, let's say you're building an interview bot. Like you're using Tavis, one of their digital twins. You want to have a real likeness.

And you want to speak to it. You may take time to think, because sometimes you have to take time to think. And that's a really difficult thing for bots to do. Because again, it's driven by a simple stop speaking algorithm.

So this is a new, I guess it's like an emerging field of models, which is looking at semantic end-of-turn. So driven off of things like

speech filler words, pauses, intonation, so things in the audio realm, and also things in the text-based realm. So just looking at context. So we've actually started, we're one of many that are doing this, I think. But we launched a model.

If you look at it on GitHub, it's under smart-turn. It's a native audio-in classifier that runs an inference on the input audio. And it simply outputs either complete or incomplete. And the way Pipecat uses this is that if you get an incomplete response, we can dynamically adjust the VAD timeout.

So we can tell the Pipecat bot, OK, he or she has not done speaking. Let's actually move the, let's give three seconds to complete the thought. And if it's not done, then the bot will actually respond. So you can create a little bit of dynamic interaction there.

And that's one of the first things. I'm sure the Google team is working on similar things. I know OpenAI is. And all the STT vendors are also looking at their own thing. So I'd say,right now, it is very much an unsolved problem.

But I would imagine, given how fast things are going in the next 12 months, we'll have great solutions that will make it even more natural to talk to a bot. It's a good question.

Any more questions?

Any questions?

Well, actually, I'm kind of wondering,

is there a way to see the transcription of the text as it's happening?

Oh, yeah. Yeah, yeah. This is actually back to the, well, this is specific to Pipecat. But also, Gemini Live will output audio and text. And other speech-to-speech LLMs do this. Pipecat offers, in terms of its, again, orchestration role, when you get a, actually, it's going to be specific to TTS provider.

There are great TTS providers that do word and timestamp synchronization. So they'll give pairs. They call them alignment pairs. So if you're using a Cartesia, or an 11 Labs, or Rhyme, they all output these pairs. One of the really cool things with Pipecat is that the TTS services output not only the audio stream, but also the text stream.

So they'll output text frames, TTS text frames, we call them in Pipecat. And if you place, we have, in terms of how the client software works, there is an observer role, where you can actually watch. There is a process that can watch things that happen in the pipeline and emit events.

So we've instrumented that for the clients. So that whenever you see those text frames move through the transport, you can get synchronized word and audio output. So in your client, if you wanted to have word-by-word output synchronized to the audio, you can do that with Pipecat.

And it's as simple as just adding an event. I think you listen to bot TTS text output, or on bot TTS text. And it will give you the synchronized output.

If I wanted to build a fully offline voice agent, what box in that pipeline would be the hardest to update?

Fully offline. Well, they're all doable. They're great models. I think it really depends on what your bot needs to accomplish. A lot of the state-of-the-art models, to do all the best and smartest things, need to have some. They're going to be run on-prem or in the cloud.

But if you have, and a lot of bots do jobs, like if you wanted to build a restaurant reservation one, like I referenced earlier, it's a very simple job. You could probably run it with some version of Llama running locally.

There are great local, something, again, Quinn has been experimenting with, a lot of great local models. Whisper has challenges. I mean, it has some challenges as an open source model for STT. But there are good and emerging TTS services.

So things are only as good as the input. And we've actually seen this with some of the speech-to-speech models, that sometimes they mistranscribe. So you really need, I mean, every part is critical. But if you can't transcribe the speech really well, nothing really matters.

It has to understand you. And having disfluencies, or hallucinated responses, or even just inaccurate responses kind of breaks everything down. So things mostly start at the STT. So maybe that's the hardest. I don't know if there are a lot of good open source options for thatright now.

Is it similar to TTS?

I don't know. We're not doing anything in the STT world. No, no. No. That's a whole different ball game. Good question, though.

Just made me realize we could have used local models and avoid this.

We could have. Yeah. Well, we're partnering with the Google team.

Yeah, yeah. Yeah, exactly. Yeah.

Has anyone looked at any of the sample projects and had questions? There's a lot of interesting things there. If any of this has interested you, we do have a Discord. You're welcome to get on it. You can find us at pipecat.ai.

And find our Discord there. You can ask questions. There's some really cool stuff with Gemini that can be done. In particular, in the Pipecat repo, we built, I don't know if you know the game Catchphrase, where you describe a word and something guesses it.

We built a version of that. We had to brand it something else called Word Wrangler. And you, as the human, you describe a word. And then you have the AI agent try to answer it. So we built a client server version of that, which I linked to in the repo.

And then we have one that's a phone-based one that's, I think, particularly sophisticated and interesting. So you might think, how the hell would I build this with a speech-to-speech model? We actually use two Gemini agents in the same call.

And we use a parallel pipeline, where one agent is the host, giving out the questions to the human user. The other is the guesser. And we kind of limit the audio flow so that the guesser, the AI player, can only hear the user.

So there's a bunch of really interesting things getting into the, majorly into the weeds of some of the powers of Pipecat. But it also speaks to the strength of having just native audio input being really, really helpful. So I'd recommend checking those out.

Really cool, easy demos to run. One's Twilio. The other is, again, a client server. I think it's like a React Next.js project.

Word Wrangler.

What's that? Word Wrangler?

Yeah.

Yeah. We could run the Word Wrangler client app. It's actually just on the web.

Test.

Welcome to Word Wrangler. I'll try to guess the words you describe. Remember, don't say any part of the word itself. Ready? Let's go.

I'm going to skip to something easier. OK, this is something you take pictures with. It's on your phone.

Is it camera?

Allright. This is a field related to the study of languages, I think.

Is it linguistics?

Allright. This is a game with a yellow ball you play with rackets. Hit the ball over the net.

Is it tennis?

Allright. This is a round dessert with chocolate chips sometimes and other fun goodies.

Is it cookie?

It's really good, even when I'm bad at giving answers. So pretty cool. This is built with Gemini Live. But again, just an example of things you can build

with voice AI. So cool, unique interactions.

Allright. I think that's about it. Thanks, everybody.