Function Calling is All You Need — Full Workshop, with Ilan Bigio of OpenAI

Cool. Okay, let's get on with it. So, hi everyone. My name is Ilan, uh, I'm on the Developer Experience team at OpenAI. Um, unfortunately I can't be there in person as much as I would love to; I'm in a wedding in Costa Rica, which is happening later today.

So I just wanted to take this opportunity to just talk through, um, one of my favorite concepts in maybe all of, like, AI and language models. So, uh, title of this talk is "Function Calling is All You Need."

It's a talk workshop. There's going to be, um, a lot of coding. Please save your questions. No, I'm kidding. Like, just interrupt at any point. We have a Slack. Send them here. Um, yeah, just send them at any point.

And if you want to, like, unmute yourself and/or raise your hand, I'll call you off. Like, as we go, the idea is to keep this super, super dynamic. Um, since we have a bit of time. I'll be fielding a lot of, like, questions and requests and trying to be coding as much as possible.

So, yeah, this talk is going to be some lecturing, a lot of coding from scratch, and then some debugging. Hopefully not a lot. So this is a little bit of what the workshop is going to look like. We're going to go over a little brief history of the Toolformer's, uh, of, I'm sorry, of Function Calling.

Then do a little crash course on Function Calling. Um, talk about just agents, how they're just loops, how RAG workflows and more are just function calls, delegation, and asynchrony. A couple random things I've found, and then we're going to do a Q&A.

Um, if you just want to see, like, the meat of it, this is pretty much it. We're going to do, like, this is everything I want to talk about, and everything that we're going to implement. So this is, uh, just a little bit upfront.

Great. So, a little history. Um, if we look at the abstractions and sort of patterns that we managed to do with language models, it started as text completion,right? Like, the original GPT-1 and the GPT-2 and the GPT-3 were all just base models, where you gave them some input text, um, and then they would just continue the sentence.

This was, at the time, really, really, like, interesting. This was the first time we were able to do, like, very, like, English-sounding, like, real-sounding language. Um, but getting it to follow instructions was pretty hard. So if any of you were testing this back then, you might remember how setting up a chatbot was, um, non-trivial,right?

You had to, like, get it to answer questions. But if you just say, like, you know, "What is the best way to, uh, get to the, like, park?" or something, it would continue, like, "What is the best way to get to the park?"

Um, that is what Sally said yesterday,right? And you wouldn't actually get a, a response. Um, so you had to, like, structure it in a way where you would say, like, "This is the question. This is the answer. Question, answer, few-shot, and then give it."

Um, then they introduced, uh, I think this was actually us, we introduced, uh, function call, um, instruction following with, uh, instruct GPT. Now you could give it some input, uh, and it would actually do what you're asking as opposed to just completing.

Um, finally, we started to introduce this notion of, like, users and assistants and roles. Um, and this was all done through, through post-training, where you actually gain these personas. Um, and then finally, uh, we, we eventually landed on this, like, you can give it additional tools, um, in order to do, like, external, um, any, like, interact with external states.

So this is what the previous playground used to look like. But as you can see, there's no chat. This is just, like, a window, and it'll complete. Now, looking over at, like, the original papers, which is pretty interesting, um, one of the first times that we actually started to do this, like, function calling was through this WebGPT, um, which was this, uh, version of GPT-3 that we trained to be able to.

Okay, um, Ilan, I have to cut in here a little bit. We got the Zoom working. Yay!

Sweet.

Um, allright, we're going to cut over in the audio. So can everyone hear Ilan when he speaks? Uh, Ilan, say something.

Hello. Hi everyone.

Hello, hello. How's it going?

Okay, so, so we're going to cut over a little bit in terms of, like, um, people having their own personal audio situation going on. So if you mute your machine, you should be able to hear it on the room, uh, audio, whatever you want to do.

But you can also obviously connect. But, uh, yeah, now we have him on the big screen. So okay, yeah, let's, let's keep going for a bit. I'll, I'll cut in again if there's audio issues. Or maybe you want to say some test words.

Uh, you're all great. Thank you for coming.

Can we bump up the audio? Where's the guy?

I think it's not a conference if you don't have AV issues.

Okay, uh, it's, it's still too soft. We can't, we can't hear you. Um, I'm going to try and bump up the audio. I'm so sorry.

I'll put up a more interesting slide in the meantime. You guys can look at this while we figure this out.

Where's, where's the fucking video guys?

Hey, um,

so anybody got

good jokes?

Maybe?

I always ask OpenAI for my jokes.

It's always the same one. I think it's, uh, what was it? Why did the, no, I had it. I had it here. Wait, wait, wait, wait, wait, wait. There we go. Why can't you trust an atom? Yeah, because they make up everything.

These are all real, by the way. And for, for those who, like, have a keen eye, the first one is, like, actually from the GPT-2 paper. Um, we, like, gave the model this, like, description of, like, unicorns in the Andes Mountains.

And that was, like, the big, like, first time that it was, like, doing multi-paragraph completion, like, continuations that referenced earlier parts of the conversation. It's cool stuff. It's cool history. Can you guys hear me?

I feel like some people are having a great time.

Yes, we can.

Some people don't know what's going on. Like, I don't know. Am I good to keep going, or should I keep waiting?

Uh, I think, I think you're good.

Okay. Great. Great, great. I do think this is the vibe of the whole talk, by the way. There's, it doesn't get more structured from here. Um, okay, so as I was saying, uh, we did, like, this WebGPT paper.

Um, essentially we trained a GPT-3 version of the model, uh, or, like, a GPT-3 model to be able to use this, like, this very specific set of functions, um, to do web search. And this was, like, back in 2021.

So really we had, like, WebGPT a long time ago. Um, but this is, like, one of the first times, or maybe the first time, that, like, we were having, like, it's not just generating text, but it's generating actions.

And then we're parsing those actions and then introducing it back into context so it can use the responses itself. Um, and how we trained it, there, um, there's, like, these, these, like, you know, clever ways of, like, we, we essentially gave people, uh, an interface and let them do the searching.

Um, and then I think we took it, like, read it, um, yeah, explained, like, on Five. Um, and then just, like, had people complete tasks and they could use these commands. And so we taught the model, and this was GPT-3,right, um, how to essentially imitate users' behavior and then produce responses that were preferred.

Um, and, and this was pretty cool. Like, this is how we, we, we, like, start to saw, um, to see, like, this, uh, this use of, like, structured, like, actions, essentially. So, but this was very specific,right? We were training, like, very specific, um, tools.

So then you might be familiar with this paper. This was from Meta at the time, um, where they essentially had a way to teach the models how to use any tools. Um, and they taught, like, they, they used a few tools.

I think it was, like, QA, calculator, um, what was this, like, translation, um, couple other, couple other tools. But, um, it was actually a pretty clever way where they, like, looked at the log probes at each spot to, like, see where it was, like, best suited to, like, retroactively put in a function call given some, like, completion.

Um, so here we can see a few examples. Like, um, essentially it's like, if you have a calculator call, if you insert a calculator call that, like, you know, it'll, it'll insert the, um, the actual call to the calculator, which then, you know, you're pretty familiar at this point, will get the answer.

If you insert it in theright spot in the sentence, it actually reduces the perplexity of the sentence. Um, and so they didn't actually have a lot of human-labeled examples, or I think it was just a few. But it was, uh, really cool because there's this way of, like, um, it could learn to use, uh, any of these tools through this, like, crazy, like, log probes technique.

Um, and it was, uh, I was pretty excited when I saw this paper. Now, this is, like, how it learns to use any of these tools. Um, but then finally, uh, in June of 2023, uh, OpenAI launched just general function calling, where we essentially, like, pre-trained it to be able to use these tools, or actually, actually post-trained it to be able to use, uh, tools.

So now you don't actually have to, like, give it, like, you can give it examples, but, uh, we just showed it, like, this, like, syntax in, with functions that we still use today, and it's just able to call functions.

So this is a brief history of, of function calling. Um, and my, I guess, my argument is this is really most of what you need for all the exciting stuff that's happening today. Um, there's, there's obviously, like, additional, like, systems you can use and, like, um, more post-training you can do.

But fundamentally, like, functions are so, so, so powerful, and we're going to look at a few cases today. Um, I'm going to try to keep an eye on questions, but, um, yeah.

Okay.

I think you're good so far.

Cool, cool. So let's do a super quick crash course on function calling. Uh, two main purposes, and I'm ripping a lot of this from the docs. So, um, they're, they're fetching data,right? Reading APIs, retrieval memory, or taking action.

Any APIs you can use to write. Managing application state, which is actually pretty overloaded. That can be, like, UI, front-end, back-end, whatever you want. And then workflow actions, which is, um, any, like, multi-step processes, or even, like, meta actions, like switching its own prompt, or, like, loading in different tools, or, like, handing off a conversation.

Right? Um, so this is also a diagram straight from the docs, but, um, I'll quickly brush past this. I'm going to assume, like, most people have at least seen this, but here it is. It's, uh, you, you essentially tell a model which functions you, you want it to be able to use.

Um, and you also provide, like, whatever the user input is. Um, the, the model tells you what it wants to do with that function, but it doesn't actually do it. This is one of the, like, a big sticking point in function calling.

It doesn't actually use the function itself. It tells you, like, it tells you the intent of what it wants to do with the function. You are then responsible for parsing that, executing the code, doing whatever you want with it, and then providing the result back to the model.

And then the model can use that respo-like result in, uh, in the generation. Uh, take a look at a quick question here. Oh, no. It's just, it's just swings. Okay. Cool. Um, these are just a few, uh, best practices.

This is all taken from the docs as well. Um, you want to write clear functions. An important one is you got to apply software engineering best practices when you write these functions. So, um, you know what? Maybe I'll pull up the docs for this.

Um,

so this is a lot of big text, but essentially, um, this is a lot of value here. I tried to pack as much, like, useful information here as, as I could, so I'm going to quickly go over it,right?

Um, you got to explain the purpose of each parameter, use a system prompt, and include examples. That's, you know, pretty, pretty, like, not, non-controversial. Um, software engineering best practices is a little bit, uh, more interesting,right? You got to make functions obvious and intuitive, and they got to follow the function of least principle.

Um, like, if you give this to a person and they don't know how to use it, then the model might not either,right? You know, models are getting smarter than us, but still, you got to make it, uh, you got to make it easy.

Um, also, you got to use enums and object structure to make sure that, like, you are not letting the model make invalid calls,right? Um, like, here you, you have this, like, toggle, like, uh, um, function that takes in, like, two, two Boolean params.

And, like, obviously, like, this is pretty wrong. But, um, there's actually many, many more, like, subtle cases where, um, like, the, the, you can, like, you're letting it represent invalid states. Um,

okay. Still no questions. It swings. Okay, um, great. And then this is where, uh.

But there actually, sorry, there are a couple of questions in the slide.

Okay.

Because I saw a routing conversation.

I see, I see. Um,

here, maybe let me see if I can pull up the slide.

Okie dokie.

Between functions and tools, in my opinion. Hey, Sam. Yes. Um, I, I think we've all been kind of, like, gravitating toward functions and tools as, like, the two main ways. Originally it was just functions. Tools was, like, later we renamed it.

Um, I think now, uh, and the way that I tried to specify it in the docs is functions are, like, the, like, raw function calling,right? Like, you provide an interface and you're responsible for executing the code. Um, tools, and this is sort of how we treat it in our API, is a super set of functions.

Tools include functions, but it also includes things like, um, code interpreter, or file search, or any of these, like, um, like, hosted solutions. I'd say this is not be, like, end-all be-all definition, but this is the definition that we've adopted,right?

There's, like, tools which are, like, hosted tools, uh, including functions, and then functions as a subset. Cool. Maybe we can just jump into it. Um, if people have questions on this, happy to, happy to field them as well, but I kind of want to get coding.

Okay. Uh, when you start to approach dozens or hundreds of functions, what technically should we apply in order to effectively tool call? Permissions is one technique. Okay, interesting. And then question two. When you require one tool to provide inputs to another, I have seen tools become layered.

How should a reasoner hard code? Great. Okay, these are actually great questions. And, um, I think to, to answer them, I might, I might, like, do a little hack and, like, uh, use some existing, uh, code. I'm going to use Swarm for a little bit of this, um, because it does some nice function calling.

Actually, no. You know what? I'm going to get to these func, uh, in a second. Um, let's just go straight into the, uh, and as always, we should start from the docs.

Uh, cool. So.

Uh, so this is where I always step in. Can you zoom in on your screen? Every single screen that you.

Yeah.

Yeah. Bigger.

Cool.

Yeah. There. Yep. Sorry. This is the thing I always do.

Yeah, yeah. No, this, this is a good callout. Uh, and then I think the

terminal here

should be good. Okay.

Sweet. Uh, we have, we have a function. Great. Um, and then

I can just run it. And we have, uh, the function call,right? Now, we're not handling it yet. So this is where, like, I might skip ahead a little bit, um, and start doing some of the, like, agentic stuff.

But first, uh, first off, uh, we, we got to have a book,right? Like, you know, you, you, I can sort of hear myself. Um. I think Soham. There we go. Perfect. Okay. So, um, I think the idea here is, like, let's make a very, very, very simple, like, uh, input loop.

Like, let's do, uh, you know, while.

Can you also switch to light mode?

To light mode?

Yeah.

Jesus. Okay.

Yeah.

Sure.

I'm, I'm just very experienced with this. There we go.

Yeah. Well, that's the least painful. This is good,right? This works?

Yep.

Oh, my poor eyes. Um, actually, maybe I, I, I can't quite see the room, but show of hands, like, who has used or implemented function calling in the past? And someone's going to have to gauge this for me.

Okay, it's everyone. Great. I'm going to skip ahead a little. Um.

There's, like, 10, 20, 20, 20% that hasn't done it.

Okay, 10, 20%. Um, the important part that you have to know is you can define the function, um, schema,right? Then the tool will specify what you want. And then in this case, you know, if I have, like, a

get weather tool, actually, I'm going to grab this from the docs as well. When in doubt, just go to the docs, you know? It's always good. So

step one, no, this is all node. Yeah, and this will be useful for later too. Um, step one, call a function. Step two, execute your code,right? So here what we're doing is we're taking, we're parsing out what the, what the function told us.

We're parsing out the args, and then we're calling this get weather function, which we don't have yet. Um, but conveniently, it's up here,right? So have it up here. Get weather, requests. And then the last step is we provide the result back to the model.

Um, uh, provide the result back to the model and then ask for a completion,right? So just in order, specify the tools, call it the first time, get the tool calls, parse them out, call the function, append, and do that,right?

So if we do that and we just add maybe the, like, add, uh, I'm going to do this by hand. Why not? Yeah, so I'll print the completion here. Start it by hand. And then print the last completion.

Then what we can see is we'll get the first one that includes a tool call. It'll call the actual temperature, uh, call the weather API, and then we'll get a response.

And then it says the current temperature in Paris is something. I can't see it because of Zoom. Um, cool. So this is, like, the very, very basic setup for function calling. Let's take a step forward. So this is an agent, uh, very, very, very basic implementation of an agent, um, that I'm going to go through really, really quickly because we're going to start using it.

This is very familiar to what you'll see in Swarm or any of the other, like, like, basic frameworks. Um, but the idea here is in, uh, as you can see, like, in the original one, essentially, like, when it had a tool call, I wanted to provide it back.

So what I do here is while, like, just keep looping. And this is, like, the very famous, like, agents are a loop. This is, this is that loop. Um, specify the tools, call the model, get the message, print it out, handle the tool calls, append it, and once we have no more tool calls, break.

This is the whole loop. I called it run full turn. In my head, one turn is just, like, you let the model do everything. Um, and then we have this, like, execute tool call, yada yada, whatever. Um, so now we can use this.

So

agents.py. Did I export it?

There we go.

I love it. Okay. So now we're just going to specify one. Um, you know what? Yeah, we will, um, and we'll, we'll do it, we'll do it with a, with a simple loop.

So we have this, that, and we have this. Okay. So now we can just do agents.run full turn. The one other thing that I'm adding here that I didn't show is, um, this simple utils, uh, that defines this very, very, very useful function.

So functions to schema essentially takes in a raw Python object function and then provides it into the, uh, like, correct schema, um, so that you can just define functions directly. And this is the same thing that we have in Swarm, and there's in a few other frameworks now as well.

Um, so as an example, we can do, like, you know, get weather, and I'm just going to, like, you know, return 20, like, degrees Celsius.

Great. I'll do that here.

And then I'll have my messages

and for messages.

Is this emergency reference? I think it is.

What did I call it?

Oh, oh, oh, oh. I see. This is not part of the

agent class.

Okay.

Cool. So we called the weather, printed it out, uh, gave us a nice little completion. Um, this is essentially what we want to see. If I say, um, and then I can just keep, keep going,right? So we, we, we have this basic, um, basic loop.

Um, now let's get back to the presentation real quick. That was, like, a very, very immediate crash course. Now let's, let's get interesting. Um, and, and just for convenience, I'm going to, like, pivot to the Swarm implementation. Um, the main reason being it's pretty much the same, um, except we have, like, some convenient, uh, like, looping tools.

So, uh,

Swarm import Agents, and then

there we go. So now I can do a simple agent and run a demo loop, and let's see this work.

Cool. Um, we have this very basic setup. Let's do everything now. So, um, another show of hands. How, who here has implemented, uh, RAG?

Okay. What about memory? Okay. Fewer. And then what about, like, you know, multi, multi-step things and workflows?

Cool. Um, how about this? There's a lot of stuff we could talk about, but I want to keep this valuable to you all. Out of this list, can you, like, just type in the Slack what you want to, what you want to see?

And I can just change the order in which we'll cover this, um, because they're pretty, they're pretty interchangeable. We can, we can build up. Um, but essentially there's, there's more interesting things later on, but I want to make sure we, we can build up to them.

So just, uh, if you can pull up the Slack and just dump in, you know, what you're interested in seeing.

Okay. Lots of memory I see. Um,

function generation from the docs, delegation, I think. Okay. Build delegation.

And random cool stuff. I will get to the random cool stuff. Um,

okie dokie. Back to light mode. Um, yeah, let's start with, like, a very, very basic form of memory,right? Um, how would we, how do we do this? Let's see. Honestly, like, we can have, um, just a list,right? Memory can just be this list.

And I'm going to implement this similar to how it's done in ChatGPT. Um, but just, just to show, like, I think the whole point of this talk is, like, or this workshop is, like, doing things from first, first principles and just really removing the complexity.

I think there's, like, a lot of, like, not fake complexity, but, like, added complexity on things that, like, doesn't really need to be there a lot of time. Like, concepts are a lot more simple, and it's all about function calling.

So, uh, we can do, you know, add, like, add to memory, append it, and then get memory. Uh, like, this is super, super simple. Um, is this, is this good enough? Let's see. Maybe. Maybe. And so what we can do is

give it the tools. Let's see. And then when will we want to use them? Let's say you, like, I can just say, so this comment is going to be used as the, uh, string, uh, as the, as the description in the function.

So you can say, like, you know, like, when the user tells you something, like, factual

about themselves,

their life, or

I can't see anything. Okay. Or their

preferences, call this function.

Um,

memory. Um, and we'll add a couple more cool things, like expiration, which is one that I've kind of wanted to add for a while. So, you know, for now, let's say false, um, memory.append,

uh, think.

And then we have, I guess, what we call it, uh, you know, memory text.

And we can say, you know, keep the memory text short size. Great. Cursor knows what it wants. And then we can just return maybe, like,

like, this is super, super, super naive implementation. Um, but now, uh, when we start off, I guess we could even start off with a, I mean, this is going to be kind of hacky, but I can just, like, um, in your first turn, not always call get memory.

This is not, uh, great. It's just because of the demo loop, but maybe I'll break, I'll break the demo loop out so we can actually do this by hand. But so

how, how could we prove this? Um,

maybe let's say, like, you know, write this memory bank or, like, keep this memory bank in a local file, JSON file, read it in at the beginning and write it out at every case.

Cool. Okay. I trust this. Shall we test it out? So,

like, someone needs to meet themselves.

Uh, okay. We got the memory, memory. Can anyone, can anyone see any bugs? Because we're about to test this out. So we got the loop. It's going to call it. Um, and maybe, like, let's

memory.

Okay. Sure. Yeah. Let's try it. Let's try it out. See what happens. So, hi.

Did I not give it the functions?

Is it hallucinating this?

Oh, yes. I think we called it functions.

Hi. There we go. Okay. So called get memory. There's nothing there. I can just say, like, um, I am six feet tall despite what people think.

Uh, cool. Uh, so now let's just check,right? It should have written this out. There we go. So now we have it in the memory bank. So now I can actually, uh, end this,right? And then be like, you know, uh, how tall am I?

Ta-da. We've implemented memory,right? There is a, yes, I'll take a clap. I saw Luis, you're like my proxy for the Luis, uh, Luis Costa. You're my proxy for the audience. You're like the only person I can really see.

So please don't, uh, turn off your camera. Um, amazing. Now we can do more interesting things,right? We can, uh, if we want, do a little bit, like, smart quering, uh, where instead of just, like, loading in all of the memory, um, we can, like, do a little bit of, like, retrieval, uh, to load in theright ones.

Um, and use, like, semantic similarity or use some kind of search. Um, I, I could try to implement that. Um, that might take a little bit longer, but not that long. But I do want to pause here. So, like, given this and, like, this is going to be the style of things that we do, like, what, um, what do we want to see next?

I can just keep going with this example. I can pivot. Uh, it could be fun just to keep building on this, see how far we can get. Let's see.

Uh, delegation, async, have it chat and work in the background. We'll get there. We'll get there. I got that working this morning on Python because I didn't want to switch to, to Node yet. Uh, delegation async. Okay. Let's get into delegation then.

So there's a few different ways we can do this. I'm actually going to leave the memory, and we're just going to keep building on this, on this, on this agent. Uh, and I am using the Swarm agent just because I didn't want to debug the one that I implemented.

But, like, if we actually look at what this is, um, like, it is very, very, very simple. And then the, like, run_demo_loop itself is, like, uh, just printing out messages. Um, and what it's doing is, like, appending client.run, whatever.

And, like, if we look at this client.run, um, if it doesn't stream, it essentially does exactly what we did before. Like, keeps looping, get the completion, throw the messages in, uh, append them, handle responses, etc. There's a couple more things around context and handoffs that we, like, don't, don't really have to look at today.

Um, but we can. So

cool. We have memory. Let's do delegation. Um, so there's a couple ways we can do this,right? Um, there's, if you think of, like, functions and agents and, and everything, you can, um, maybe I'll skip to this slide. Skip, skip, skip, skip, skip.

These are, like, a few of the forms of, like, agents and delegation that people might be familiar with. We have handoffs, which is like the Swarm style. You take a conversation and fully swap it to a different agent.

Um, and what that means is just, like, replacing the system prompt, replacing the tools. Um, you can have nested calls, which are the easiest to implement and, like, often somewhat overlooked. Um, and then you can have manager tasks that's more async.

Uh, we will get to that today. So

let's do a very basic one,right? Let's say I want to do, like, um,

maybe let's give it a chance to, like, call a bigger model to do a harder task,right? So we can say, like, you know, uh, I can delegate to smarter model. Uh, and I'm going to give, like, the task description.

Um,

so I'm laughing because this is saying, like, it's just going to make it smarter by telling it to be smarter. That's not, despite how well that would work, usually we're not going to do that. So we're, we can just make an API request directly.

Actually, let's just do that. Let's just do that. So, um, let's do from OpenAI client. And here we could do client.chat completions create. Let's call

o1. Um, I won't provide a system message. I think that's okay. Content is the description. Look at that. And we did it. So now, um,

man, I love Cursor. Okay. Did everyone catch all of that, by the way? All we did was, like, implement this, this function that calls OpenAI API and then here. So now I can say, like, uh, so now I can add a bit of a description here.

I'm like, like, uh, if, you know, if the user asks you to, like, um, I don't know, do something, like, that seems difficult or says it's hard, use this instead of try it yourself. Uh, and it infers, you know, how to use this based on, like, the fact that I called it task description.

It's, you know, so let's give this a shot.

Hi.

Um, let's see. Uh, you know, give me a poem about, I don't know, dogs and the earth where each other word starts with the next letter. Yeah. It might just try it because I know GPT-4 can do a variation of this, but, um, oh, it's giving it a shot.

Okay. Let's see if it gets itright.

Okay. I don't have, uh,

I don't have the patience. Let's say

answer briefly. One sentence max. Okay. Great. And then

let's do, you know, uh, write a haiku. There we go. It's short where each other word starts with the next letter of the alphabet and ends with the previous letter of the alphabet. I don't know. You guys want to try this while this does it?

Is it around? Did I lose, did I lose something? Let's see.

Hi.

Okay. You there?

Cool. Um,

did I not copy it? Oh, no. Okay. Uh, give me a haiku.

Now make sure each word starts with

the letter of the alphabet starting with A and each word ends with the previous letter. But this is hard.

Okay. So it should be making the function call. Oh, I see. I see. I see. Okay. So I'm only printing the function call when it returns. So that's probably why we're waiting so long here. Um, but this is actually a really good, uh, example of, like, okay, we are doing, um, this, like, task delegation technically.

And, like, it is happening in the background. And I'm going to give this a, a second to figure this out. Um, but, like, this is obviously a bad experience,right? Like, you don't want to be waiting here. You essentially want to keep doing other stuff.

So wow, it's still not, still not back. Well, we'll, we'll let, we'll let it figure it out. Um, so maybe let's skip straight to async. Um, we're actually got, like, both more and less time than I thought. So I don't know if this keeps turning away.

Um, now let's think about this for a sec,right? If we want to do something async, it means that, like, what do we want to happen? There we go. So call the model. Yada, yada, yada. Buzz breezy whispered. Is this correct?

I don't know.

Wow. Yeah. This is sort of correct. What did it say? Note.

Great. Great. So it, it did something,right? Um, and it did something that 4.0 probably could not have, which is great. That's delegation, but we were just sitting there waiting for it. Um, so let's try doing this async. Now, I, I, I actually want everyone to, like, kind of just stop and think, like, how would you implement this, um, in terms of, like, what behavior do you want?

And maybe I want to see people, like, drop this in the Slack. Just, like, take a couple minutes, um, and, like, drop in the Slack, like, a proposal for how you would want to do this.

And then I might just pick one or we'll, like, talk about them.

Let's call o3. Yeah. Not, not yet. Not yet. Yes. Import async io definitely feels like, uh, important. But I guess the questions are like, you know, when we delegate something, uh, obviously we want to have this happening in the background.

Um, do we want, like, when it finishes, do we want to be, like, do we want it to be injected into the conversation? Do we want it to give us a response? Um, how many, like, do we want to be able to, like, interact with tasks that are running?

Um, like, do we want to be able to batch stuff? I am not setting up a Kafka cluster.

I'm going to give people maybe a couple more, couple more minutes to just, like, dump some ideas here. Um, batch calls.

Yeah. I guess batch, but how, how would batch calls work? Maybe Stephania, if you want to add some, like, detail there. Just keep working until it generates a stop word. Right. Okay. Yeah. This is, this is a good idea.

JS and set time out. Yeah. Switching to JS is always a good option because you already have the event loop implemented.

Smaller model, smaller model. Okay. You guys are all suggesting some good ideas. Um,

but we can actually go simpler,right? Like, like, essentially what we would want, and I can implement, like, the basic interface of this. Where's my code? Um, is like, instead of actually doing this,right, I can, like,

turn

delegated,right?

Response pending. Um,

and then later I can say, like, you know, check tasks or something.

So this, this pattern is actually one I quite like a lot where you, um, you call a model, it's a function, it's non-blocking, and then later you can, you can check up on them. Now, the thing is, Python is like single threaded and, uh, async is real.

Uh, it's just a little bit tricky. Um, but let's, uh, let's give it a shot. So I do have some async emergency reference. Now, here's the, here's the, um, thing to notice. When something, like, for this to work correctly, we have this loop that asks for our input and blocks on our input.

Um, and, like, is displaying that on the screen. And we don't want to have that be displayed while it's also, like, injecting messages. Um, so we essentially want to separate out, like, where you give user input and where actually interesting stuff happens.

So that's what I've done in this, in the basics of this async folder. So let's take a look at that.

Cool. Um, it looks complicated. It's not that really that bad. So, um, I'm using web sockets. Essentially what I'm doing, or sockets, sorry. Essentially what I'm doing is, um,

I have a server and all it does is, like, has a handle user input function, uh, and a start message processor. Um, these aren't super important. We can take a look at them in a second, but essentially it just, like, waits, uh, on a specific, like, uh, port and host.

Um, and then I have a client that connects to that and then just does while true loops and lets me enter in,right? So if we look at what that looks like, I have it here. Um,

so let's go to async. Let me zoom in here a little bit.

Also, um, while I do this, is there, are there any questions that I've, that I've missed? Um, and if there are, just feel free to shout them out.

No? Great. Cool. Covered everything. Everyone is, uh, perfectly up to date with everything you've said. Okay. Um,

sorry.py.

Okay. So I have this and I can say, like, hi.

Let me show you the, the rest of this. I guess that's probably important. Uh, so we have the server, we have the client. We also have the very, very, very basic agent implementation. It looks just like the old one with the main exception that I'm using async OpenAI instead.

Um, handling tool calls is happening in parallel. So for each tool call, grab it, create the task, um, await for them, and then just, like, await for them all at the same time. Um, and this just lets it, like, create a bunch of tasks that are all going to run in parallel.

And once they're done, like, if they, if they yield back at any point, we can keep doing other stuff. Um, by the way, if these are just, like, heavy functions that are doing heavy processing, it's not going to matter.

The fact that they're in parallel, like, in sync means that it's still going to happen one after the other. But if they're, like, network calls to other models, then it's perfect. Um, the run full turn is the same one.

You, like, call a model, check to see if there's any function calls. If not, break, return the response. Um, the only difference here is we're awaiting, uh, the chat completion,right? This is the only difference. And then if we take a look at our agent handler, we've declared, we've declared our agent like normal,right?

Right now it has no instructions. It looks very familiar. And this loop is also pretty familiar. It's like, get the messages. Um, here, here's the only difference,right? Um, and you'll see why this matters in a sec. I have a message queue.

Uh, and this will be useful because we don't want to process multiple messages in the same conversation at the same time. Like, we want work to happen at the same time, but we don't want, um, like, multiple generations to happen with the same history because then you'll get conflicting.

Like, like, if two messages or two functions return and they both need to be handled, uh, you essentially, while you want them to happen in parallel, the results should, should only come in one after the other. So we have a queue for that.

Uh, and it treats user messages as well like that. So this is the handle user input that is being called from the server. Essentially just throws in a user input message into the message queue. Uh, and all we're doing here is, like, pulling off a queue, run it, like, you know, uh, put, put the messages back in the, in the message array, uh, and then, like, sleep,right?

This is just a very simple, uh, loop, but it does it with async. So this is what we saw and, and it feels pretty normal,right? I can say, like, you know, my name is Ilan and I can say, you know, what is my name?

It's Ilan. Great. Um, now this still doesn't exactly answer how we're going to do things asynchronously, but it does give us the space to play with it because now things are happening async and we can do a few more delegations that, that are actually async.

So let's start with a, you know, simple blocking get weather function. I think I required all functions to be async here. Um,

67 and something.

Uh, and then skip that and let's test it out. The only annoying part is we have to, like, restart two things now. So hi. Uh, what's the weather?

Great. So we can see it called the function, got the response. So far everything's normal. Now let's do some delegation stuff. So let's say, let's say we want to call this function,

uh, three times for different places. So we have location. Um,

yeah. And let's say, you know, like, pick a random number from 50 to 80 to return. Um,

so if we do it in the previous case, do I still have the round? Yeah. Maybe, maybe not.

Okay. Uh, so if we look at, okay, this is the non-async one. Do we have weather? We don't have weather. Let's give it weather. Let's give it the same weather function.

Cool. Um,

we only need random. We don't need this. And it's not async.

Okay. So now we have this weather function. Let's test this out in the non-async case. Um, I'm going to get rid of the rest for now.

Um, so if I say, you know,

uh, weather in SF.

Cool. Calls it. Fantastic. And now it's like, weather in SF, New York, uh, you know, and five other cities.

It's, um, it's still going to do the parallel function calling. And here it's fine because they all return immediately,right? But now let's add, like, an artificial weight. So let's say

time.sleep1,right?

Um, cool. So now what's going to happen is we're going to ask for that again, but it's going to take so long. Weather in five random cities of your choosing.

So it's going to take a while. And the reason is each of these is having to run, um, in, like, one after the other. There we go. It took, like, over five seconds. Now let's do the same thing here.

So the equivalent is, um, there's async io sleep.

Someone let me know if I'm doing this wrong, but, um, this should emulate, like, very, very similar behavior. So if we instead run it here,

that, that I can say, you know, give me the weather

of five random cities of your choosing. Um, we should see it return. Okay. So it called all of them and it got back

because they all happen in parallel. Um, this is the magic of async IO,right? Um, anything that can be parallelized or I guess scheduled in a way where, where it's non-blocking, uh, like sleeps, these sleeps are non-blocking. Um, essentially we can, we can do it.

And so you can imagine switching this sleep for, like, an actual API call. Um, so, uh, we can actually do thatright now with o1,right? Like, we could call o1 multiple times and they would all run in parallel. Now, the main problem is that we're still going to be waiting back, like, waiting to, to get all of them together,right?

Like, like the fact that it, we can run them in parallel means that, like, we can have five 10-second tasks running in parallel. So it'll take 10 seconds, but it's still going to take 10 seconds where I can't talk to the model.

So instead, let's have this notion of tasks. There's many ways to do this. Um, I'm going to do a pattern that I quite like. So let's do this. Um,

let's define a, you know, create task.

So I have a create task function that makes a task. Um, no. And I want to create, like, a task ID.

And I want to keep it short.

Okay.

So now what I've done is I have this function that makes a random task ID, sets it, creates it, and then calls, like, let's say, get weather or something. Um, and then it's, you know, it's suggesting this check tasks, which is the next thing that I want to do.

So the next one is check tasks. And so maybe you can just say, like, check task.

You know, do forward. Amazing. Um, okay. So let's take a moment to, to look at what just happened. We now create a task with a random ID. We give back the ID to the model. And then later we can call check task, get that task, see what the status is, and see if it's done.

Um, so let's, like, add, I don't know, five-second delay here.

Andright now it's just get weather. So the description would be maybe create task for, you know, let's see. Again, this is all, like, sort of live coding. So we'll see if it works. But I can say, like, you know, create a task,

um, and set the description to the, to be

San Francisco or SF. Let's see what happens.

Oh, I didn't give it these functions.

There we go.

Um, let me just check. Cool, cool, cool, cool. Okay. Let's do it again.

Say, you know, create a task for the location where the description is just. So now it creates the task and I can say hi. And look at that. It, that's not working correctly. Maybe it was. Maybe it just took a while to respond.

Um, here, let's add a longer delay to know for sure. 10 seconds. Let's run it back.

So create a task with description SF. Uh, cool. So I can say hello there. Cool. So it can still respond to me,right? Um, check the task. So I can still interact with it and it can check to see if it's not done.

Um, I guess we need to keep checking to see if it'll work. So, you know, how about, or, you know, tell me a joke.

Who would've thought? Who would've thought? Um, cool. So now, like, check them again.

So it called the task. It is done. And we have 78 and sunny. I will take a clap for that. I want to see everyone clap. Please clap. Thank you. Thank you. Thank you. Thank you. So what you just saw was, uh, live asynchronous, uh, programming.

Uh, it's very impressive. The models can also, you know, do pretty well. Anyway, here's why this is interesting. We now have a system where we can give it tasks and it'll queue them up. Uh, and then we can check on their progress.

So, so already we have the basis for, like, a really, really interesting thing,right? Like this, this thing here,right,right now is just get weather. But if we say just like, you know, uh, run

what do they call it? Like, um, you know, call model. And then, no, but we want

OpenAI async OpenAI get the client. Um, I guess we could actually, we can just leverage the agents. They're already async. Let's not deal with that again. Um, or whatever. Let's do, let's do it this way. They did all.

They did all. Fantastic. Okay. So now we can switch get weather for call model. And let's try this again. So let's give it the same task as before, which is, you know, hi. Just make sure it's running. Um, you know, write me a, uh, haiku about, I don't know, uh, a coup where the theme is so incredible it makes me cry.

Make this a task. Okay. So it's created the task.

And now I can actually say, you know, now make a second one, but pick a theme yourself. So now I have this interface where I can essentially keep chatting with this and it can essentially spin off additional tasks.

Um, you know, check, check all tasks.

Um, let's see what the progress is like.

Look at that. So we have, we have both. Oh, was this a one? That's pretty fast. Um, but yeah, so now we have this system that you can actually just call, call multiple ones. So I'm going to pause here.

Just open it up for questions for a little bit. Conversation, like other directions we can take this, other ways we could have done this. Um, but what I want to look at next is, like, how can we do this in a way where we don't actually have to check?

Um, becauseright now, like, we don't need these two terminals. This is just overcomplicated. Like, I think the, the, the point of having the second terminal is so that we can push things to the conversation without a user having sent a message first.

So I'm going to pause here. I'm going to field some questions and then we can get back to it.

Okay. Um, would you not use generators for nested tool calls?

Um, Tom, do you want to, I don't know, if there's a mic in the room or like, uh, just get, get it, get it to them?

Yeah, there is. Also, they can mute themselves.

Uh, who's here?

Uh, sorry. Soham

is the one that was, uh, answering. Is Soham in the room?

Yeah. Can you hear him?

No, I can't hear him.

Hello. Can you hear me in the Zoom?

Yeah, I can hear you now.

I can't hear you anymore. I don't know if you're saying a questionright now.

No, I was just wondering

if it's possible to treat agent objects as a generator in Python. So you can nest, like, you can yield from individual agents and then yield those tool calls. Does that make sense?

Yeah. Yeah, that makes sense. This is also very impressive that you have to do this. I think someone needs to mute themselves. I don't know if it's Alon or, uh, like someone in the room unmuted and, uh, put, put this man through, through hell.

No, no, it's me. It's me.

Oh, is your laptop unmuted?

I can't hear anyone anymore.

Maybe just speak into your laptop.

Okay. But I'll, I'll answer your original question of like, you know, if you put, um, can you like essentially make agents into generators so that you can like yield the results as you go? The answer is yes. And this is actually like, um, theright way to do this.

The way I'm doing it now is I'm only exposing the final response mostly because, um, implementing like the, the generator gets a little bit tricky and I don't want to have to like debug that. Not too much, but you can essentially just surface each of the steps, each of the function calls, each of the everything.

Um, that would, um, that would let you

essentially keep track of more agents at the same time. And you could maybe have like a bit of a flat structure where you just have multiple agents going yielding events and then you can essentially see like which one's coming from where.

Um, but I guess the thing is like there you have to deal with a bit of the complexity of, um, handling all the like, like essentially listening to like all the different events and like, uh, associating them back to like a specific agent and like maybe there is an agent in the front, maybe there isn't.

Um, here the idea is just I have one agent that I'm interacting with. I'm always just going to interact with like that one agent and it's responsible for spinning off other, other ones. Uh, uh, uh, and like dealing with, dealing with that as well.

But you can absolutely do it the way that you're saying. And like if you wanted to build this out like more fully, you probably would in order to be able to surface progress.

Um, did that answer your question? I'm now scared to get people's talking in the room, but I'll just read them from Slack. Um,

is there any, is there any good design patterns to create projects with agents? There's like a million. Um, you know, like I like prototyping with Swarm. Um, I know Pydantic AI is also like a really nice one. I think Sam's there in the room or around.

Um,

there's, yeah, I don't want to like, I don't really use, uh, any myself mostly because like you saw, it's pretty simple to implement your own loop. So that's what I end up doing. I think usually for every, every project, I either write my own like, it's like, I don't know how many, how many lines is this?

Like,

like 70 lines. And so like depending what you want, you might want more, you might want less. And I'm sure there's good solutions. Um, but like I just have, like I just copy this around. Um, I don't really like working with too many dependencies, especially for something that's so lightweight.

Like I, I want granular control. Um, and like, except for example for Swarm, I ended up having to hop in here and like handling specific kinds of tool calls in certain ways so that I can do handoffs, um, which we can actually look at and we can implement this without too much trouble.

Um, but yeah, my answer is like there's many you can choose from. I don't have many I would recommend personally. Um, but I, I am a, I am a fan of, of Pydantic AI. It's pretty cool. I haven't used it too much, but like the interface looks nice.

Um, it reminds me a lot of Swarm. The one I use the most for prototyping is Swarm. I think it's just 'cause it's the one that like I'm most familiar with. Um, when you have to start up to approach dozens or hundreds of functions, what techniques should we apply in order to effectively tool call?

There's, there's a few answers there,right? Um, you can have multiple agents and essentially like split up the responsibilities or like the, the groupings of the functions. Um, and so into, into, yeah, like clusters where you have like a set of related functions that are needed for specific tasks and then you can invoke the correct agent.

Uh, and this is where like multi-agent patterns start to make sense is like specifically when you have tons and tons of functions, like how do you, uh, go to theright ones? If you for some reason need them all at the same time, uh, you could try fine-tuning.

Um, in, in projects for OpenAI, I've ended up fine-tuning up to like hundreds of functions. Like it was like 120 functions. This was with, uh, GPT-3.5. So the fact that that works gives me like pretty high confidence. Like you can fine-tune, uh, smaller models with a lot of functions and get them to work pretty well.

Um, the last one is like some kind of dynamic function loading where based on the input or based on the conversation, you load into memory or like you load into context, um, the most likely relevant functions. Um, and there's a few different ways to do this.

You can do this with embeddings. You can do this with like, um,

having like a two-step function call. At that point, you're essentially having agents. Like if, if you call a function to then load more functions, that's what a handoff is essentially. So a lot of these start to look very similar.

Um, it's just like how are you loading multiple different ones? Um, okay. For reasoning models, are there tools being called within the thought text? Um, so

because we don't expose the thoughts, like the, the, the chain of thought, um, that's a little bit hard to answer. For, for, for o1right now in the API, the answer is no. Um,

I'm trying to see how much I can say here. Um, it is something that is technically possible,right? Like you can do anything you want with, with post-training. Um, we do not currently allow you to call functions within the chain of thought.

Um, so yeah,right now the, the function calls happen at the very end. Uh, do you have any good code examples for router patterns in these lots of functions cases? Yeah. So, so my, like my answer there is some one of these.

Like if you really just want to route, um, then like the, the idea of having like multiple agents and handing off to one of them is actually really nice. Um, like you can just define multiple agents, each of them with like multiple functions.

Um, and then have the first one have like a, we can actually do this quickly. Like we, I'll do this in Swarm, but like I said, you can implement this yourself. Um, I don't actually know who else supports handoffs the same way Swarm does, but, um, essentially here, let's start a new file like, uh,

routing. So,

no, let's see. Swarm repo demo loop. I can say like, you know, triage

equals one. And then I can have my other two. It's like, you know, maybe I have like some collection of like, um, you can call them an agent, but I can also just call them, yeah, like, uh, what would it be?

Like, you know, uh, email functions agent

and you can have like, you know, like send email,

check email. I don't know what else does Cursor want me to write? Knock at weather. Um, cool. We have these two and then we can do like maybe we have the emails and then we have calendar. I don't know.

So make like create event. Can do like calendar. I'm just going to say like, you know, finish what I'm doing. Update the prompts and tools because I'm lazy. So let's see what, uh, let's see what Cursor thinks. Great.

So now we have an email functions agent and a calendar agent. Call them whatever you want. Um, now like let's pretend that instead of just having three functions, we have 30 and each one of these has 10 or 15.

Um, then like if you maybe give all 30 to one agent, first of all, try it. Like if that works, amazing,right? You don't have to deal with the extra complexity. Um, so you don't really want to do handoffs and stuff until you really, really need to through like evals.

Um, but, um, yeah. So, so, and then the special like handoff functions here, it's like, you know, uh, transfer to

email agent

and I'll return email agent and then cool. So now I have my two transfer functions. Oh, oh, did it. Okay, cool. So I think this should just work,right? So I've defined like the actual functions and agents that in your case would be many, many more functions.

Um, I've defined the transfer functions and I've given them to the triage agent. So now if I run this,

say hi. Um, you know, I want to send an email.

Cool. So now I'm talking to the email agent. Now, if you want to do like more, you know, like transfers,

uh, oh, this is not delegation. Okay. So trail assistance. Um, maybe we can tell these assistants like,

you know, if you already know what the user is asking, just call that function,right? And this is if you want to have like a case where, you know, it still routes you, but it's like a faster to hop.

So maybe I can say like, what are the functions here? Um, what are the parameters? Send email to subject body.

So I can say like, you know, send an email to bob@gmail.com about taxes.

What was the other one? Body. Um, saying, yo, do your taxes. So it should transfer me to the email one. And then there we go. It immediately sends the email. So this felt like an immediate function call, but there was a transfer in the way.

So this is kind of an example of like triaging does work. It's really convenient to model it with agents and handoffs. Um, I'd say the primary use case for agents and handoffs is just the glorified triage through, uh, multiple functions.

Um, yeah, let's see. Let's go back to questions.

Uh-huh. Consider having something like Streamlit Radio until you—are you considering having something like Streamlit Radio Util that allows porting all interaction functionality with one command to up? Uh, I don't think so. I don't know. Uh, what a people answer.

There we go. Sam's in with, uh, Pydantic AI. So check that out. Um, how many tool calls can you get in one iteration? You like parallel function calls? I don't think we have hard limits on either the number of functions or the number of parallel function calls.

How big a tool library will the models perform well with?

Super, super general rule of thumb is like 10 to 20 you shouldn't really pass over. Um, but like I said, I've gotten like this was a very specific case where like it was extremely latency sensitive. And so we had to have flat, like flat function calling and we did 120 functions with GPT-3.5.

So

you can go pretty far, um, with fine-tuning, but like I'd say reliably without, without very extensive prompting. Yeah, probably like 10 to 20. Like past that point, you really ask yourself like, what are you trying to do,right? Like why are you putting so many functions?

Is it super latency sensitive? Can you split it up? Like, yeah. Um, was there a follow-up here?

Okay. Um, rather than tool calls, I feel like we're moving to generated code agents. Will I soon be able to supply my tools functions? Ah, okay. That's a good idea. We should try that. Um, so essentially have something write its own function and then use it.

Uh, yeah, I feel like we can probably—how would we do that? Yeah, we can try that. We can, we can find a way to do that. Uh, I'll, I'll do that. Actually, let's doright now. Why not? Let's doright now.

Uh, I've never done this before, but I feel like it shouldn't be too hard. So, um, okay. Uh, you know, what do we call this? Like Bootstraps. Okay. So I'll keep using Swarm because I, I will use, um, handoffs from Swarm import agent.

Cool. So now it's always weather. So let's see. We want an agent that writes its own functions. So agent, was this one? Um, we want it to write its own functions. So maybe we can have a handoff to itself.

So we can do, you know, like refresh, you know, what would it be? Refresh functions. So we can actually return the same agent. Um, and then, okay, is it not declared? Is it unhappy? Uh, yeah, it's not defined yet.

Okay. So now we can define it down here.

Cool. Functions. Okay. Um, Bootstrap, Bootstraps, Bootstraps. Okay. So we have this and we want it to write its own functions. So how, how are we going to do this? We want it to produce Python. Does anyone have any ideas?

Uh, if you want to shout them out, like make just pass around the mic while I code. Just feel free to—I can't, I can't read while you're, uh, what, what you write, but we can, we can try this.

So, um, let's say we want it to, you know, um,

you know, add tool. There we go. And then let's call this, you know, Python mutation.

Okay. I'm going to do something very unsafe. So, you know, what would it be? It's like function obj equals

eval of Python. Don't do this. Don't do this, kids. Um, and then will this work? Is this how Python works? Sam's there. Uh, okay. So if I have this and I eval like, you know, def

a,

does anyone—um, okay, you know what?

Um, so I guess it's like, uh, I want to write a, write a function that takes a string presenting an implementation of a Python function and returns the actual Python function as interpreted.

I don't know. Let's see what it does. Um, so essentially we want to do that once we have the function, then we can just append it to the agent and then we might need to reload it. Uh, so we can then just return the agent.

And this might be it.

This might be it. So

add tool

exec, not eval. Why didn't anyone say that? You guys can shout out.

Okay.

Okay. So it's exec. And then

what is this? A,

I don't know.

Someone save me. Like,

uh, hey, there's a couple implementations in this lock. Sorry. Check the slash.

Did someone do this?

A, okay.

It's almost like a copilot. No, no, no. But this is not—okay. No, no. But this is not what I want,right? Like, I don't want it to run it in a subprocess. I want it to—I want it to evaluate, like evaluate the function and then turn it into a—is that—oh, go up.

Oh, Sam, of course. Amazing. Uh,

Jesus Christ.

Okay. Um, make it

so the add tool function, uh, evaluates the implementation and adds it to the tools. Similar to now, here's a great reference for how to. Let's—and then we want to grab—was this a good idea? This is a terrible idea.

I hope people are enjoying this. Um,

okay, let's see. Um, and let me just read this 'cause it might not be that complicated to add in. So

I can't—okay, it got in the way. No, not Zoom. Where's Slack?

Was this it?

Is this it?

Okay. I'm going to—I'm going to put this on hold for now. It sort of ignored your implementation, Sam. I'm sorry. Um, let's see. So if you have parse function, can I just

do this? Does this work? Uh, okay. Let's see. I've—this is first time for everything. Uh, wrong one. Sub Bootstraps. Hi. Um, add a tool that prints hello when called.

Call it.

Oh, but no, no, no, no. Make it print it, not just return it.

My heart's pounding. Okay. Um, so say hello tool. Call it. Look at that. Look at that. Okay. So we now we have—yes, I will take a clap for that. We did this together, guys. Um, so there's actually a lot less code than I was expecting, but now we have a system.

Look at that. It's tiny. Um, that can write its own tools. Um, and so like maybe we can do something like, you know, um, what is, you know, like make yourself a little calculator.

I can't believe this just worked. Uh, you know, what is 2 * 3 *

can someone check this? Uh, wait, wait, wait. 3, 4, 6, 9, 8. 3, 4, 6, 9, 8. This is—this is crazy. This is—this is so fun. Uh, yeah, I hadn't done this before. And this is a lot less code than I thought.

Look at this. Look, this is all you need. I mean, it's—this is super dangerous code. Like this, this is not good. Don't do this, but it's fun. Uh, I would put this squarely in fun things. We can—we can now transition to looking at other fun things.

Um, they're definitely not as fun as this one, I think. Um, they're just a couple of like random things that I found related to real-time since I did sort of put it in this, uh, title. So whatever. Okay.

These are the two main tricks that I—that I kind of thought were pretty cool, and you might have already seen them. Um, one is if you've ever dealt with like the real-time API, um, like jumping in, uh, before you're done with an idea or like done talking or something.

Um, like what I was thinking is like if really what you want is you want it to like use the model's own intelligence to decide if you're done talking or not. Um, and you can treat our VAD, like our voice detection, as like a trigger happy version of that.

It's like a—it'll always tell you when maybe the user is ready to stop talking, but if you want the model to check, you can have a stay silent function or something else that essentially handles the other side of that where it's like, okay, we're definitely going to let you know whenever the user might be done talking, but then you can actually verify with a function call.

And so this implementation is like super, super simple. You literally just give—you give the—the function itself to real-time API and tell it to call it when the user is not quite done talking. And I'm not going to try to demo this live.

Like real-time demos are tough. Audio is tough. All that fun stuff's tough. But, um, it works like surprisingly well. Uh, if you just describe—I, I can—you can probably find this tweet, but it has the full prompt. But essentially it's like you can—you can like pause.

You can say like, you know, I've been thinking about and then like stop and like it'll get triggered. Call stay silent and then you can keep talking. Um, so pretty, pretty cool, uh, useful thing. Um, and then the other one is also related to real-time API.

And if again, if you—if you, uh, are following me, you might have already seen this, but, um, someone at DevDay just like came up to me and asked me like, "Hey, um, is there a way to like make the real-time API talk in like a specific way?"

Uh, and I was like, "I don't know. Let's try it." And so like we had a demo booth and I sort of pushed someone out of the demo booth and grabbed the laptop and I was like, "Let's just like try—try this stuff."

And apparently if you ask the model to just—you give it like a—a script and you're just like, "Hey, read this out according to these XML tags." Um, you can have it follow them. Here, let me—let me find, um, the original—the original one.

Yada, yada, yada.

Sorry, this is—I'm not trying to show you the whole timeline. Where is it? Okay, this is the stay silent one. Um,

where is it? Where is it? There we go. Um,

uh, it's impossible you'll hear this,right? You guys don't hear this.

Yes. No. Uh, we don't hear it, but you can share your computer audio. Can I? Zoom thing. I'm—I'm—I'm actually—I'm not going to try. It's fine. Um, wait. Can I?

Same. No, no, no. It's fine. Whatever. Anyway, um, yeah. So if you—if you—if you like give it like a script like this, um, we didn't actually train it for this,right? Like this is just a really nice consequence of like behavior.

It's technically not function calling, I'm realizing now, but it's like it's very function calling-esque. It's real-time. The title of this talk does include real-time. So this is, uh, this is it. Um, I will pause here. I have so much more I can go through, but I want to give you guys a chance to like ask questions, poke around with some of these ideas.

Uh, Swarm is public. You can just try it out. Um, we can also just try creating other functions. Uh, this—this is one of the—this is easily one of the coolest like little programs I've ever made. Um, so

yeah, let—let me see if there's any other questions in the Slack.

Uh,

it's a room full of people autocompleting. Make. Yeah, yeah, yeah. Okay, let's not do that.

Uh, okay. Revisiting memory. Let's see. Do you have any suggestions for trying to enforce consistency of stored memories? Identifying inconsistencies and figuring how to resolve them. Okay. And then the second part is what data structures do you suggest for more structured memory, helping enable more meaningful comparison of objects?

I think, I mean, this question opens like you start to go down a path where you can get as complex as you would like,right? Like—like you can start simple and you can end up with like an entire operating system to manage memory,right?

There's like a whole range. Um, one way to do, off the top of my head, like one way I can think to do this is, um, when you are about to store memory, do a retrieval, like do search to find similar memories or like memories that like are semantically similar.

Um, and then do an explicit check, uh, with a model to see if like, you know, one is like updating or contradicting another,right? An example of this is like, you know, what is the latest state of a project,right?

You know, if someone's like, is the project like at some point you're saying like, "I'm working on the project, like it's not ready yet," and it saves that. And then later you're like, you know, I, um, how like the project's done now,right?

Like those are two contradicting memories. Um, what you can do is essentially, uh, have a timestamp for them, but also when you are about to store the second one or any memory, check for similar ones and create like a direct, like essentially, um,

like node, uh, pointing from the original memory to the new one. And so I guess—I guess what I'm thinking is like that way if anybody asks

and you surface both, like you can keep both in memory and when you do like retrieval and semantic similarity, you can surface both. But essentially you can present like the whole chain of updates. And so you can just present the last one if you want, or you can present the whole chain and the model has some idea of like, you know, maybe if you ask it like, um, you know, how long is this project delayed because that was the last you heard about it, but somebody at some point said, "Oh, it's actually done."

Um, like if it raises both and has one with a later date, you know, maybe. But if you like if you've already made this chain explicitly, um, like you can actually represent that and you can choose to not show the previous ones.

So that's one idea. Like I said, there's so many you can like so many ways to do this. Uh, and—and like as soon as I stop talking, you guys walk out, someone's going to like be like, "Oh, there's actually this like much easier way that like this idiot didn't think of."

So anything you think of probably works. Um,

here's a real-time API prompt. Yeah, cool. Uh, any other—any other questions about anything? Um, it's been like an hour and 40 minutes, so happy to like keep going with random content. Um, there's a couple demos that I have that are like demoing some public repos that we have that you can use.

Um,

yeah, but I want to get—I want to give anyone a chance to like speak up, say something if you want, ask questions. Um,

I—I don't see—I don't see a lot happening in like I can't—I can't—you guys are tiny on my screen.

Uh, cool. Okay, then I'll pose this to you. Sorry. What? Oh, no. Go ahead. It sounds like you're wrapping up.

Yeah. The last thing we can do, I guess, is like either I can, um, try and pull up a real-time, uh, demo that shows how to do the o1 stuff, uh, in a—in a slightly easier way. Okay, cool.

People want to do that demo. Okay, let's do it. So, um, the reason I'm a little iffy about it is because, yeah, it's real-time, uh, and those demos are tough to do publicly. But, um, there is this repo, OpenAI, um, real-time Twilio demo,

um, that essentially sets up your whole, um, like phone calling assistant. Uh, and it just worksright out of the box. It walks you through the steps. Um, I may end up sharing a phone publicly. Don't call it during this demo and don't share it, and I will delete it.

But, um, yeah, I trust you guys. So let's see.

Okay, so it looks like this is already running.

Oh, I do have it already running. Okay. Um, oh, never mind. I—I did, uh, I did make it all dots. But yeah, essentially this, um, this little checklist is like live updates. So as you set up the account, as you set up the phone number, as you set up your local web server, etc., it'll like get checked.

So setting up Twilio and Grok and everything has been one of the most annoying things I've had to do multiple times, um, related to the real-time API. So this hopefully makes that process like a lot—a lot easier. You can do most of it directly from here, and it lets you know like when things are done.

So you don't actually have to do a lot from the Twilio console at all. Um, anyway, so, um,

I—let me show you the backend. So there is a handler function. There we go. Function handlers. So, um, you can either implement tools, uh, locally as a JSON schema, and we can—we can try that really quick. So this is the real-time playground, and I can say like, you know, make a function to, you know, get the answer to answers to everything.

No params. Maybe one param.

Okay. So it gave—it gave us this nice little, uh, nice little function. Then we can just paste it in here.

Save it. And cool. So now we have this get universe answers, and I can save this configuration and call the phone number. So let's see.

Hello.

Hello.

Okay, this is why I love them. Let me—let me try once more. Okay, let's go tool paste this. Save changes. Save config. Um,

hello. Hello.

I don't think it's happy.

Is someone else? No, I didn't show the number. Hello. Okay, it's not working. That's fine. Sort of expected it. Um, essentially what you can do like when you—when you use this, um, it's cool 'cause these tools you can either specify local ones with schema and they appear here, or you can define them in the backend.

Um, so if you define them in the backend, you can actually give like code to execute. Um, if not, they'll just show up here. Uh, and you can enter in like what the response will be, uh, like a mock response.

Um, but you can also like set up backend functions that will actually get handled in the backend. Uh, the sample one is like a real implemented get weather one, but as you can see, there's like it's pretty straightforward to implement like the o1 one.

Now, the main difference between the real-time API here and like what we were doing before is the real-time API does actually allow asynchronous functions natively. So you can—the model can call a function, um, get no response, and you can keep talking with the model, um, until a response is back.

So that—that is behavior that like we specifically had to teach the real-time models because unlike a chat conversation, you can't really enforce, um, like you can't really halt the whole conversation until the function response comes back. So we had to do that for—for launch, um, because originally when we hadn't done that, you know, we hadn't ever shown it how to do asynchronous functions, and so it just couldn't—couldn't handle them correctly.

Anyway, sadly, uh, demo didn't quite work. Maybe I can try this one more time. We'll see. Third time's a charm.

Hello.

Hello. No. Okay, great. Anyway, um,

I believe that is most of what I wanted to cover today. Um, I'm going to play with this hyper-unsafe function making agent a bit, but, um, yeah, thank you all for coming. I'll hang around for questions for a bit if anyone has any.

Otherwise, I think we're ending a little bit early. Um, like I said, yeah, most of you can—can hang out or—or leave as you please. Um, but I'll—maybe I'll call it wrapped up here. Um, maybe what we can do is if you're interested, you can stay and like hack on some of the stuff that you saw.

I'm happy to hang around, uh, and answer questions as well. So thank you all for coming. Uh, hope you—hope you got something out of this.

Um, Anoop is also here, uh, to help with sports. Uh, otherwise, Ilan, I guess you can share. I'm sharing a little bit in case people have questions. Mhm. I'm actually going to dump this, but this may be a really bad idea, but, um, here you go.

Here's the code for the super unsafe self-like tool writing agent.

And I'm probably going to stop sharing my screen. Um, yeah, I can—I can—I can share the repo and the slides. There's not a lot in the slides, but yeah, I'm happy to—happy to share them after.