A Piece of Pi: Embedding The OpenClaw Coding Agent In Your Product — Matthias Luebken, Tavon

Allright. I was introduced to Pi by, uh, looking into OpenClaw. There was a conference, a meetup, and I said, like, okay, we're doing OpenClaw. And I wasn't so much interested into, like, all the craziness things that people are doing, but I was more interested in understanding, uh, of how these things work.

So I was looking into Pi and, you know, uh, understand the, the whole world of what Pi is able to do. Um, this is the one picture you need to take. Please feel free to take more pictures, uh, but all the slides and the examples are there.

Uh, so that's the one slide. Allright. Very quick, uh, about myself. Uh, we're creating a small company, uh, Tavon AI. We're building agents for, uh, organizations, small, out of Europe, uh, but getting started. And, uh, what I really like, um, about oh, sorry.

Uh, what I really like about, um, uh, Mario's talk is this is quote, uh, you probably have seen, uh, this this morning. We are on the, uh, we are in the fuck around and find our own face for coding agents,right?

So everything that I'm gonna show you is what I know today,right? And, uh, I'm gonna do the talk again in a couple of weeks, and it's gonna be most likely be different. Uh, but, um, as, as Mario was showing this morning, um, he has created this minimal set,right, this, this coding agent that is available, um, uh, for, for you, uh, for you guys to, uh, to fool around with.

And that's what I'd like to encourage you. So coding agents and why is it so exciting for us to build more products? This is Ken Thompson, um, inventor of, uh, Unix. And this is the famous quote by him, uh, one of the quotes, "Write programs that do one thing and, uh, one thing well."

And, um, I really like that because that's, that's kind of like works, uh, to our advantage with agents. And, um, the best part where I show this is with Cowork. So this is Cowork, uh, Claude's desktop. Um, and they're basically bundling their coding agent into something where they feel is more applicable.

Um, and to be honest, I've seen very good receptions around this. And when you use it, uh, with, uh, financing tools, with their finance tools, you always need to work with Excel,right? So, uh, they have this Excel skill down there.

Um, and it talks to, uh, Excel,right? Well, it doesn't. Uh, instead, it uses a, a set of small tools, small CLIs, um, uh, Pandas, uh, OpenPyXL, uh, stuff from LibreOffice, and package this into their own skill, uh, to make it, uh, up and running.

And I think this is a great example to kind of get your going, get your thoughts going of what, what is doable.

Um, I haven't written a book, and nobody can write a book about this,right? Because there are no patterns,right? We need to figure this out. We're seeing some emerging patterns in the coding space,right? There's obviously tons of different coding agents, and we're seeing this, but there's no authoritative resource around this,right?

So get going. One thing, uh, when I was talking to Ivan yesterday, uh, we realized is, like, one architectural pattern that we're seeing is that make it easy for coding agents,right? Now, that is very broad, but think about it,right?

Like, like make not don't try to be, you know, very, um, complex and things, but think about the, the coding agent, uh, what is it good at, and how do I build my system so that the, um, agent is easy, make it accessible.

And I, I have some examples. Allright. This is the rough agenda, uh, for the next 10 minutes or so. Um, I'm not gonna talk too much about Pi in OpenClaw. Uh, I have a few slides. Slides are online, so we'll take it from there.

So again, very brief, uh, introduction of Pi. Um, uh, Mario, uh, uh, great work. Something he didn't mention is that he's joining Arundel, uh, which I think is awesome. Uh, it seems like, uh, great, great folks working together.

And, uh, yeah, it's open source, it's minimal, so it's, it's just perfect to get started. And the other part that I do want to re-empha-emphasize on is, is give it a try,right? We're gonna talk about a little bit different, but open up Pi and ask it to build what you want,right?

It's amazing of what it what it actually is able to do by the system prompt, uh, that Mario has shown. Allright. These are the extensions. Um, so again, uh, uh, all the extensions you can download, uh, uh, build yourself or download and, uh, yeah, ton-tons to explore.

Allright. So let's going. This talk is not about the coding agent itself, so using it for your daily dev works, but what can we potentially do with this? And the starting point are actually not coding agents,right? The starting point is, um, and I encourage you to do the, the same, is looking at the, uh, core agent itself.

And there's other SDKs, but, you know, we're, we're talking about Pi, so let's, let's, let's use Pi. And what is an agent? An agent is actually just an LLM agent that runs tools in a loop,right? So you have some goals, you have some context information, agents MD, uh, uh, in many cases, and then you do, do co-tool calls,right?

And you get some results, and, you know, you basically do in a do it in a loop,right? That's it,right? There's not, not much more. The rest is magic, trying to put it in your use case a little bit more, in the other use case a little in that direction.

So that's really it,right? So pretty please, uh, uh, don't, like, open the curtain, uh, play around with it. Now, with agents, um, uh, uh, agent core, this looks a little bit something like this. You have an agent class.

This is all TypeScript. Um, you can, uh, you know, ingest all, all sorts of information information. You can prompt it, um, uh, with different information. Uh, and, um, also, you know, you have an event system, so you know a lot of things that, that, that are going on.

So, um, small example. Uh, this is a CRM lead qualifier. I don't know. I've started the CRM use case, uh, for my personally, and it, it just sticks around. So, um, terminal interface, obviously, uh, small, uh, TypeScript application, three, uh, uh, uh, three files, really easy.

And you can see this,right? You have a couple of commands that you can execute and, you know, show me all leads and score them,right? So that's what we do. Uh, show all leads and score them. And here you see all these, you know, things that are going on under the hood,right?

You see that, that the assistant is calling, uh, tools, that you get some results, and eventually, you know, you get some input. Now, obviously, there's tons of things to do, but, you know, I've just vibe-coded this away, uh, uh, uh, and it's a good, again, good, uh, learning exercise.

The system prompt, um, uh, you know, um, as you could imagine,right, you know, calling out the different tools and what you do,right? So all pretty straightforward if you are, uh, building an agent. This is an example of how you inject here,right?

So, um, we said we want we, we do call tool calling,right? We reach out to this, uh, uh, uh, and call a specific tool. But for the agent, for, for steering it more,right, you know, a typical hook would be before the tool call, do something,right?

And in this case, we don't want to update a contact, uh, without, you know, checking something or I don't know. You can imagine any types of authoritative, uh, uh, role-based access, whatever enterprise feature in here, but basically, you know, uh, just before the tool call.

There's another one, events. So we've seen these, you know, uh, uh, the stream, and you might have seen a little check mark there, okay, the tool call was, was fine and re uh, returned some result. So again, we're subscribing to events.

All pretty straightforward. And again, please give it a try. Allright. So this is simple agents. Others a-agents SDK, uh, are, are available. Um, and now we're moving to the coding agent. Now, what's, what's a coding agent? At the end of the day, it's really the same thing as we've seen, uh, before.

It's a, you know, normal agent,right? It runs tools in the loop. But now we have a runtime and some type of shell,right? Bash is, it seems to be the, uh, the shell that, that everyone is using. But we have a shell and a runtime to, to start executing.

And now things are getting interesting. And now the, the, the magic of, of what you've seen with OpenClaw, uh, suddenly shines. Uh, um, Peter, uh, uh, shared this, this example, uh, on some presentation where, uh, he, uh, sent a message to his OpenClaw and, uh, sent a voice message.

Now, at that time, OpenClaw, um, and I still don't know if there's any, like, uh, special plugin, but at that time, OpenClaw didn't know anything about voice, about voice messages. So what, what it did is, it, it, uh, created and used different tools.

Uh, in the end, one of the tools was, uh, FFmpeg,right, on the local, local machine, and it started this. And this was one of the tools,right? So from the outside, it, it looks like learning, but in the inside, it's actually just another tool call that is available to the agent.

And that's why these things make it so interesting. So, um, a-again, uh, the example here, um, now, this is a little bit more sophisticated, but the, uh, important part and, and this is the Extension API, and, you know, please look it up online.

We're, we're gonna do two things or the, the things that I'm most in mostly interested is in, in session events and UI interaction. And yeah, uh, uh, look it up online. But here's, here's the, the actual extension. Now, again, this is what you would in a coding agent, you probably just generate by asking it.

But here, if, if we have a look, um, this is a CRM, uh, TypeScript, uh, a small snippet of it. And basically, what we're now doing is we're doing the same example as before,right? And we have a new command called pipeline,right?

So if you have the slash commands and you have a new, new command called pipeline. And now we are able to we're loading all the context. Um, and, uh, you see this little in, um, uh, don't have the lines.

Just below step one, uh, you can see, uh, context UI select,right? So all of a sudden, we're not only interacting with the backend systems and, and sessions and, and of those sorts, but we're also interacting with the UI, and we're able to select,right?

And that, that's got, got me thinking. Um, so,right? So you have this, th-this command. And again, this is now just the coding agent,right? We're not talking about the core agent class, but, but this is how you would load up Pi if you just don't download the, the coding agent.

And now, with this new extension, we have Pi,right? And we can start selecting things,right? So this is a simple, simple select here. Um, and, you know, you, you eas even have dropdowns. Now, the important part here is these are extensions and the framework, uh, that currently Pi, um, has included is catered towards the use cases of a coding agent,right?

So we, you know, there's lots of work and other things to do to make this ready for others, for other types of applications. But I hope you can see and understand the vision where, where this is heading. And, um, yeah, you know, this is all terminal,right?

So you wonder, how would this look like in the web? Um, it currently is not possible if you ask Pi to build something. So I ask Pi to build something,right? And this is the web UI it would be a web UI, same command, same selection, all based on the same extension mechanism.

Now, um, there's a refactoring going on to make this, uh, better accessible and make it more clean. But I hope, again, it shows you a little bit of, of where the where the things are going. Allright. Now, um, Pi in OpenClaw, um, is, um, is a special, special setup,right?

So Pi in OpenClaw, what we have there, um, is that, that now we're not only talking about like, like, um, a, a single agent and a single session in a coding environment, uh, but now we have a multi-channel, uh, environment where, uh, we have, um, you know, multiple threads going on, multiple agents going on.

So there's a little bit more to it. Um, this is, um and, and the interesting part,right, that's, that's where, where I got started is, is, like, if you look into, um, you know, the, the, the packages, um, uh, the core packages of, of, of Pi, all of them are used in OpenClaw,right?

So OpenClaw has this, uh, uh, this function, run embed, uh, uh, Pi agent, and it creates a session,right? So sessions, um, uh, Pi itself has a great session support. Um, it creates a session agent and streams all the information back.

We have, um, the coding agent, which we just talked about. We have agent core as, um, uh, the other part that we talked about. And there's two other, uh, minor, uh, or major packages, uh, Pi AI for the unified LLM abstraction and, uh, a terminal UI interface.

Um, there's, um, uh, OpenClaw has built its own plugin mechanism, and that's because, um, uh, you know, it's a different use case,right, and has different requirements. So you have, uh, plugin support for multi-channel routing, different or, uh, uh, um, provider orchestration, sub-agents, gateway support, yada, yada, yada.

All the things that you know by OpenClaw, but it's based around the core mechanics of, of Pi and, and, and leverages it. Cool. But, uh, one thing, and that's, that's, that's the, like, the, the major gist I would like to bring across is, like, okay, what do we do now with this?

What are our o-other options for us to do? And this is one of the applications we've, uh, been building, um, for, um, for a client. Um, and basically, um, uh, uh, the, the, the use case is a sales process.

Um, they get, um, uh, requests for proposals, um, of, of an o-ordering another, um, another system,right? Um, parts, parts being sold by that company. And we're taking all this coding agent, all, all of that we're taking away,right? We're, we're, we're new, fresh, new thinking,right?

And look at the process from the get-go. So, um, an email comes in,right? We, we, we monitor basically that inbox. Then we have some gateway because what we wanna do is we wanna forward this to different agents,right? So here, I have, um, multiple agents,right?

Uh, the way it's structured is we have one agent per customer. And that agent has a general harness,right? Agent MD, uh, um, agents MD as an example, but you can obviously also use different ones. And that helps, um, understanding the role of that agent.

In this specific case, it, it tells, uh, of how to use the system and how to react to certain, you know, inputs, outputs, etc. Now, um, the other one is customer MD, where we basically explain the agent. Like, you know, the specific customer might have, you know, specific twerks,right?

Specific, um, uh, access, specific, um, um, discounts, and all of that sorts. And then,right, and that's what I said, like, earlier, I, I like using sessions. Then for each case,right, we're, we're, uh, creating and reusing existing sessions so we can back and forth, um, um, know what, what was previously talked about.

Allright. So email comes in. We're looking at the gate bo um, inbox, and we route this to these different agents. And now we have tools,right? So we have these different tools, uh, to talk to the CRM, to talk to the ERP, um, and get theright information out of the system for this agent to look on, like, like behave.

Like, you know, maybe it has, you know, new contact information or, or that sorts. And again, we make this available. We make it easy for the agents to access,right? And our way currently is doing this with CLIs,right? So CLIs, our agents are really good at using CLIs.

So we make it available as a CLI. We put we make sure that the data is secure. Uh, we have our own sandbox. And then we're creating the drafts again,right? So that's the system. And I hope by this point, you basically understand, like, logically, where these things, uh, uh, uh, fit together, but how would this look like?

Um, oh, one, uh, uh, final thing,right? There's always the question around, okay, sandboxing, etc. And, and, uh, to be honest, we're on the, uh, just on the on the steps of, of getting there. But if you've seen, um, NVIDIA's announcement, uh, around, um, OpenClaw, their policy, their open shell is really, really interesting.

And, um, um, it's, it's, it's a way of it's one ways of securing an, um, an agent. We're looking into this. Please do as well. Allright. So how does this look like, um, to, to, to kind of, like, get you an understanding of, of how these things,right?

So here's the dashboard. Um, rather, uh, boring, but here's the in uh, the email, the inbox,right? So again, we see the, the email coming in. And yeah, we, um it's one of one of the many emails. Most of them are ignored, but this one is, like, the, the, the LLM call said, "Okay, I'm, I'm interested in this."

And it is associated to a case,right? We see the case, uh, up there. Now, this case is, again, is an agent session,right? Uh, so we find the session and associate it to it. Um, we then create a draft.

Uh, so there's tons of calls, which I'm gonna show you in a second. But basically, the output of all that is a draft email that the user will be able to use,right? So our thinking is, uh, let them users stay in, in email, let them stay in the, the inbox and drafts, and they don't even, you know, need to do a lot.

So this is more like an admin interface. They can stay in email. But basically, the output would be a, a draft generated. And how does it look, uh, uh, behind,right? We, we had the, the different sessions before, uh, uh, the threads.

And this is the same thing,right? The assistant says, uh, well, uh, apologies, this is German, but, uh, uh, now I'm looking at the articles. It does different tool calls,right? It gets, gets results and does this in a loop to result,right?

The end effect for, for the user is, I'm looking at my inbox, there's a new email, it's associated to a case, and I get a new draft, which they can freely edit. But, um, under the hood, we have all these, um, uh, uh, agents working.

Allright. That's, that it's it's for me. Um, again, um, here, here you find the slides. Um, key takeaways, please. Coding agents are and will be a core building block, uh, for your software systems. I'm, I'm betting on it.

A lot of people are betting on it. So please give it a try. Pi is perfect for tinkering, whether you like it or not. It's minimal. You can rip things apart and put things together. It's perfect. So please go tinker.

Allright. Thank you.