BotDojo Launch: Enhancing AI Assistants with Evaluations and Synthetic Data

So, hello. My name is Paul Henry, I'm the Founder of BotDojo, and as a previous CTO I was working with teams deploying LLMs applications for hundreds of thousands of customers. And like many of you guys know, it's super easy to hook up a vector database with an LLM over the weekend, but really hard to get it production ready.

And so that's what we do. We are an AI enablement company, and we let companies deploy AI to prod. Live demo time. Allright, so today I'm going to show you a demo of our product. We're going to take synthetic data that we're going to generate, and we're going to combine it with evaluations to see how we can improve the performance of a chatbot.

Or at least that's what I hope happens. Allright. So I'm going to open up our template of our chatbot. And we have customers live that are using this template. It's kind of battle-tested. And so let's test it out.

How do I create a vector index in BotDojo?

OK, and as you can see, all the little nodes are lighting up as they execute. We're taking the question, we're looking at the chat history, we're going to the vector database to retrieve the information, and then we're answering it with an AI model.

So if I pull this up, you can kind of see in our low-code editor, this is the prompt that we're sending to the LLM. We're getting the results out here. And we also support JSON schema. So if the model supports JSON output, like Grok, Claude, and all that stuff, then we just conform to that.

One key thing is you can pull a trace of each node and see exactly what we sent to the LLM, what came from the retriever, the exact data, which has been super useful for debugging apps. Allright, and cool, we have an image.

It's got citations. We should ship it. That was supposed to be a joke, but allright.

So this is where evaluations come in. So I'm going to demonstrate the evaluations that I previously ran. So we have a feature in BotDojo called batches, which allow you to run a whole bunch of questions through your chatbot or your AI flow and run evaluations to kind of see how things are doing.

So if you can see this, we have five evaluations that we ran. There's a little bit of red. That's because we don't have enough information from our vector database. It also checks for things like hallucinations. So let's try to fix that.

And so I'm going to clone this batch. I'm going to rename it with generated data. I'm going to increase the throughput a little bit because of time. And I don't have enough time to generate all the data for this demo.

So the previous run was filtering out the generated data, and so I'm going to remove the filter that we're passing into the flow so it takes in the generated data. You can also change the model and all that kind of stuff to see how it performs.

Allright. So while that guy is running, I'm going to open up another flow. And so this is the actual flow that we generated that synthetic data. And so

let me run this one real quick.

And so this particular flow takes in multiple inputs. And so I'm going to paste in some JSON from a previous run. And what this is going to do is kind of a trick that's been working well for customers, is where you extract questions and answers from support tickets.

So these are live agents talking with customers, and you use this as test data to send it through your chatbot. And we take relevant information from the existing index and we have it write a document. And so it uses the same writing style, and then we do an inline evaluation to where we check to see if the document has enough information to answer the question.

And then we also have a code node here where a lot of times when you're using these low-code editors, there's situations where you have 40,000 different boxes. And so when you have to rewrite code, we support TypeScript and soon Python.

But you can see that, hey, we're getting the information and we're writing to the vector index. Allright. Running out of time. OK, let me go back to the support chatbot. A moment of truth. So I'm going to compare the batch that we ran before with the new stuff in 20 seconds.

Oh, shh. You do it, you do it 15 times, and it doesn't work.

10, 9. We're also hiring. So if you're an AI engineer, help us fix this. Allright, there it comes. OK, whoo. Allright, one second left. It's all green. So it improved the, you know, it measurably improved something. So thank you.

botdojo.com, check us out. Thanks.