# Shipping complex AI applications — Braintrust & Trainline

AI Engineer · 2026-05-01

<https://aie.addtry.com/c70e51fe-e591-40bb-ad86-38625d21b65a>

Giran Moodley of Braintrust, joined by Trainline's Oussama Hafferssas and Mayank Soni, demonstrate how to ship production-grade multi-step AI agents by combining rigorous tracing, evaluation with golden datasets, and automated online scoring. They walk through building a support triage agent that progresses from a single prompt to a five-stage tool-calling pipeline, where tracing captures latency, tokens, and costs per step for deep debugging. Trainline shares using Braintrust to run offline evaluations before switching LLM models and to enable cross-functional self-service. The workshop covers identifying failure modes via production logs, tightening prompts, and re-evaluating to complete the feedback loop, emphasizing that observability and iterative evaluation are essential for moving from prototype to reliable production systems.

## Questions this episode answers

### How did Trainline use Braintrust to switch LLM models without degrading performance?

Trainline's Mayank Soni explained that high costs forced frequent model switches. Before Braintrust, they lacked systematic evaluation. With Braintrust, they ran offline evaluations to simulate how cheaper models would perform, then used online evaluation to confirm the intended effect matched expectations, ensuring quality while reducing costs.

[20:55](https://aie.addtry.com/c70e51fe-e591-40bb-ad86-38625d21b65a?t=1255000)

### What is the “flywheel” approach for shipping production AI applications?

Braintrust's Giran Moodley described the flywheel as: start with an evaluation dataset (even for new apps), instrument to get production traces, identify failure modes from traces, remediate issues, ship the fix, then monitor and loop. He emphasized that this iterative process helps teams systematically improve AI systems despite non-determinism, akin to agile development.

[12:22](https://aie.addtry.com/c70e51fe-e591-40bb-ad86-38625d21b65a?t=742000)

### How did the workshop identify and remediate a failure mode where a “not urgent” support ticket was mishandled?

Giran Moodley demonstrated a case where a user said “not urgent” about a CFO unable to export invoices, yet the AI agent classified it as low severity. Using Braintrust, they traced the error, tweaked the triage prompt to better weigh business impact, ran an evaluation against a dataset with this edge case, and saw the severity classification improve.

[19:28](https://aie.addtry.com/c70e51fe-e591-40bb-ad86-38625d21b65a?t=1168000)

## Key moments

- **[0:00] Intro**
  - [0:15] Giran Moodley welcomes attendees to the first AI Engineer conference and introduces the hands-on workshop with Braintrust and Trainline.
- **[4:07] Agenda**
  - [4:07] Workshop agenda covers building a multi-stage support triage agent, instrumenting with Braintrust, evaluating with a golden dataset, and deploying to production.
  - [5:45] Q: How many attendees have built a generative AI POC but failed to productionize it? Many hands go up, showing the common industry hurdle.
  - [6:05] Giran Moodley: 'LLM systems, as my 5-year-old would say, two plus three equals 10, daddy,' illustrating the non-deterministic challenge.
  - [7:26] Observability is key: logs tell what happened, but tracing reveals system behavior and failure modes.
  - [8:36] Breaking monolithic prompts into microservice-like stages improves reliability, similar to traditional software engineering practices.
- **[9:55] Braintrust**
  - [9:56] Braintrust is an AI observability platform that raised $18M at an $800M valuation to help organizations ship quality AI at scale.
  - [10:36] Braintrust's founder previously led ML at Figma and built a document extraction company, motivated to solve evaluation and tracing at scale.
  - [12:22] The AI development flywheel: start with evaluations, instrument production, identify failures, remediate, and monitor continuously.
- **[12:56] Trainline**
  - [12:56] Oussama Hafferssas explains Trainline sells 6.3 billion train tickets annually, serving 27 million active users with a multi-agent travel assistant.
  - [14:25] Trainline's travel assistant is a proactive agentic system that handles refunds, train changes, and handovers to human support, not just a chatbot.
  - [15:25] Trainline does both classical ML for predicting train disruptions and generative AI multi-agent systems, operating at peak scale.
  - [17:34] Oussama Hafferssas: 'Are we breaking things at Trainline? We want to move fast without breaking things,' using Braintrust for quality.
  - [18:06] Trainline views agentic systems as a blend of deterministic software and non-deterministic ML, requiring combined quality checks.
  - [20:08] Oussama Hafferssas: 'We are here just because we are convinced Braintrust works for us. We are not paid.'
- **[28:35] Agent Build**
  - [28:35] Giran Moodley outlines the support triage agent: a five-stage pipeline collecting context, triaging, performing policy review, drafting replies, and escalating.
  - [33:57] Building a basic single-shot agent: a single LLM call with structured output works for a demo but lacks reliability for production.
  - [40:32] Adding local tools for determinism: help-desk articles, account history, and internal data give the LLM more context to reduce failures.
  - [41:30] Implementing specialist stages breaks the agent into explicit functions: collect context, triage, policy, reply, finalize, each with its own prompt.
- **[46:19] Observability**
  - [46:19] Instrumenting with Braintrust: wrapping LLM calls and function spans creates nested traces to debug the full execution path.
  - [49:01] Tracing enables real-time insight into what's happening in the application, not just what happened, crucial for remediation.
  - [49:44] Running the demo with Braintrust tracing: logs appear in real-time, showing prompt tokens, cost, latency, and metadata for each stage.
- **[56:43] Evaluations**
  - [56:43] Evaluating AI systems requires a golden dataset of edge cases and scoring functions—deterministic and LLM-as-judge—to measure performance.
  - [58:20] Creating a golden dataset: 10 test cases covering severity, policy, and structure, uploaded to Braintrust for systematic evaluation.
  - [1:00:47] Running an evaluation against the golden dataset produces an experiment result, allowing comparison of scores across runs.
- **[1:05:07] Deployment**
  - [1:05:07] Deploying with managed prompts: moving prompts, tools, and parameters from local code to Braintrust's managed infrastructure for safer changes.
  - [1:09:57] Non-technical users can change model or prompt parameters in the Braintrust UI, then run evaluations to test the impact without code changes.
- **[1:13:58] Online Scoring**
  - [1:13:58] Online scoring: applying the same scoring functions to live production logs via automations, sampling initially at 100% to establish baseline.
- **[1:19:13] Remediation**
  - [1:19:13] Production failure mode: a CFO's urgent invoice export is misclassified as low severity because the user said 'not urgent'—severity must consider role and context.
  - [1:20:46] Replaying the failure in Braintrust: running an evaluation against the specific edge case confirms the misclassification before fixing.
  - [1:22:58] Inspecting the trace of the failed case shows the agent's reasoning and tool calls, revealing why it misjudged severity.
  - [1:26:26] Experiment comparison shows score degradation from the failure case; Braintrust's diff view highlights the drop and guides remediation.
  - [1:27:51] Remediation: updating the triage prompt to explicitly consider user roles and business impact, then re-running the evaluation.
  - [1:31:14] After prompt fix, re-running the evaluation shows improved scores, completing the flywheel and confirming no regression in other cases.
- **[1:33:05] Summary**
  - [1:33:05] Key takeaways: prototype vs production gap, break into stages, observability is table stakes, golden datasets for evaluation, continuous improvement.
  - [1:35:13] Giran Moodley: 'If you've got a production AI application and you're not tracing it, you need to go back to the drawing board and get that done.'
  - [1:36:12] Start by picking a critical production system, add Braintrust tracing, collect edge cases as evaluation cases, build deterministic scores, and iterate.

## Speakers

- **Giran Moodley** (guest)
- **Mayank Soni** (guest)
- **Oussama Hafferssas** (guest)

## Topics

Agent Evaluation, Agent Engineering

## Mentioned

Anthropic (company), OpenAI (company), Trainline (company), BrainTrust (product), GPT-5 Mini (product)

## Transcript

### Intro

**Giran Moodley** [0:15]
Good afternoon, everybody. Welcome to Sunny London, hey. Um, is everyone's first time here? I think it is, because this is the first conference. Amazing, amazing. Well, thank you very much for joining today's session. Um, hopefully you are in theright session, but for those who, uh, need to double-click, this is a hands-on workshop to delivering quality AI applications with Braintrust, and we're also partnering with our colleagues at Trainline, which I'll introduce themselves shortly.

Uh, so probably you'll be wondering, who's this guy? Does he even go here? What's his name? Introduction to myself is, uh, you can call me Gerard, but like the band, Gerard, Gerard with a G, so hopefully folks are Simon Le Bon fan.

And, uh, I've had a bit of part of helping organizations and enterprises help scale adoption of mission-critical systems, and now moving into the age of AI. Um, you know, background in, uh, formal mathematics, so I know there's all the rage going from machine learning to data science and now AI engineering, and this comes at a very topical time.

Uh, feel free to connect me with LinkedIn if you want to talk, or just want to, uh, you know, just do general chit-chat around this area. I'm also joined by my two friends over from Trainline. If you want to come over and introduce yourselves, team.

**Oussama Hafferssas** [1:21]
Of course, if the mic is working. All good? Yes. Hello everyone. Uh, thanks for coming to this workshop, especially after that lunch break and, uh, and the sunny time outside. Thank you very much for coming here. Uh, so my name is, uh, Oussama.

I'm a senior AI ML engineer at Trainline. Uh, for those who know me, yeah, one person. Uh, yeah, I was a staff platform engineer before, and I have, uh, background in computer vision, um, and I was doing also mobile apps on the side, so nothing to do, uh, with, with AI at, at some point, but yeah.

Now we are doing AI together with Mayank at Trainline. Here we go.

**Mayank Soni** [1:59]
Hi everyone. My name is Mayank. I, um, I'm a senior AI engineer at Trainline. I have background, research background in LLMs, but my research is in the pre-big LLMs, so like if you've heard of BERT or the older LLMs.

Um, so I'll be continuing building a state-of-the-art, uh, agentic products at Trainline. If you've come here from abroad, I'm sure you must have used Trainline. If not, please do, because it's a, um, it's the state-of-the-art app for, um, buying tickets and other things.

So, uh, yeah, welcome, and we're very excited to host you at this workshop, and, uh, yeah, we'll guide you through the, the hands-on, uh, experience.

**Giran Moodley** [2:38]
Fantastic. I'm also joined by some of my colleagues as well from Braintrust, coming over the pond. So Phil, Eric, and Rose, have you just shot your hands? So don't worry, folks, you're in safe hands if you're stuck, so just holler and they can help.

Fantastic. Uh, just want to do a little bit of housekeeping as well. If everybody could join into the AI engineer Slack channel or, uh, well, there's an AI engineer organization, but there's also a specific Slack channel which we'll be using today to help, uh, progress the workshop.

So if you are stuck, we can use the Slack channel to help each other out. Again, we're already on there, but as we progress and get to the hands-on element, and if you are stuck, um, this, this will really help.

We're also providing a, uh, cheat sheet, so if you are encountering any particular hurdle, there's a step-by-step instructions which help you just get to where we need to go to the workshop without, you know, having to, to feel like you're, you're, you're falling behind.

Again, a lot of the assets which we share today, uh, you know, is publicly available. Um, again, we can have any particular follow-ups, um, if needed, but we'll give you a few minutes to make sure you get onto that and then join the channel, AI Engineer Europe 2026 Braintrust Workshop.

Uh, it's a public-facing channel.

Allright, team. Um, I guess we can start. Um, yep, just for the people who are in the back, um, we'll need you to join the Slack channel. So if you've seen Pierce, then we'll pick up. So with that, team, thank you, everybody.

Um, let's proceed. Okay, so just to kind of help orient today's, uh, workshop, we'll be breaking it into kind of three main sections. We'll spend a little time just setting the bit of establishing the background, why we're here, and why this workshop is relevant.

### Agenda

**Giran Moodley** [4:20]
Hopefully it'll set the context for when we go into the workshop, uh, building, uh, the system, talking about this, you know, how do we ship, uh, AI at quality. And then we'll wrap up for, uh, the key takeaways, and we'll be around to answer any kind of, uh, open, uh, questions and answers that we have from the field.

Okay. So hopefully today, um, there's going to be a lot of people from very different backgrounds. Um, we intended this workshop to be catered for, again, probably everyone here knows what an LLM is. I don't want to insult anybody, but hopefully you're starting to, uh, explore your journey in terms of your, um, mature, maturing your, your operations in terms of building AI systems.

So whether you're an AI product engineer, probably an applied team, or come up from a traditional machine learning folks, maybe you might be a platform operation infrastructure, um, this is really appropriate for you. Just kind of for show of hands here, like who here comes from a, let's say, a traditional data science background, perhaps.

Okay. Interesting, interesting. Who here perhaps comes from maybe software engineering or then pivoting into AI? Okay. So this is definitely theright room for you. Uh, so hopefully we'll, we'll be able to kind of accelerate things, uh, as we progress.

Okay. Um, just to set the context here, um, I think this is not an uncommon expression, which we are seeing more broadly across the industry. Um, again, bit of show of hands. Who here has done a, a machine learning or AI POC?

Let's say, let's say more specifically, a generative AI POC, but then has failed to kind of take that into production? Okay. It's got a few hands, yeah. I'll be worried if they know everyone didn't, but this is kind of a, a key thing which, you know, speaking to many of my customers, you know, executives, top-level folks, all the rage, you know, they're thinking about this new technology.

Uh, it's not necessarily new, but, uh, it's new for a lot of folks, especially in kind of more enterprise and regulated industries, and then they're trying to take this to, uh, delivering value to their customers. But unfortunately, there's a big hurdle between, um, taking what you might develop locally on your machine, and then industrializing it and making sure it works in Angular.

But the key thing that we've seen from all of the research out there, it's not that the models aren't particularly smart. Like, we've got very sophisticated models, whether you're building something in-house or you're using kind of a top, off-the-shelf, you know, top, uh, commercial LLM providers out there.

What we do see, more broadly speaking, is the type of operational rigor when it comes to delivering these systems at scale has not kind of kept up. Because traditional software engineering, very deterministic, one plus one equals two, great.

LLM systems, you know, as my 10-year-old, or sorry, 5-year-old would say, um, you know, two plus three equals 10, daddy. So it's having to kind of adjust it and make sure that we're delivering that to scale. And again, holistically, what we're seeing, uh, when shipping these things is the fact that, again, people think a demo state is suitable.

Um, but clearly, doing two, two to three to five demos, great. Putting in production, everything goes awry. Um, treating some of your logs as observability, and then this is really critical to, to how we work at, at Braintrust, is, you know, logs will tell you what has happened, but sometimes you need to go deeper into the system and understand its behavior.

And this is really where observability comes into play. Uh, something as real is, you know, works on my machine, fails in production, I try to patch the prompt, um, and then, again, it's operational until the next issue happens or the next failure mode.

But again, how do we keep track of that? Um, if, especially if you don't have a system in place, um, irrespective of what tooling you use, um, this can, uh, you know, has a categorical effect. So again, a lot of what we see, again, is not to do with the tooling technology.

It's down to more operational workflows. And this is really what we're aiming to help you in today's workshop. Okay. So again, as I mentioned, it's not the prototype. It's getting to a state where we're knowing exactly what's changed in the system, how do we interop with that, and then how do we systematically put a set of rigor so that we can get better and better.

Remember, 100%, uh, our target is not 100% coverage. It's getting as close as possible, uh, while maintaining, um, fixing the gaps that might have existed.

Um, and again, something that we see time and time again is, again, a one prompt might work, but as you move and industrialize it, you want to probably do things like breaking this down into each individual, um, sectors of responsibility.

So again, if you do come from the software engineering background, you know about these, you're breaking down the model into microservices. We'll outline a very similar approach here when we talk about building these systems. Um, again, making sure that we're understanding these changes, uh, putting a set of systems in place.

This is really what we want to do in today's workshop. Okay. In terms of today, um, hopefully, as I mentioned, this is a hands-on workshop, so we're going to be going into the terminal, we're going to be going into the UI, and we're going to be going through the step-by-step and guiding you along the way.

So we'll be doing a staged AI system, uh, with, uh, multi-stage, uh, tool calling, which really allows us to, um, see this more, uh, let's say, agentic flow. We'll then use Braintrust to instrument and see how the performance of the application is working.

We then also want to then, you know, take a look at creating what we call, um, or identifying failure modes using, uh, a Golden Set. So we'll push that through. We also then want to talk about how do we industrialize it.

So then moving from, hey, it just works on my machine, to something that you can use in production and have it managed forward with a system in place. And key thing is identifying, you know, those, those edge cases.

Because again, you can create a, a test data set, but ultimately, um, there's no substitute for real-world data. So again, we'll be able to show you how we would take those real signals in and evaluate and complete the loop.

### Braintrust

**Giran Moodley** [9:56]
Um, so just a bit of introduction here today as well. Who here has heard of Braintrust or played around with Braintrust? Can I get a show of hands? Okay. Great. Fantastic. So okay. Love this. Love your own hands.

So just a bit of introduction to Braintrust. We're, we're a company now that's, uh, I think just, uh, shy of three years old. Uh, we're approximately a, um, a Series B company. So we just announced that a few months ago.

We raised $18 million at a $800 million valuation. Um, we have investors such as Iconic, AZ16, uh, as well as Graylock, uh, to really talk about, you know, helping organizations ship quality AI at scale. So we're the platform for AI observability.

Uh, we've got a heavy user base, uh, globally, but we're also expanding our presence heavily in Europe. So again, I'm one of the first engineers to, to join and help build out, uh, our go-to-market function here. And we're very excited, again, with our customers and our friends over at Trainline to do that.

Some of our other local customers include like Lovable, DoctorLib as well, uh, who are really pushing the forefront of these AI systems.

Uh, again, when it comes to using kind of Braintrust, again, I don't to, again, I want to kind of get hands-on with it, but where we really distinguish ourselves is being able to do this at scale. So our founder, Ankur Goel, um, this is actually his third time building Braintrust.

Um, so he's an expert when it comes to sort of database systems, and he's built, um, uh, uh, a company called Impera previously that was acquired by Figma that talks about document extraction. And he's led the ML, uh, machine learning team out there.

And because he realized, you know, building these, these evaluations are hard, understanding production traces are hard. So if we're having this issue, I'm sure there are other organizations out there which are doing the same. And so he founded Braintrust to really help do this at scale.

And as we're doing and understanding this, these, these traces which are coming in and again, being very highly semi-structured data, which changes, uh, he realized that traditional analytical systems weren't fit for purpose. So we've put, created a, um, a new category of databases called Brainstorm, which really helps identify and accelerate this at scale.

With us, again, we're tool-work, we're platform agnostic. So irrespective of the agent framework you're using or, or the LLM providers out there, we're, we're, we're, we're intending to help you deliver value, uh, irrespective of that. Okay. One thing that I will talk about as we progress, uh, in this workshop is the, this is the concept of a flywheel.

So, uh, again, if you ever come from agile development, um, you know, perfection is the enemy of good. We want to start somewhere. So even if it's a case that it's a new application and you don't know how it's going to be in production, we can start off with an evaluation set.

If you have an existing application to, to instrumenting, that's great. We can pull that information in and identify the failure modes here. So the key thing is get information into the system, identify those modes, remediate, ship it out, and then monitor and complete this flywheel, uh, again and again until you get to where you need.

Allright then. Um, you would have heard, uh, a lot from me. So one of the things I do is I'm going to provide, uh, my colleagues over at Trainline to maybe just share their experiences of, you know, prior to, to Braintrust and, and how they're helping us.

### Trainline

**Giran Moodley** [13:08]
So, uh, Oussama, I'll give it to you.

**Oussama Hafferssas** [13:10]
Thank you very much. Uh, and hello again for people who are joining us, uh, just now. Um, as my, uh, mate, Mayan, uh, introduced Trainline, uh, we are, uh, a company that's actually helping people, uh, get on the trains.

Uh, trains are different than planes, uh, if you don't know. Uh, there is this one, like, worldwide system, central system for all planes around the world. It's not the case for trains. Uh, and in Europe and the UK, it's very hard, I would say, if you would like to install an app for each career, it will take up the whole space on, on your mobile phone for sure.

So Trainline is actually being that platform, uh, basically, uh, to help you book tickets, uh, mobile app agnostic, platform agnostic, career agnostic. You do it on one app and you can book a train from Paris to London, uh, from, from, uh, from Lyon to, to, uh, to Milan, or whenever you like, basically.

Every career in the U. Uh, we sell, like, almost, almost like 6.3 billion, uh, of tickets on the trains. 27 million active users and counting. And the other interesting probably one for, uh, for this, uh, conference is how many, uh, AI conversations actually that we have from our, with our travel assistant.

So we do have a travel assistant that is exposed to people. And it's not just a chatbot. Uh, it's actually a multi-agent system that actually can handle refunds for you, uh, can handle changing trains for you. So it is very, I would say, a proactive, um, agent system, agentic system, and not just, uh, a chatbot.

Probably Mayan would like to add more about that.

**Mayank Soni** [14:48]
Yeah. So one of the benefits of having 27 active customers is you've got a huge space of how you can serve agentic applications live to the customers. One of the examples of that is what Oussama talked about is a travel assistant, which you can get to from a ticket window in the application.

Um, it's an agentic system, which is something we want to talk about, uh, a little bit later in the slide.

**Oussama Hafferssas** [15:10]
Yeah. That's great. Awesome. Which brings us, I would say, to the next, uh, point. Uh, I will keep it short. Uh, selling train tickets and being like, uh, a, a train, uh, ticket company. How come that we are doing, uh, machine learning?

Of course, we, we can. Uh, there's so much things, uh, to do and to help people, uh, in terms of their journeys, getting their tickets, getting their trains, getting back home, basically. Uh, we build, we do two things.

So the classic ML part, which is actually building models, we do that. We build ML models inside of Trainline from scratch, from data to model. This is something we do. And we also do the multi-agentic, um, generative AI systems that we are now familiar with, on top of LLMs that we, uh, love and cherish, all the tooling and the context engineering, all of that.

We do that. So we do these both sides of the story, uh, at Trainline. And these are two examples of, uh, what users are actually, uh, using, uh, on top of those, um, systems. On the left side, so this is, uh, you can, you can think of it as your weather application, but for train disruptions.

So basically, you have a ticket for a train. You get there. We know if this train would be, uh, would be disrupted or not, if it will be, uh, pro-probably late or not. We, you know, we know that, but based on huge data that we have on top of it, it was a machine learning model that was trained and can actually, uh, predict, uh, train, uh, disruptions, uh, being late and all of that.

So this is the classic ML part. The other one, that's the travel assistant that I told you about. It is very, as I said, it's very, I would say, advanced, uh, multi-agent system. It can show you alternative trains if your train is canceled or something wrong with it.

Good luck doing that yourself, even with ChatGPT. And, uh, the, um, the other one is handle refunds. So you can actually get for a refund, um, uh, on your ticket if your train is late or all of that.

And it actually can give you all of that, uh, without a handover. And also can, uh, can do the handover to actual, uh, uh, human customer support, uh, in our lovely customer support team. Um, which means that if you are doing this at production, in production level and that scale at, uh, Trainline scale, it means that, uh, or you can ask the question, are we breaking things at Trainline?

Uh, of course, we don't want to, to do that. Um, and we are moving fast because technology is definitely moving fast. And, and this is why we are here. We are, we are here to show you, like, what are we doing to move fast without breaking things, uh, in terms of course, uh, of AI.

And we can do that on top of handling the complex, uh, software systems that we, uh, we love and cherish, cherish from APIs at scale and serving millions of users. Um, so how we would like to think about it is this scale.

So we know for sure that whatever we have as software systems, this is the deterministic side of the story that we have. And on the other side, building the ML models, this is the non-deterministic side of the story.

And we know for sure that the agentic systems are in between. There are parts of them that are deterministic. There are parts of them that are not definitely deterministic. Uh, uh, and this is how we, uh, the framework of thinking that we have.

In terms of quality, how are we handling that? That's the question. For the ML models, we do, we do care about the quality of data, uh, and, and the that we use for training the models. But also we have on top of it, of course, uh, ML, machine learning evaluations, uh, whether offline or online.

Offline, it means like before going to production, you, you need to do your evaluations. And online is the data from production. You get it and evaluate your model if it's doing, uh, well or not. In our case, for instance, for that for weather forecast, uh, is it predicting the state of the train, uh, if it's disrupted or not?

Is it, is the model correct in its prediction? So that's one, uh, one example. On the other hand, for people who are, who are familiar with software engineering, we do all, I would say, quality checks and the diagrams or whatever and the tooling that we have for handling, uh, quality at scale, very large scale at Trainline.

Uh, and for those systems, I think you already guessed it, it's basically a combination of both. It's not one without another, for sure. We do everything that we do quality-wise in terms of deterministic systems, but we also use the, uh, the evaluation, um, uh, side in, um, from the non-deterministic systems.

Uh, so yeah, that's, I would say, how the, I would say, the framework of thinking that we have at Trainline, thinking about these systems. And, um, definitely, uh, Braintrust, uh, is helping with that. And by the way, disclaimer, we are not, I would say, we are here just because we are convinced that it Braintrust works for us.

We, we are not paid. So that's 100%. Uh, we, we are happy customers. We, we have been with Braintrust for a long time. Uh, and we use it in different, um, some of, uh, Braintrust features. We actually use them here.

So for instance, for the ML AI evaluations, we do that. And we, we, we follow the scoring of travel assistant on many, many, many levels from, uh, from tone of voice to, uh, actual helpfulness, uh, when it comes to tickets.

And tickets are very, very, I would say, complex, uh, in terms of reasoning and what you should get, what you do not get, depending on the if the train is late or not, uh, the type of ticket. It's a return ticket or advanced.

So many complex cases. Uh, but yeah, we do follow that, uh, evaluation side. Probably Mayan would like to add more about that.

**Mayank Soni** [20:55]
Yeah. So can I get a, a raise of hand from people who have struggled with LLM costs, number of tokens, switching models, problems like this,right? So yeah, that's nice to see. So it was a problem with Trainline as well because we do this at scale and the amount of OpenAI, Anthropic amount that we pay is just like bizarre.

So we have to keep switching models, which is like the best model for our use case, cheaper models, uh, talk to models with efficient tokens. Now, anytime you want to switch models, you want to make sure that it is performing at least at the same level as your current model,right?

Before Braintrust, we had no specific way of doing it because we didn't have score set up. We didn't have, you know, sort of like the evaluation. So with the usage of Braintrust, what it has enabled us to do is to simulate how the performance of the lower model would look like.

And we've used Braintrust extensively to run offline evaluation and see what the effect is going to be and also online evaluation to see that the intended effect that we observed in offline is, is as expected. So that's, I think, one of the use cases.

The other one is more generic, which is like it would have taken us like a lot long to evaluate a new feature shipping into Travel Assistant. But with Braintrust, we have been able to make sure that we are assured that the new experience for the user is going to be good.

So Braintrust has helped us a lot, uh, in shipping fast.

**Oussama Hafferssas** [22:17]
Cool. And also the other part is observability, for sure. Uh, we do, we do use Braintrust for observability. If this one is working, yes. Um, so yeah, this is a true example from, uh, from Braintrust. We are spoiling basically the workshop whatever you are going to see there.

But yeah, we can, we can track everything regarding tool calling, the other agent, um, the in, in terms of numbers, in terms of quality, that those kind of insights, you will need them to get from proof of concept to actual system in production or in your company or for something actually, um, uh, that's being used there out there.

Uh, and users find it, I would say, true working product and not, I would say, just any slope, uh, AI generated system. No, we are, we need, we will need that for definitely for, for those cases. Uh, would you like to add something here?

**Mayank Soni** [23:09]
I was just going to say that Braintrust enables you to look inside complex agentic workflows up to like tool call level, token level, which is like very insightful and helps you debug a lot of things in production and before you deploy it.

**Oussama Hafferssas** [23:23]
Yeah. And one last thing probably is the cross-functional, uh, uh, friendly, uh, point. Uh, that's something that we have discovered, uh, along the way. Uh, because we are building those systems from Travel Assistant to the models, at some point we need to have, we are a big company and we have people from product, people with non-technical backgrounds, and we need to communicate and we need to share many things.

And they also need to self-serve some of those things. We cannot, I would say, babysit any, uh, people and say, "Hey, you should do this, should you let me get you the logs, download it, and send them down."

That does not work. At scale, uh, we need like a way like where we can help, I would say, can work cross-functional way, um, and let people be, I would say, um, uh, uh, free to do whatever they want and self-serve with, with data and insights.

So this is what we have also discovered, uh, along the way. And Braintrust helps us, I would say, uh, with many requests that we had, but yeah, really appreciate it, uh, for sure. And, and of course, more we are using just a part of, uh, uh, that system.

We have our own things. Braintrust, I think you are building even more, more, more toolings. So, so definitely, uh, more to come. I think the workshop will definitely help you discover all those, uh, things. And, um, of course, have fun building, uh, during the workshop.

I would say laptops out. And if you are interested, Trainline is hiring, uh, in the AI engineering side, of course. And, uh, give it to you, Shao.

**Giran Moodley** [24:50]
Perfect. Well, thank you so much for that, Tim. Uh, yeah. And just to say again, an impersonal thank you, uh, and extension of my wife. We thank you so much for the split-saver functionality. So, uh, give me kudos to that.

Allright then, team. Allright, let's proceed with the, the setup. So again, this is going to be a hands-on workshop. Uh, better dust off your bash skills, team. Jokes does. I've got a nice raffle command to help out with that.

So hopefully everybody, or at least most folks, have joined the Slack, uh, organization for AI engineer and then also joined the, the channel. I've put a link to this, uh, but there's also a QR code to the repository which we'll be using onto there.

So I'll give it a few minutes. Um, really key things is, um, signing up to a free Braintrust, uh, account. Um, if you use Gmail, you can use a little plus sign trick to just create a new account.

If you, um, you know, keep it if you are an existing Braintrust, uh, user today, uh, so hopefully it doesn't pollute your, uh, existing one. You will also need access to an OpenAI API key. Um, this should be easy for you to generate.

If for some reason you're not able to generate a key, then please let my colleagues know. We'll be able to send it a DM you on, on Slack to use for this specific session today. Um, and I think obviously as an engineering conference and especially in AI, uh, if you are using AI coding assistants, feel free to use that within your IDE or in the terminal to, to guide you along the way, ask questions around the code base and so forth.

Um, I've tried to simplify a lot of the scaffolding in place. So I'm using MIZE to, uh, manage, uh, Node, uh, and PNPM, um, on, on the machine. But alternatively, you can download, uh, Node v222 and the specific version there.

It should be fine. Um, but again, I just want to caveat, especially for this workshop, I fixed that specific, uh, let's say runtime. I'm using Make to, uh, just again, a bit of syntactic sugar to wrap the commands.

Um, if for a reason you don't have Make installed, if you're on a Windows machine, you can then just run the, um, PNPM commands, which are then just wrappers for the package.json file here today. So yeah, we'll give it a few minutes, uh, for folks to, um,

to, to scan and clone. I just want pointing out as well, um, I have a, uh, in the sheet today, I can see lots of folks have already, um, gone through it, but, um, yeah, we provide a step-by-step logo.

So as mentioned, as we're proceeding with the, um, workshop today, if you are stuck, you can please ask questions on the Slack channel, but also you refer to this. Uh, as mentioned, this will stay this is a public-facing asset.

So, uh, even if you, uh, outside of this workshop, you're stuck, you can kind of go through it at your own, uh, leisure.

**Oussama Hafferssas** [27:47]
That's good.

Okay.

**Giran Moodley** [28:06]
Just kind of maybe a show of hands, um, who here, um, is not able to get an open API key, open AI, sorry, API key. Hopefully folks will be able to provision that. It's, uh, really critical to this.

Yeah.

**Guest** [28:24]
There are some people joining late who don't have the QR code.

**Giran Moodley** [28:27]
For the Slack.

**Guest** [28:28]
For the Slack, yeah.

**Giran Moodley** [28:28]
I can do that. Yeah.

**Oussama Hafferssas** [28:34]
Mic.

### Agent Build

**Giran Moodley** [28:38]
Okay, folks, I think we did have a few late starters, so, um, yeah, I don't want to spend too long with this, but if you did join late, please scan that QR code, uh, which will give you access to the AI Slack engineering, uh, organization and then join the AI channel, uh, AI Engineer Europe 2026 Braintrust workshop.

It's a public-facing channel. Through that channel, you'll be able to get access to the pinned, um, uh, content. So the repository as well as a cheat sheet, which we'll be following as well.

Just, yeah, just a bit of a sense check, folks. Um, hopefully most of you would been able to join the Slack channel and are able to clone the repository, have access to that. Allright then. Um, as I mentioned at the start of the, um, the agenda as well, what we'll be doing is creating a, um, an example, let's say, support triage agent.

So, you know, hopefully this kind of, uh, exemplifies a lot of, uh, systems that folks maybe have already built or, or trying to build. Um, it is, it is a fictitious, uh, um, application, uh, designed specifically for this workshop.

Please do not use it in production. It's just really around to just teach us, um, you know, how, how do we build these AI patterns and for scale. So the idea there is, um, given, uh, a ticket might be raised in the particular system, uh, we have a set of agents that goes through a, a pipeline process.

I'm saying pipeline, but a stage process for tool calling, uh, that produces, um, a set of information that can be emitted into downstream systems, uh, from that perspective. Okay. Um, just to kind of help visualize what, uh, the system does, again, mentioning we're taking the ticket input, uh, we have a first step, which is collecting the context.

It's quite a deterministic way of extracting information. We then proceed with the agentic type of portion where we've broken it out into three stages. So we have an LLM and two calls to, to triage that particular issue. Um, we then want to do a policy review to make sure the output is, is correct.

We then want to create and draft, let's say, a customer-facing reply with reply writer. And then we want to package this together. And depending on, um, how severe this particular ticket, we can invoke another tool call to whether we need to escalate, uh, this to, let's say, a human in the loop, uh, and then draft a final result.

So, uh, again, not too, too complex, but just give you an idea of, of the, the system that we'll be trying to build and, and operationalize, uh, in this session. One thing to point out is, you know, as we begin to use Braintrust, it's like really where does it fit into helping you, you know, deploy and, and manage this at scale.

I just isolated, uh, this here where, you know, everything we're going to be doing towards the end, especially as we progress, we'll trace it end to end, um, as my colleagues talked about. Um, we will use manage mode for prompts.

So the offloading that what you'll do locally into a secure environment. Tool calls as well will be managed. So again, uh, how do you, you know, talk about getting to external systems? And lastly around evaluations and scores, which will be run through, uh, our Braintrust infrastructure.

It's worth pointing out as well, if you do go into the GitHub, uh, repository, uh, there's a page GitHub pages version of this as well. So again, the slides will be readily available for you to use and consume.

Okay. Key thing as well, uh, with this is I've tried to help you, um, do each phase and checkpoint this. So if you are stuck, the idea is just use get, get checkout to a specific tag or the branch.

But in this case, each individual tag or branch is fully runable at that stage. So if you're ever stuck, again, check get checkout to that branch. So you'll make install, make setup, run the commands, and then you should be able to get, uh, near identical output every, every, every time.

So again, this is all documented in the, the README, but it's also included in the cheat sheet as well, uh, for you to progress. Okay. So yeah, just to kind of give you the sequence of events, uh, we'll be doing kind of the scaffold and, and setup.

Uh, I'll talk about building a basic agent. Again, this workshop isn't really designed to talk about building agents. It's about, okay, once you do have an agent, how do you then operationalize that? So, but just for brevity, we want to kind of do it step by step.

And then we'll kind of get into the, the core of it where we'll add the tracing, talk about the evaluations, you know, talking about the, the golden set and identifying where we can, we can improve. And then again, tying it all together and using kind of managed infrastructure within Braintrust to, to help operationalize this out.

And again, talk about that collaboration that my colleagues at Trainline talked about. Also kind of the key thing, which is once you do identify a production failure, it's then how do we apply a fix to this and then complete that flywheel and give you a finalized asset.

Okay. So step one, we're going to be talking about building the agent. So in this checkpoint here, um, again, we need to start somewhere,right? So what do we do? Uh, we will take an initial call to an LLM.

We'll set up a prompt, one shot, one in, one out, and get an output. Um, so again, if you're doing a proof of concept, um, this might be great, uh, as part of that initial spike, but again, we know there's, there's work to be done.

But yeah, for context, we're just going to be stepping through this. But as I mentioned, just because it works in the demo doesn't mean it's necessarily going to work in production. Um, I've put some pseudocode here. I just do articulate here.

But you can see here, as, as with all, uh, these, um, you know, language models, we provide a set of assistant prompt. We provide the user and especially the text, and we pass that output back into our application.

So one function, one model call, uh, we have the structured output, which we want to receive as part of the, the output. Okay. So what we'll do as well, uh, when we do the, the scaffolding and checking out to the first, uh, branch, uh, we can run a set of pre-built tickets.

So it's, it's a case that's kind of already done in code, or you can use a command line. So I'll just demonstrate that shortly, what it would look like, uh, with the outputs, and you'll get something similar to that in, uh, JSON format.

Okay.

So hopefully everyone can see this at the moment. Um, I have the, the application here.

Okay. So I'm just going to go to basic checkout.

Okay. And, uh, what's this?

So if I take a look here, so I'm now checked out into, uh, that first tag. So building a basic agent. Um, I've got the application here, which is again, very similar to the, the prompt code. Um, we're creating the client, uh, calling OpenAI SDK.

Again, for brevity, I've just, I didn't use any agent, agent SDKs, but we fully support that as well as we proceed with the, the workshop. Um, so this is really available and you'll see again from make file. Uh, if you don't have Make installed, then you can just execute, uh, PNPM scripts or I don't know if you, if you use NPM or YARN, that's also possible as well.

I haven't tested it, but theoretically it should work 'cause it's just using, uh, package.json. So in this case, let's say you want to do something like, you know, make tickets.

So I want to give it a thing. Say, you know, my password

needs to be reset. Um, in this case, I've provided some defaults. So I'm just click enter, enter, enter. And now it's just making a call to OpenAI. I'm just using, um, you probably would have seen from the environment variable file.

I'm using GPT-5 mini. You can switch it if you wanted, but, uh, just for the purpose of this, we'll keep it simple. Um, so you can see here, I've got the ticket and it's provided some, um, some output here as well.

But again, that's a singular shot. I think we've something to say.

**Oussama Hafferssas** [36:53]
Okay.

**Giran Moodley** [36:57]
So, uh, you know, based on this output, you know, it looks fairly plausible. Uh, but again, it's not going to account for a lot of the edge cases that we want, especially if you've got a lot of nuance to the organization, which we're trying to build into, uh, the logic here.

Uh, I can even do things like make, uh, demo.

So make demo.

If we got the scripts, um, yeah, it's the same thing. I'm just calling the same function, but I've got it codified as JSON here.

So yeah, the JSON fields are available to see.

Awesome.

Okay. So that's fairly straightforward. Uh, I don't want to dwell on that too much. So the next thing I want to do then is talk about, um, adding kind of local tools. So we probably want to say, look, let's try to make this a bit more deterministic.

Even though a prompt might be very well structured, I may want to bring in, uh, different ways of how it might operate. So, um, in this case, uh, I'm calling three different tools to look at relevant, um, help desk articles around that.

Uh, this could be both internal and external. I may want to look at certain things that have happened to the account. So let's say a customer might have done a certain migration and that might have an impact on backend systems.

And that's probably a reason why I want to, you know, create an escalation here. In the case, what I've done is I've made it a bit more deterministic. For the purposes of this workshop, again, this is kind of treated as code, but, uh, in reality, you probably are going to be interfacing with external systems like vector search, MCP, CLI, and other types of, uh, uh, interfaces to, to build out, uh, this capability.

And again, key thing here is, uh, the more things that you add, the number of ways that it can fail will also increase. So again, this is why tracing as we begin to go will become more important. Right.

So just one here. So against README here, next thing we'll do is go to add local tools. So I'll do it.

Allright.

So in this case, yeah, the tools that I created, uh, are then available here. But again, they just kind of checked in as code for simplicity's sake. And again, I can do the same thing where you, you know, make a ticket, say, password needs reset, account locked.

Okay. Now see, I just provide a little bit more information. You can see it's a little bit more verbose, uh, because we've introduced, uh, tool calls into this and it's given more context to the, the LLM.

Um, also worth pointing, if you are feeling stuck, folks, that a lot of these work, uh, or the, the tags of the workshop branches are built sequentially. So if you, let's go into, let's say, get tag number six, it's going to include everything as part of that.

So don't feel like you have to go through each one. If you are feeling stuck and you want to skip, you can do that as well.

Okay.

Let's get on to tools. So again, we already, I've already showed the code anyway. So, but this is an idea through pseudocode what it would look like.

Um, stages. So I think this is the next thing where again, we're breaking down that monolithic call for an LLM. We're now introducing tools and the next thing is even further drilling down to special stages of how the LLM should behave.

So you would have seen from that sequence, uh, diagram where I've done effectively five stages for this where I'm now setting up things to collect the context, triage, determining if it's meeting brand policies, providing a customer-friendly reply, and also something internally for our systems, and then finalizing the result for downstream systems.

Um, again, it's just coming from, you know, traditional software engineering. You break it down your problem, you can see the exactly where something's going wrong on the stack, and you'll be able to remediate. So again, where possible, try to be more explicit and break it down into challenges that you can work on.

So yeah, just a bit of pseudocode. Um, this is kind of like what it would look like. Um, probably going to put a debugger, um, with a new ID and, and take a look at that.

Okay. So I've kind of been doing this in part, but, uh, we've already started with the, uh, starter point and then I'm going to talk about, you know, doing the, the specialist stage here. So let's take a look at the next part of the README.

So get checkout special stages here. So if I take a look now, um, in the source folder, it's, it's got the individual, uh, functions, which are, are, are pieced out. Uh, the prompts associated with that, uh, is being loose.

So let's say here is a triage from what we're using. And if I go down to the application, you can see, um, uh, it's, it's done here. So I'm using a, a synchronous functions to, uh, execute.

Yep. And similarly as well, if I do something like, you know, make ticket, maybe I'll do something a bit different. What's another classical problem? Oh, it's just say, I need to upgrade my plan from pro to enterprise, but the website

is not working. Getting $500. You know, uh, so in this case, I'm in customer tier two. Uh, we're talking about, um, billing here and my account is actually account number three. So again, just to show you that we are, this is live.

It's not doing something that's hard-coded in the system.

Um, worth pointing out as well, this stage is slightly slow because we've broken down the individual one-shot LLM into sequential calls. So it is expected to take a little bit longer, but again, that's, that's part of building out this agentic flow.

And now you can see again, it's a bit more, um, verbose, but it's, there's a lot more thought into, um, this, this agent here. So I guess you can see again, we, we talked about a billing issue. Um, you can see that it, it believes that it's quite high because a customer wants to do that upgrade from a different tier, but there's obviously an impact to this from a revenue perspective.

So in this case, we should escalate to the appropriate, uh, people on our side. So you can see here what the escalation reason should be. Um, and the, both the internal and the, uh, the customer-facing reply as well.

So we've included a, a confidence score, um, to say, you know, if this is true issue. But again, um, as mentioned, um, tool calls will be able to pull information which is happening across different systems to provide a, a greater level of confidence.

Okay.

Perfect. Allright. With that in mind, um, so, you know, again, we've now hopefully shown how we can build an, take an agent, break it down, build something that's multi-stage, introducing tool calling, uh, to give us where we need.

The next thing is then to provide that information and, and start tracing it so we can actually see what's happening, you know, down to the individual details. And this is where, you know, observability piece comes into play. So what we want to do in this kind of section is break down the full execution past.

### Observability

**Giran Moodley** [46:28]
We know that the stuff is very, uh, nested in structure. So there's tool calls, there's additional function calls behind it. Um, we do want to track some of the key things. And I think, um, you know, Mayank pointed out, you know, the early struggles that they had to talk about, you know, latency, cost, uh, tokens account, especially for time to first token is a very important metrics that we see many of our customers trying to identify.

Um, also, you know, what were the inputs, what were the outputs, metadata associated with this, and then also including additional, uh, types of fields so that again, when we talk about monitoring and observability, we can query it, uh, within the UI on the fly and set up alerting if needed.

So again, having an output is not enough. We need to understand, uh, the full execution path and that's what tracing allows us to do.

Um, yeah, just to give you an idea of the context. And I know it's a very contentious topic to someone because again, I come from an, uh, uh, a background in full stack development. So, you know, I spit both coins.

It tells both, you know, Python, Go, as well as TypeScript. But, um, yeah, we, our SDKs is, is multilingual. So Ruby, Go, um, I think even .NET, we've got some folks who are using it and, uh, that's, that's all good.

But yeah, I've just kept TypeScript for simplicity here today, but yes, our SDKs do cover a range of, um, different languages out there that you can start tracing your application with.

So yeah, uh, a real key thing to this and, uh, one of my challenges I do see with our customers broadly is they will do an individual interaction, uh, against a, a singular, um, parent span. And this even might work with, let's say, multi-turn, multi-conversational agents.

What we want to be able to do is trace that into a nested structure so you can see everything, but in one interaction, let's say one conversation, uh, in a, in a parent span. So it's really critical as we start instrumenting our application to make sure that we're getting the, theright structures in place.

Otherwise, you're not, you're not going to be able to see the full effect of where things might go wrong in your application.

Okay. Um, again, this will just come down to reading the trace. Hopefully, folks, um, especially I know some of the, you've got more software engineers in the room, uh, that are using kind of the traditional observability tools out there.

This is not too dissimilar. Uh, but for folks who may be new to this, a screen, a trace just allows us to, uh, not find out what has happened, but what's currently happening with your application, uh, in, in, in real time.

So, uh, again, tracking every single call, bringing that metadata, and again, depending where the failure mode is, we can then identify, okay, what might be the, the course of remediation that we need to take.

Okay. So in the next stage, what we're going to do is add tracing to this, um, and then we're actually going to run it and then go into Braintrust and hopefully we'll see it happening. But one thing I do want to bear in mind before I do that.

So,

so let's go to our README. It's telling us to add tracing.

If I can spell. Okay. Now we're good.

And I want to keep it there. Okay. So I've introduced a, uh, helper script here called tracing. So, um, what we do quite well, uh, within our SDK is again, we want to not introduce more complexity where it's needed.

So if you are using, again, the standard, um, LLM provider's SDK, you can simply just wrap that function. We provide that out of the box. But then when it comes down to the individual, uh, calls, again, we can set up, uh, some, some helper, uh, scripts here to help just kind of wrap up, uh, as needed.

If you are using something like Python, then you can also have a, a decorator function which helps as well. But in this case, you know, uh, I've got a nice little, uh, function which helps with parent and child.

And then when it comes down to the actual, um, application tracing, uh, you can see here like I've got a, um, a child span which has been, um,

executed throughout this. Uh, again, I don't want to do too many code diffs at this point because the code is, um, expanding pretty heavily. Uh, but yeah, just kind of want to walk through the kind of key concepts for this.

Uh, okay. Um, before I run the, uh, application, I did want to go over into the, the Braintrust UI. Sometimes you might create a, uh, if you've already signed up for a free trial account, so hopefully you would have done that earlier today, create like a test project.

That's totally fine. We will create a new project as we execute this. Uh, really key thing, uh, as well, um, if you haven't done already, top left-hand corner, you go to your profile. Um, oh, okay. Let me just create one.

So, okay. Let's create that just for now. Uh, go to your profile and then where it says, um, API keys, um, you know, enter your, uh, a name for the key, generate that in, and then use that within your environment, uh, your, your .env file or secure that, uh, in a key vault if you have that already.

Um, there's also the OpenAI key, which we're hoping you generated. Um, that should also be used, uh, in

AI providers. So I've set this at the organization level. You can also do this per project, um, you know, depending if you want to segregate it by particular team or environment. Uh, that's also fully supported. Uh, but in this case, um, we'll need this key for later when it comes down to the managed and online scoring.

So, but just a bit of a tidbit, um, that key that you generated, make sure you put it into your, uh, Braintrust, um, organization here to be used later.

Okay.

Allright. With that in mind, I'm going to run the demo.

So here. So, and, uh, just point of reference, if you are ever stuck with some, you should just use make setup, make sure everything's in place. In this case, then we want to do kind of make demo and just we're going to run and execute those tickets.

I could even do it, uh, using the, uh, make ticket command as well.

Okay. So while that's running in the background, oh, sorry. While that's running in the background, um, if you go back to, uh, your application,

you would see, uh, there's a project called Helper Workshop. So that's the one that's will be created as part of this, uh, workshop there today. Um, if you navigate to the logs tab, you'll start to see this, uh, coming through in real time.

So what pointing out, again, thanks to our capabilities of Brainstore, uh, we have near instantaneousright into our system and read available shortly after. So especially it's, it's a non-blocking function. So a lot of our customers, uh, especially the more sophisticated ones are really using Braintrust at scale and really pushing the envelope.

So it's not the case, hey, I've got a thousand traces, you know, they're pushing, you know, tens of millions of traces at a, at a time across a very short period and they want to be able to again aggregate this.

And again, as you build more sophistication in your application, you're sending this out to more users, that's going to grow up pretty quickly and you need to be able to, um, have a system that can handle this at scale.

So once you start to see the logs that are coming in, um, come in reverse order, I'm just going to take a look at the first one here. Uh, we'll start to see the implementation application will be traced.

Um, there's a particular button here that allows you to view this in full screen, which I think is quite helpful. So, uh, as I mentioned, because we've got this nested structure in place where we're going through each, uh, in the sort of tree, um, this interaction at the top, um, is the, is the, is the demo ticket.

You can see the very top level where saying how long it took to actually run that invocation, uh, the prompt tokens in, out, cost, and latency associated with that as well. I've also included some metadata because I want to be able to kind of extract and filter that out as needed, and I'll show you how that would work.

Um, and everything's available here to view.

Uh, metadata is here. If I also want to take a different look at this, we can even look at individual steps. So in this case, I want to look at what's happened with, uh, triage specialist down to the actual, um, invocation to the LLM.

So again, SDK, uh, provides a lot of, um, flexibility around this. So you can see what was put in, uh, the reasoning behind it, and the information that we set output. And then coming down to the last, you know, should we escalate or not.

Um, another use that we will see as part of this is, uh, taking a look at the timeline. So just gives you an idea like a waterfall methodology to see, okay, was there a particular step which is taking longer?

Do we need to remediate? Uh, it's all possible to do. Um, allright.

So if I take a look back, logs page, let's get a bit out of that. Refresh. So I think there's kind of four tickets that were pushed. Yeah, they should come throughright now. Okay. Yeah. So that's, it's two tickets at the moment.

So again, that same information that we replayed is also available in the console.

Okay.

Yep. So we've covered tracing. Let's talk about the evaluation portion for this. Okay. It's quite interesting. Uh, again, depending where you are in your journey of building this application, uh, a lot of customers already build an application already, have it sort of monitored and pull that in.

### Evaluations

**Giran Moodley** [56:59]
But what happens when you have, uh, effectively a cold start problem where you don't know what you're building,right? So what's, what does good look like? Um, and effectively what does good, good enough ship to mean? Um, in the case of the support application that we're developing today, um, kind of non-negotiables, you know, have we categorized the support case?

Have we made sure that there's no severity or low severity outcome for, uh, these issues which are blocking? Um, does the escalation stay in policy with, with SLAs? Um, does the structure look, uh, look sound? And if we're making any particular changes, does it actually improve, uh, the way we look without actually breaking or having a regression, uh, to, to the application?

Um, we can do this using evaluation. So an evaluation, um, for those hopefully many, many folks who know what they are, but for those folks in, in the room, think of it as a way you have your, your data set, your input, you have a task, and then you have an outcome which you want to evaluate against a scoring function.

Uh, and this is kind of, uh, a little bit different, uh, comparing, say, traditional software development, uh, into working with AI systems because of its non-deterministic nature. Okay. So in this particular, uh, portion, what I'm going to be talking about here is creating what we call a golden data set.

So in this support application, I'll be kind of testing anecdotally, but I want to create a, a set of edge cases where I think this is really going to help give us at least an initial level of confidence to the business that what we're releasing out into production, you know, is, is sort of a full purpose.

We al there's always room for improvement, but I want to be functional. I, I want to just think, look, it's not just me releasing this application based on vibes. I have a concrete way of saying, okay, this is how it's, it's performed over time.

Um, to do this, again, we use kind of two main types of scoring functions. The first of which is deterministic. So I think a lot of folks have may have already started using this. Again, coming from traditional software engineering, uh, you know, unit tests, uh, I wouldn't say analogous, but they are quite similar in nature where, um, they're very easy to run, uh, cost-effective.

You're not actually using a model at this point. The secondary type, which is a little bit more sophisticated, which is then using an LLM as a job. So another AI system. And this is really helpful when it comes to systems where there's nuance which can't really be determined on deterministic systems alone.

So again, creating one to talk about, you know, branding style. Is this meeting, you know, customer satisfaction, uh, and so forth? So these most important thing is if you cannot write it in a deterministic way, you want to be using LLM as a judge where possible.

So, and why this is more important, again, it's just making sure that any change that we make is a safe to change, uh, as we progress this. Okay. So I'm going to pivot into, uh, the idea again. Let's take a look at the, uh, README file.

And then we're going to do, well, that's a dictionary. Where are you coming from?

Okay. Google C, skip checkout.

There we go. Okay. And then based on this here,

make demo, well, make setup, we should be fine. Okay. So one thing I'd like you to do as well is then to run the C data set command. So we do make C data set.

Okay. So what this does is going to upload our evaluation, uh, test cases into, into the Braintrust UI. So if I pivot into, uh, my data sets now, you'll see it's called Helper C data set. Uh, again, just for simplicity, I've created 10, um, inputs.

Um, I've also categorized them, um, around there. So you input what we expect and some metadata associated with that. So kind of the core structure to, uh, creating an evaluation.

So, um, as part of this as well, um, in the deterministic and non-deterministic, I've created some scoring functions, uh, which we'll be using as part of this. So it's available here.

Uh, and then, yep, scoring functions. So again, checking the category, uh, is the schema in place? Um,

is there an escalation reason when, when needed? Again, very easy to run, uh, and codify.

Uh, yeah, we have a bit of more sophistication with the, the customer rubric. So this is more of the LLM as a judge use case here.

Allright. Uh, then what we can do is then go back to

the README. So we don't need to do the demo. We already pushed that through. Uh, let's go ahead and then do, um, make eval. Okay. So this is then going to run an evaluation.

Uh, I'm going to say here, um, the C data as well. Uh, again, you don't necessarily have to put it into code, but, um, whether it's coming from a database. So in this case, I'll just use a flat file JSON for this, uh, just to give you some context.

So we are running this evaluation.

And we should have, if we go to the UI, it's a place for experiments.

So I'll just double check in here. Yep. So that experiment ran against all the JSONs associated with it.

So we should, you should get an output like this at least in the terminal. But if we go back to the UI, uh, again, you can, we're starting to track the, our application against the inputs and outputs for this particular data set.

Um,

and quite similar to the, um, the tracing view you would have seen from the online, uh, traces, you get to see a very similar view here, uh, and seeing how it works across, um, your experimentation as well.

Yeah. And I think, yeah, pointing out, uh, as we progress with the workshop, uh, we'll start to see how do we then improve it using the, the set of difference functions and track that across the UI.

If you need a bit more real estate as well, um, you can collapse the menu, uh, which does help.

Okay. Time. Perfect. Okay.

Um,

allright. Let's proceed with the next checkpoint, uh, around deploying and managing this. So, um, as I mentioned, I think a lot of folks, or at least anecdotally when I, when I've seen and spoken to customers is, again, things will work really well on my machine.

### Deployment

**Giran Moodley** [1:05:24]
Okay. Now checking that into code. I kind of want to take this into a place where, um, I can start to collaborate a little bit better. It's in a, there's a point of reference. Um, I've got versioning history.

You can start to identify, again, who's made what change. And you need a way to be able to bring these users together. So I think Oussama talked about, uh, our collaboration. Um, what's interesting as well is, again, changing the prompt on your machine and then trying to ship that code to a repository and then maybe somebody who's, let's say, a non-technical SME, a product manager perhaps, they want to update the prompts, they can't do it.

They have to tap you on the shoulder. Maybe that's happened to some of the folks in the room here today. Uh, I know it's happened to me, uh, a few times before. Um, and again, I understand it can get really frustrating.

So we actually just want a way to be able to pull that together. Really key thing, especially for those who work in very regulated industries, like reproducibility is a very big key thing. And I've worked in, you know, been a part of over a decade in, in both banking and, and capital markets.

I know obviously with, uh, you know, the regulatory out there, especially, uh, things likeright to be forgotten, understanding, you know, who's made a change, especially in a stress exit scenario, how can we put this into a new system?

This is really key to, to helping unlock that. And interesting when it comes to, you know, to identifying changes before you do that. So again, we just want to, we don't want to just be making changes, pushing out production and asking us what's happened.

We, we need to be able to kind of get that in place. Um, and so for us, again, uh, we're introducing some capabilities now. So what you've been running at the moment, you've been running the tools, you've been running the prompts, everything on your local machine.

What we want to do is then offload that capability into Braintrust. So when your application is running in a secure environment, it can refer to Braintrust to, uh, pull that information, uh, and then, uh, help with the part of execution, which again, follows the tracing mechanisms that we've done.

So, um, by default, when you're running the make commands, um, the runtime mode was set to kind of local. If you want to use managed mode, just use the prefix managed and whatever the remaining make commands, uh, will happen there as well.

So what I want to do then is, um, pivot into, um, the ID. So going back to make, uh, my README.

Okay.

It's a little tricky. I'll see.

Okay. So get checkout.

Okay. So just taking a look here. We'll do make setup. Allright. And the key thing I do want to emphasize at this point because we're managing this in, in Braintrust, use the setup, um, Braintrust, um, command here. So what that's going to do is going to package up those scoring functions as tools, as prompts, and push it onto the secured, uh, infrastructure.

So you should get an output like this. And what that would look like, uh, in the UI. So I'll just go back to overview to see. So on the left-hand side, um, if we take a look at, uh, prompts, you'll start to see the, the three prompts that we created as, as part of that, that workflow.

Um, give you an idea here. If we go to the, uh, the triage specialist. So, um, you can define a slug. So let's say, uh, an immutable ID, which you can refer to it in the code. Um, this can also be generated if needed.

Um, the prompt, um, is again, treated as code. Uh, we also use some interpolation there if you want to parameterize this, and I'll show what that looks like shortly.

If I go to the scoring function, uh, so let's click on scores. Uh, okay. It'll probably come up in a second. Um, uh, take a look at parameters. So I'm just going to take a look here. And I've created a parameter specifically just to simplify things, changing the baseline model.

So maybe just kind of a show of hands. Um, you know, these models get released so quickly. Has anybody had like a PM or let's say, uh, let's say a non-engineering SME say, "Hey, by the way, can I change, uh, the prompt?

Can I change the model or the prompt and see what that would look like?" Has that kind of happened to you folks? I think we've got a few hands here. Great. Okay. So the good thing with this now using the, the managed parameter is those non, uh, technical SMEs, they can come into Braintrust and change a prompt here, write a comment to say, "Look, um, let's use, um, a different model.

Let me use something like 4 mini." Uh, just say, "Testing a new model." I'm going to save this version here. Write the comment.

And what I'm going to do as well, just to give you an ending claim, I'm going to do is, you know, make setup Braintrust. So every time you change something, um, you can run this command, but I'm just doing it for brevity here.

Um, that's more to keep, um, this model in sync. So if I go to prompts, uh, you know, uh, that's kind of in sync. But if I take a look at, um, the, uh,

uh, sorry, the parameters in place.

Yeah, it should be there. And so what I want to do is if I do something like, um,

uh,

I think this is also runtime.

I'll unlock.

So in this case, by running the managed model, I'm just changing the course of the execution, uh, not to run the model locally, but to follow the path of what Braintrust has, uh, set for the model.

Okay. And, uh, if everything goes well,

see, see something just now.

You can see that I've changed the model here. So again, I didn't have to do any code changes. All I had to do was go into the UI, change the model I wanted to, any other parameter, run that, and have that, uh, use as a, uh, establish a new baseline for the evaluations.

Um, again, if you want to, uh, you can also run the, the demo scripts to push in the demo tickets. I'm just skipping that for the workshop today.

Um, I think I do see this with many of our different customers is saying, "Oh, you know." Um, there could be not necessarily a cause of concern. It's saying, "Hey, by the way, Braintrust is now having access control to these certain parameters."

Um, what I would say is Braintrust is not really intended to replace that rigor. You probably still want to use things like version control systems anyway to track that. What we're just saying is when it comes to operationalizing and making sure that other users are able to work on a shared system, this is a recommended path that we would take, uh, to help out with that.

So again, you would probably still have to have your prompts, your tooling, your parameters in a kind of centralized way, but then provide automation in place for you to synchronize that and, and work. And that's the best way we've seen customers take advantage of this.

### Online Scoring

**Giran Moodley** [1:13:58]
Okay. Uh, it's the next portion we're going to talk about online scoring. So now that we have those evaluations in, in place, um, what we're going to do is then apply those, um, scoring to actual live production logs that are coming through, uh, in the application.

So, okay, it's great that we've done our test cases. We, we've got some level of confidence that it's working, but again, there's no substitute for production data,right? We all know this. So what we're going to be doing is, uh, creating, uh, again, moving that logic, uh, into Braintrust and setting up what we call automations that would then track and, uh, evaluate this as logs are coming in, in, in real time.

Uh, worth pointing out when you start your journey, um, it's probably, especially if you're using, uh, LLM as a judge, you want to start with the, let's say, a higher sampling rate. So again, as logs are coming in, uh, where possible, you, you want to make sure you identify a baseline.

But again, there's a trade-off when these calls can be quite expensive, especially if you're using more sophisticated models and you need a higher rate of reasoning. At that point, when you do want to, uh, are happy with, um, the output, you really want to reduce that, that sampling rate down to 5% to 10%.

So again, you, you're managing your, your cost effectively. Uh, deterministic scores, again, they're cheap. Recommend running them all, all the time.

Sorry, what was that question?

Yes. So I'll bring that up in the UI. I'll show you. Yeah. Okay. So if I go back into the, uh, IDE, I'll go into README here.

Um, oh, so that's why I missed out the managed tools. So let's just say get checkout.

Okay. There. And as I mentioned, they all build upon each other. So skipping this is totally fine.

So get checkout. Allright. Now, if I do, um, make setup, everything should be fine. Make, uh, setup

Braintrust. Um,

so in this case, I'm actually taking the, the tools as well, um, for the production. Again, this is just to really help accelerate things where possible. Um, coming down to, um, if I hit refresh, um,right. So scores, the scoring functions that you saw earlier, uh, in code, they're now managed in Braintrust.

So you can see it's, it's available here. Um, the ones that I want to call out is the, the triage. So I've got an LLM, uh, as a judge, um, which has been applied. So put an output taking, uh, this.

And there's an automation rule in place. So route quality online. Um, so what I've done here, again, I've automated the setup, uh, as, as code. So to, to bootstrap the project. But again, uh, what we're able to do here is to say, look, depending if it's an individual span, we can run the execution or the entire trace.

And this is why metadata is so important because we, we may only want to trace maybe specific failures that might happen within the code. Uh, again, depending on the use case. Um, my sampling rate, as I mentioned, is set to, to 100, but again, for more expensive calls, we want to taper that as well.

So this is what the automation, uh, or how we would do that in, in Braintrust there today.

It's like packaging the scorers and No. So automations is more like the execution against, uh, incoming logs. So it's, um, with that.

Yeah. But I'm more holistically, I've applied automation to setting up this environment and scaffolding. Yeah. So that's just want to delineate that here.

Sorry, is the question? Yeah. What kind of things are you scoring? You don't have the latest data. Um, could you expand on that, please? Yeah. So the automations are like a judge,right? Yes, correct. Where are you having a judge or properties?

In If there's like no ground truth to run. Okay. Like how can you perform online scores if you don't have ground truth? Yeah. Yeah. Well, you probably want to take that as an edge case, push that into a dataset, identify it, and then move that back.

So that would be the approach that I would take for this, um, if you don't have any ground truth already,right? So this is why I said depending where you start, it's better to have some kind of data and then begin your flywheel around from that.

Oh, so if you have some data, how do you apply that to the LLM as a judge automations? Oh, that case. Um, we can probably put that into a dataset and then replay that through the playground. That's, that's going to be the way we would do that.

Yeah. Yeah.

Allright. Um, so we checked up online scoring. Cognitive at time. Okay. Now to the remediation portion. So hopefully seeing the delta or again, why we, we're here today. So hopefully this might help you, uh, with that particular question.

### Remediation

**Giran Moodley** [1:19:28]
So again, here's something that might happen as a, as a plausible input to our agentic system is, you know, customer, uh, user might say, "Hey, this isn'turgent, but our CFO can't export the invoices, um, before, uh, the board meeting."

The model says, "Look, hey, this looks, this looks okay for me." Someone says noturgent. Come see, come sigh. But the business is very different,right? This probably does need, uh, immediate attention. Uh, your CFO shows end of quarterly report that needs to be done.

And this is the difference between, uh, what we're doing here today is trying to identify, um, what is a proper failure mode and then remediate that where possible. Um, so in this case, again, I can run, um, um, this particular mode here.

I've got this in, in this, in this dataset.

So, uh, just to kind of give you a play of scene where we want to replay the failure, we want a specific, uh, evaluation against this, want to tighten the prompt, run it again and see, uh, what that looks like.

We probably want to do this against not just one particular test case, but across our entire test cases as well, just to see if it does work as intended and we have regressed on something else, uh, from that perspective.

Okay. So let's go ahead and have two, we have two separate branches for this. So I'll split out branches A. One's going to have the, the failure rate in mind. We're going to go to the UI and view that.

Um, and then we're going to talk about the, the remediation path, uh, there as well. Okay.

Actually,

so just go here.

So many product failures.

Okay. So, um, make setup.

Okay.

So we can even do

runtime. So runtime mode

managed.

Then we'll make,

uh, replay failure.

Okay. So I've got this set of five cases, which are, you know, the regression of failure modes in this case.

So that first one that you saw in the example ticket, yeah, that's, that's done as a JSON file here.

Okay.

So if I take a look here.

Okay. So if I take a look at the, the failure mode here, I can drill into this.

See the, uh, and you'll notice as well, now that we set up the online, uh, the managed tools as well as the online scoring, the trace becomes, um, even more sophisticated in the fact that we're executing this against the secure, uh, Braintrust, uh, environment.

So again, moving from local to, to managed.

Allright. I'm just going to go here to, uh, the make file. Sorry, package JSON.

Let's go to

here.

Manage

scenario here.

Wording

impact. Let's run an evaluation here. So I'm just doing an evaluation against a specific, um, scenario. Allright.

Okay.

Do you also want to show the monitor? It's about latency. Yeah. What do I just go here.

Okay.

Um, yep. So as we can see, as we're progressing with, uh, the application, the experiment, again, viewer just allows us to see the progress of our changes. Um, so you can see we've run the, the latest set. We notice some, you know, degradation here, which just allows us to track it a little bit better.

So the managed dataset that I, I had in mind, uh, the failure rates are being captured. And you can see, you know, I have the ability to compare it against existing, um, experiments to say, to track the progress and remediate where possible.

Okay.

So forgive me. The next thing is I'm going to go to the README and then proceed with the, uh, remediation.

Okay.

So do get checkout

remediation.

Make

just

Okay. So in this remediation, I've changed, um, the, the prompt which I've used. Um, so, uh, one way to, to view that is if I take a look at the, the prompt and the, uh, the change.

So

see,

so you'll see any, any differences there.

Just in place.

Okay. Coming down to the, uh, README file here.

Let's say I do, I want to say,

say code X here.

Let's give this a say.

The, the

Braintrust command

is contents.

Yeah, there's a specific flag I think. I don't know if you want to just double-check, but that

Yeah, that's it. If, if it fits.

So if you want to just put that if, if exists, replace, um, ENV in the chat. Yeah. Yeah.

Do you like to test it? Huh? Do you like to test it? Yeah. No, no, it's fine.

So yeah, just folks, uh, if you want to, in the part of the remediation script, if you set the environment variable Braintrust underscore if underscore exists, you set that to replace, um, it's going to push in the, the updated changes into the, into the environment.

So then if I take Oh, sorry for that.

Okay. Um, and then you'll see as well, now that we've updated the problem for your code and pushed it up, uh, it's, um, it's available here as well.

Um, so as well in the UI, um, again, part of the operation operationalization, uh, we can see, you know, who's changed what, but actually what has been changed to that particular prompt, uh, as in this, in this case.

Take a look here. So including tool facts. Bring that out. Okay. So coming back to here, let's say we didn't do a, um,

run this evaluation again.

So we made the change to the prompt, and now we're running, um, the remediated version to see how that performs.

Perfect. Okay. So that's around the experiment here with the new changes, hopefully. And I'll pivot into the, the UI. I'm going to go to experiments here. And it's barely there, but you can just see, you know, kind of tape it back up.

Just, you know, any improvement going up is that improvement. But yeah, just to give you an idea here that, um, you know, now that we've done the evaluation, uh, actually what I can do is, um, do a diff.

Uh,

and we can do some do a, a comparison in the delta here.

Yeah. So yep, that's come up, uh, and improved over time. Um, uh, which way, which way, uh, the intended outcome. Yeah.

Okay. So

### Summary

**Giran Moodley** [1:33:09]
I think we're approaching the end of the, the content. Um, so I know it's been a number of steps, so I really want to thank you for your time and attention to kind of walk through that. Uh, as mentioned, the, the artifacts are public.

We've got the cheat sheet there. We've got the Slack channel to help if you have any, any questions. But, uh, hopefully just to give you a summary of what you've, uh, accomplished today, uh, in, in this order is you went from taking that single shot prompt into building a five-stage, uh, AI agentic workflow using, uh, tool calls.

What we're able to then do is then inspect how this works pretty much into diving into this by adding Braintrust tracing, making sure everything is recorded. Um, we also then want to talk about, you know, how do we then evaluate the system from a, um, you know, when it's not online, when it's something new, uh, creating those, those, uh, effectively golden set with those test cases which we want to execute against.

We then deployed, uh, that, uh, those managed prompts, those tools and parameters into the Braintrust secure architecture, uh, to be able to use. And we've also added online scoring to then evaluate the system, uh, as it unfolds. And then we picked a particular, uh, production failure.

We looked at the trace, we modified the prompt in our, uh, case, and we saw the delta there. And running the evaluation again, we saw it, uh, achieve, uh, back up to, to where it needed to be. And in this case, completing the full evaluation of, you know, building, observing it, deploying it, and taking it to that moving forward.

Um, so yeah, just to kind of, uh, call it, um, and bring it home. So again, hopefully this is not uncommon, but again, what might work in production is not really going to work in prototype. Uh, we really need to break this down, identify the failure modes, and, and move forward.

And that's where, again, explicit stages become really, really important,right? Um, again, this does introduce more, uh, areas of, of where things could go wrong, but it's easier to debug if that's the case. Um, again, there's no substitute for diving into the code and tracking everything.

So I would say it's, it's observability is table stakes at this point. If you've got a production AI application and you're not tracing it, you need to go back to the drawing board and get that done operationally. And hopefully, again, we can show you how to, to be able to do that, um, using Braintrust.

Um, again, no substitution for production logs, but better to start somewhere from nowhere. If you have an idea of what an issue might happen, these are your perfect ways to, to set up an evaluation. So using those, uh, failure modes as, as your test cases.

And as I mentioned, this is a continuous process,right? Nothing's ever done. If you've ever worked in agile development, constant feedback is, is important. And again, we're bringing this, this operation model, but with, uh, a newer surface of, of, of operating.

Um, and yeah, just hopefully bring it home to your teams here today. So, um, you know, my encouragement to you is to, if this is something of, of, of interest, is pick something that's already operational today. It doesn't have to be the entire suite.

Maybe start off with something that's maybe a bit more, um, uh, I guess more critical that you really want to improve, um, your operational modes, add the tracing, collect your edge cases from that, that mode, um, build those deterministic scores, and then route everything back as possible.

So again, the faster feedback loop that you have, uh, you have more insight, and the more that you can again improve the overall, uh, delivery and operations of, of your system.

Um, yep. And just to kind of call to action. So again, I know we've thrown a lot of content at you. It's, uh, we'll obviously try to get better feedback. We'll try to put some tables in place. So I appreciate everyone's kind of juggling everything to do this, but, um, as mentioned, you have to start somewhere.

Let's just try to accelerate you. Uh, we have a list of documentation that's provided, um, that's, uh, you can also use our AI agent on the agent to, to search if you have any questions. But we also have a cookbook available.

So I tend to throw that cookbook directly into, you know, cursor, codex, and whatever and say, you know, based on this, take the SDK, uh, and start tracing my application. It does it pretty effective. We even do have, uh, we've actually announced a CLI, um, for our the Braintrust, um, application.

So that allows you to even do things as auto instrumentation. I'm just going to plug my colleague Eric, who's doing some fanatic work, and please check out his booth because it's, it's, it's amazing. Um, and again, if you this is something that's interesting to you and you want to explore more, then, you know, please reach out to your account team at Braintrust.

Again, we're happy to support where possible. Uh, and if you're on Discord, you know, feel free to, to join. I'm happy to answer your questions there, uh, from that. And, uh, yeah, again, really want to thank you for your time, your attention, energy.

I know it's a really sunny day and I don't want to keep people in here. I want you to get some fresh air, but, uh, yeah, just on behalf of Braintrust, Trainline and myself, like, thank you so much for your time and attention.

It's, it's been an honor, and look forward to seeing you out there tracing and, and getting value from delivering AI in production. Thank you.

---

This library is powered by PodHood (https://podhood.com), the podcast website platform.