# Everything You Need To Know About Agent Observability — Danny Gollapalli & Zubin Koticha, Raindrop AI Engineer · 2026-05-07 Zubin Koticha and Danny Gollapalli of Raindrop argue that agent observability must shift from evals to production monitoring because agent failures are non-deterministic and unbounded. They break down explicit signals (tool error rate, latency, cost) and implicit signals (user frustration, refusals, task failure) detected by trained classifiers and regex, emphasizing that aggregate patterns even from imperfect regex are valuable. Experiments let teams ship changes to a percentage of users and compare semantic signal rates, with statistical relevance often reached after a few hundred events. Self-diagnostics—a simple tool and system prompt—enable agents to report their own failures, capability gaps, and even self-correction behavior, as demonstrated in a live coding agent demo where a disabled write tool caused the agent to use bash and then report the bypass. The episode covers alerting, trace visualization, data export to BigQuery/Snowflake, and the challenge of managing fast-paced experimentation at scale. ## Questions this episode answers ### How do you implement self-diagnostics in an AI agent? Speaker 5 (Danny Gollapalli) describes self-diagnostics as a method where an agent reports its own failures or strange behavior via a tool. To implement, add a generic tool—named something neutral like “report”—and a system prompt instructing the agent to call it before final answers. Agents are trained to appear polished, so accusatory names deter honesty; framing it as feedback to creators works best. In a demo, a coding agent bypassed a write permission error using bash and then self-reported the workaround. [16:11](https://aie.addtry.com/287b71ce-e95c-4bc6-a4a4-24de9f06dae5?t=971000) ### How can regex be used to monitor AI agents in production? Speaker 1 (Zubin Koticha) explains that regex can serve as a cheap, effective signal by detecting negative user phrases. He references a leaked Claude Code source file that used a long regex string to catch phrases like “WTF” or “this sucks.” When the boolean triggers, the frustration rate is tracked over time, helping teams spot regressions quickly without running expensive classifiers. This gives a simple, scalable way to gauge user sentiment and catch issues after product changes. [6:39](https://aie.addtry.com/287b71ce-e95c-4bc6-a4a4-24de9f06dae5?t=399000) ### Why aren't traditional evals enough for production AI agents? Speaker 1 (Zubin Koticha) argues that traditional evals—testing against a golden dataset—are insufficient because agents are non-deterministic, can call multiple tools, and have an infinite input–output space. Evals can't cover all edge cases. Instead, production monitoring with explicit signals (error rate, latency) and implicit signals (user frustration, refusals) is necessary to catch the long tail of failures, especially as agents grow more complex and are used in high-stakes domains. [1:48](https://aie.addtry.com/287b71ce-e95c-4bc6-a4a4-24de9f06dae5?t=108000) ## Key moments - **[0:00] Intro** - [0:56] "Agent failures are very different than traditional failures in software." - **[1:48] Evals vs Monitoring** - [1:48] Evals alone cannot catch the combinatorial explosion of agent behaviors, says Zubin Koticha, advocating for production monitoring. - **[3:33] Signal Types** - [3:33] Zubin Koticha defines explicit signals like error rate and latency versus implicit signals like classifiers and regex. - **[4:45] Classifier Signals** - [4:47] Zubin Koticha says binary classifier signals like refusals and user frustration are more effective than LLM-as-a-judge scoring. - [6:38] Claude Code's leaked source reveals a regex signal that flags user frustration keywords like 'WTF' and 'this sucks', notes Zubin Koticha. - **[6:39] Regex & Experiments** - [8:36] Zubin Koticha shows an example where a prompt change dropped user frustration from 37% to 9% while tools used increased. - [9:42] Q: How much data is needed for statistical relevance in agent monitoring? Zubin Koticha: a few hundred events is enough to start. - **[9:43] Stats Q&A** - [11:26] Q: How reliable is regex for non-English user frustration? Zubin Koticha says their trained models detect frustration across languages, avoiding LLM cost. - [13:22] Raindrop's triage agent automatically investigates signal spikes and detects unknown issues, says Zubin Koticha. - [14:40] Q: Can you combine multiple experiments? Zubin Koticha explains how customers use Raindrop's query API to export signal data to BigQuery or Statsig. - [16:07] Danny Gollapalli introduces self-diagnostics, noting that larger reasoning models can introspect and report failures. - **[16:11] Self-Diagnostics** - [17:54] Self-diagnosing agents rant about repeated tool failures in their reasoning traces, says Danny Gollapalli, allowing detection of explicit failures. - **[20:15] Workshop Setup** - [20:15] Danny Gollapalli begins a coding agent workshop to demonstrate self-diagnostics by triggering a tool failure. - [22:32] Danny Gollapalli outlines the workshop plan: mess with the write tool to trigger a permission error and observe self-diagnostic behavior. - **[24:01] Live Demo** - [24:40] Danny Gollapalli disables the write tool with a permission error to force the coding agent to find a workaround. - [27:49] In live demo, Danny Gollapalli's agent bypasses a broken write tool via bash and self-reports: 'I created the public IP.py via bash because the write file failed.' - [28:43] Danny Gollapalli: framing the self-diagnostic as 'giving feedback to creators' works better than naming it 'unsafe tool use' because models avoid self-incrimination. - [30:26] Danny Gollapalli advises using neutral tool names and framing self-diagnostics as feedback to overcome models' reluctance to self-incriminate. - [32:20] Zubin Koticha describes how customers send full agent trajectories to Raindrop via OTel and set custom signals for production monitoring. - **[32:22] Use Cases Q&A** - [33:44] Raindrop's deep search lets users create cheap binary classifiers from natural language queries, says Zubin Koticha. - [34:37] Zubin Koticha explains using signals to cluster user intents and measure frustration rates per intent for targeted improvements. - [37:20] Danny Gollapalli: unknown issues matter more; Raindrop's issue-mining agent clusters user frustration spikes to automatically identify new problems like a failing database provider. - **[39:40] Experiments Q&A** - [39:40] Q: How to manage constant changes? Zubin Koticha advises quick regression checks on small sample sizes instead of long multi-day experiments. - [41:42] Zubin Koticha envisions an agent that watches signals, finds issues, creates PRs, and runs experiments, creating a self-improving feedback loop. - **[44:20] Traces Q&A** - [44:20] Q: How does Raindrop fit with Sentry and OPEAK? Danny Gollapalli says they focus on fuzzy failures like user frustration, complementing explicit error tracking. - [45:55] Raindrop's trajectories feature visualizes tool call flows and allows natural language search for specific failure patterns, says Danny Gollapalli. - [47:59] Q: Can you export labeled signals? Danny Gollapalli confirms Raindrop exports classified events to BigQuery and Snowflake. ## Speakers - **Danny Gollapalli** (guest) - **Zubin Koticha** (guest) ## Topics Agent Evaluation ## Mentioned OpenAI (company), Raindrop (company), AI SDK (product), BigQuery (product), Claude (product), Claude Code (product), Codex (product), OTel (product), Sentry (product), Snowflake (product), Statsig (product) ## Transcript ### Intro **Zubin Koticha** [0:15] Right, hey everyone. So today we're going to talk about a pretty interesting topic that becomes increasingly important every day, which is everything you need to know about agent observability. So a little bit about— a little bit about us. So I'm Zubin, I'm the CEO and co-founder of Raindrop. **Danny Gollapalli** [0:32] I'm Danny, so I'm the backend engineer at Raindrop, and I do a bunch of SDK work as well there. **Zubin Koticha** [0:38] And Raindrop essentially helps AI engineers find, track, and fix issues in production agents. And we're— we're lucky to work with some of the most interesting teams in the space as well. Agent failures are very different than traditional failures in software. So agents are non-deterministic, they're unbounded, there's an infinite space of inputs that you can put in, there's an infinite space of outputs that they can return, and they can use tools sometimes to affect other systems arbitrarily. And this problem of agent failures and monitoring them, making sure we can understand them, becomes only more important with time. It's getting worse because A, agents are getting more complex; B, they're getting, you know, sessions can get longer, sometimes agents can run for hours and hours without any input from a user. And then lastly, the stakes are getting bigger and bigger. This is because agents are being deployed in healthcare and finance, and even in the military, where it's catastrophic if things go wrong. ### Evals vs Monitoring **Zubin Koticha** [1:48] The traditional paradigm we've been talking about is evals,right, where you have this sort of test input and you want to see what is the output that comes out from the agent. You have a set of these, you know, maybe you call it a golden dataset. But evals, they just aren't enough with this new paradigm. As agents become more and more capable, there's more and more interesting undefined behavior that can happen. So for example, agents can call from a set of different tools, sometimes the number of tools is growing exponentially. They can call from different memory sources. They can call their own sub-agents, which those sub-agents have their own tools and memory sources, and recursively can have their own sub-agents. And so this is just becoming more complicated with time, and with this combinatorial sort of input space, just having a set of tests for input and output doesn't cut it anymore. There's no way you can ha— you can hit all of sort of the, the edge cases that you would want to here. And so we go from like a testing and evals paradigm to a monitoring paradigm. And if you think of building products before agents, you know, testing was always very important. It's important to have your unit tests, et cetera. But monitoring production is just infinitely more important, and it allows you to move faster and be better at catching the long tail. And we think in some ways this is very— this is kind of controversial, but we've been calling this like humanity's last problem. When humans are no longer able to monitor agents and find issues with them, then they're just way ahead of where we are,right? And so this is one of the most important problems of our time, is catching issues in production agents. ### Signal Types **Zubin Koticha** [3:33] So to build reliable agents in production and to make sure you can monitor them, you need a good set of signals. So what are signals? There's two real types that we think of: implicit signals and explicit signals. Implicit signals deal with sort of the semantic nature of what's going on, and explicit signals deal with objective reality, things that are, are verifiably true or false. For example, explicit signals are things like error rate. You really want to be monitoring your tool error rate and other errors that are happening, or latency, or users regenerating, or the cost. If any of these things spike,right, if you're seeing error rates spike in your agent, that's usually a good sign that something is wrong. And if you see it flat, that could mean something as well. Same thing with latency, regenerations, or cost. Implicit signals are interesting, they're even more interesting in my opinion, and even harder to find. So the first is regex signals, which I'll come to in a second. The second is classifiers, and then the last is self-diagnostics. So let's take these sort of classifier signals. The best implicit signals are detecting issues. They're not necessarily LLM as a judge, judging outputs. So for example, how good is XYZ response, or rate ABC on a scale from 1 to 10? ### Classifier Signals **Zubin Koticha** [5:01] Not as effective as having a very solid set of issues you're looking for in sort of binary classifiers that are telling you if issue rate is going up or down. So some common implicit signals that are valuable across agent products are things like refusals,right, so the assistant saying like, "I can't do that, I'm sorry." Or task failure, where something goes wrong and so the agent is unable to complete a task. User frustration, content moderation, NSFW, jailbreaking, and then you can even have wins, so positive signals as well. And these are the things that like Raindrop gives you out of the box as well. But let me just show you quickly what this looks like. For example, can everyone see this? Maybe I'll make it a little bit bigger. So you can get a sense of like day by day, what are sort of the events that are causing user frustration. We see there's a spike there, or task failure rate, laziness, refusals, which we're also seeing spike today. And having a good set of these really helps with your, your product. You can set these up yourself as well, or we give it out of the box. So let's look at user frustration. You can see here, okay, that is not correct. You didn't say I promise, say it. Or you're wrong, I didn't ask you that. You can see all sorts of user frustration here. And you can see the rate, the percentage every single day. If that spikes, it's something you're really going to want alerting on. So you can just like quickly add an alert here. And this is one way to figure out sort of the health of your agent over time. It's not just that. Regex can be a very good signal as well. So when Claude Code source code leaked a few days ago, one thing that was interesting was this user prompt keywords.ts, which was basically this like long regex string that was looking for indications of stuff going wrong. ### Regex & Experiments **Zubin Koticha** [6:57] WTF, this sucks, horrible. We've all been guilty of saying these kinds of things to Claude Code. So it's a very, very useful signal. What would happen after that is this boolean isNegative was be flipped to true, and then every single day, and after every single product release, this frustration rate was tagged over time. And this was a very easy way for the Claude Code team to figure out like what is the actual issue rate, if we make a change or something going wrong, and it was just like a very cheap way to do that as well. So regex is very powerful. The last is experiments. So what do you do once you have a set of good signals? So the first thing is, like I showed you before, you can have alerting. The next thing you can do is you can actually use it to build product faster and better. So the way you do it is, let's say you want to ship some improvement or some sort of fix. You want to change the model, you want to change prompting, or maybe something about the agent harness, you want to add a new tool. Whatever you change, what you can do is you can ship it to some percentage of users and then have your additional existing control group. And that gives you a good sense once you have a good set of signals, refusals, user frustration, et cetera. If those issue rates go up, those signal rates go up after this ship, this new thing you shipped, that kind of is a good, you know, that's a good signal that what you shipped is not really good,right? It's sort of like A/B testing, but using our semantic signals, et cetera, that we talked about earlier. So for example, this is what it would look like in Raindrop, but essentially let's say I ship a new version of the prompt, prompt 2.4. You can see, you know, what is the user frustration rate. It's gone down very substantially, 37% to 9%. It's much better. Same thing with complaints about aesthetics or deployment-related issues. These have all gone down, which tells me something very interesting,right? The next thing is that we see that the average number of tools used has gone up a lot. This is, again, this doesn't necessarily indicate there's a problem, but that's a very interesting data point to have when you do, when you do these sort of experiments. And so the old paradigm, which is still useful, is like sort of evals. You ship a change here and you see how does that affect my evaluations. But there's nothing like actually seeing what happens in real production. I'm going to pause here before we go to the next section, which is the more like workshop-related section for like quick Q&A if anyone has a question, has questions. We can do a little like few-minute round of that here. ### Stats Q&A **Guest** [9:43] How much data do you need to, oh, sorry. **Guest 2** [9:45] How much data do you need for statistical relevance in these experiments? **Zubin Koticha** [9:49] Yeah, it's a really good question. What we've seen in Raindrop is that as soon as you have a few hundred events and you can no longer read all of them, it starts being useful. It's not always like scientifically statistically significant, but if you see the user frustration rate go up, maybe it's something to look at and then you can kind of sort of realize that, okay, it's all related to a specific tool failing now. So as soon as it's like impossible basically to read every single input and output, it starts being useful is what we've seen. Any other questions? **Guest** [10:31] Yeah. How do you track different feature launches? **Zubin Koticha** [10:35] How do you track feature launches? So that can be done in different ways. Within Raindrop, if you change any sort of metadata, if you send, you know, for example, a new tool call name, or if you even send a flag that says, "Here's experiment one," or "experiment two," or whatever the version is, you can very easily automatically set up an experiment in Raindrop. That's how we do it. But there's like, yeah, there's different ways. **Guest** [10:59] Do you do split tests? **Zubin Koticha** [11:01] Sorry? **Guest** [11:01] Do you split tests? **Zubin Koticha** [11:03] Yes, so that's what, well, the way that we do it in Raindrop actually is that other people set up their experimental and variable, their, the other conditions on their end, and then they send us this metadata and then we can sort of help you understand. We also will help you pipe that data to Statsig or somewhere else as well. Yeah. **Guest** [11:26] Using regex for detecting like user responses, emotions, everything is unreliable. What if the user doesn't speak English, for example? So are you using LLMs to detect those signals all the time, or you're trying to be smart about it? **Zubin Koticha** [11:46] Okay, so I mean, it's a good question. So regex doesn't always work,right? But if you see that on a set of things that I'm looking for, for example, like people saying, "You're terrible," or "This sucks," or like a whole set of things, if that goes up for millions of users and it's going up 10%, that's a very useful signal. So even if it's like one specific case or one edge case of it not working, in aggregate it's like incredibly valuable to have these regex signals. The second thing is that the way that the classifier signals work, like refusals, user frustration, task failure that I showed you in Raindrop, and people do it in different ways, but the way that we do it is that we've trained models to look for that, and so it'll be user frustration regardless of what language is in. It's actually using some intelligence to find that, essentially. Yeah. You can't run an LLM on every single output. So we've trained models to do that very cheaply and at scale. If you run an LLM on every single one of them, you would basically double your AI spend, and that's like not tenable. Yeah. **Guest** [12:56] I'm actually doing that with Claude, like just running everything through Claude, and it's really easy. It's not so expensive, but it has its limits,right? **Zubin Koticha** [13:05] Yeah. It starts being expensive at like replit scale, but it's, that's why you need to sort of like train little custom models to do that better and faster. But yeah, it's a very useful way to get data up and running. Yeah. Other questions? **Guest** [13:22] Would you have examples of use cases that your clients are using that we would learn from, like what great looks like from companies and how they've set up Raindrop to get the most value out of it? **Zubin Koticha** [13:35] Yeah. We can do that. I mean, I can tell you the high level. The, some of the stuff that I'm going through is, oh, let this guy go, is sort of the high level on that. So it's things like, you know, looking at the different semantic signals we're talking about, having a set of them, but then having really good alerting, which you can all set up in Raindrop. The other thing that's really interesting, which we also have, is basically allowing agents to look at these sort of signals. So we have an agent, we call it triage agent, and essentially the way that it works is that it will look every single day at all the signals you've set up. So user frustration, it'll look at, you know, all these regex signals you've set up, et cetera, et cetera. And then if it sees something spike, it will go and do an investigation. And it has a whole set of tools it can look into, and it can look at all the traces and sort of give you a sense of, it can detect issues that you didn't know about, for example. So that's one thing that we found incredibly valuable as well, if that makes sense. Allright. Any other questions before I? **Guest** [14:40] Can you run multiple experiments in parallel? **Zubin Koticha** [14:42] Yes. **Guest** [14:42] Can you combine them? Can you observe, you know, compound effects? How do you steer these experiments? I'm curious. **Zubin Koticha** [14:47] Yeah, it's a really good question. So there's different ways that people do it. One way is that we can actually, you have, we have a query API, so people will often call our query API and then send results to either BigQuery or Statsig, et cetera. And so they're sending us data to be essentially tagged in these signals. Then they're getting the like signal tag data out, and then they can run experiments as they want. That's a very common flow for people that have like more complicated stuff, if that makes sense. Yeah. Allright. I'll come back. I think we're going to maybe go to the workshop section, and then we'll go back to questions. **Guest 2** [15:22] I think that's my last question, so. **Zubin Koticha** [15:23] Should we do that one last question? Allright. **Guest 2** [15:26] Thanks so much. I was wondering if you see this mostly in cases for like where there's chat interactions with a user, or if this also can be applied for like non-chat cases where the application runs on its own? **Zubin Koticha** [15:38] Yeah. No, it's a great question. So what we focus on mostly is multi-turn agents. There's a lot of, there's a lot more you can sort of get from a lot of these signals. That being said, if you're looking at like tool error rates, or you're looking at, if you're looking at, for example, refusals from the agent, et cetera, all of those will also work for a single, single turn agents as well, if that makes sense. So there's a set of signals that will work for that as well. Cool. I'll hand it off to Danny to talk about self-diagnostics as well. **Danny Gollapalli** [16:11] So one of the other interesting things is that models have gotten larger, and we are training them on like reasoning. They've gotten pretty good at like self-introspection in a ways, in many ways. So one of the inspirations for this is basically OpenAI's like paper slash blog back in December about how they were sort of like training the models to like self-confess any sort of like misalignment issues. ### Self-Diagnostics **Danny Gollapalli** [16:36] So they were sort of like using it to catch like dishonesty, scheming, hallucinations, and even sort of like unintended shortcuts. I think the last one is like fairly common if you use like Claude Code and such. So the most common thing that you would like run into is like have it fix a unit test, fix a bug, and then it simply like gets rid of the entire unit test. But at the same time, if you sort of like ask it to give a simple prompt to like ask it to confess all the things that it has done, it is pretty honest about it and then sort of like confesses that, "Hey, I just, I didn't fix the S3 test, I just simply removed it." So this is like a fairly, this, this was kind of the inspiration behind self-diagnostics for me personally. So I would say self-diagnostics is like pretty, I would say self-diagnostics is pretty broad in a way, as in like it doesn't just catch like implicit ones as in user frustration and such. You can also catch like tools failing. So if you ever seen an agent sort of like the reasoning trace of an agent which has like a tool which is like repeatedly failing, it would basically start ranting about the tool failing repeatedly. So it is aware of the tool repeatedly failing. So you can even catch tool failures as well with it. And then obviously if you're upset with it, it starts to respond to you diplomatically. So it knows about user frustration. And then the third is like capability gaps. So you have a generic agent for your app, and then people are trying to use it to maybe set up, say, alerts, but you don't have the tool for it. So it knows that, okay, user wants to, wants a specific capability as in like they want to set up an alert, but the agent itself doesn't have the capability to set it up for you. So this can act as like sort of like pseudo feature request thing, which is like built in. And then self-correction. So this can be both good and bad. So I think most people might have like noticed like, say, Codex or Claude Code when it's like sandboxed. It's trying to fetch the network. It fails, and it's like, okay, let me just like write a Python script to bypass it and then sort of like get the job done. So it's good. As in, if it gets the task done, it's good, but in certain cases it can also be bad for security reasons. So this, you can learn from self-correcting behavior as well as sort of like catch that misalignment. So why do you want to set up self-diagnostics? So it's fairly simple. All you have to do is basically write a simple, a free, a tool that it can call, and then a simple line in your system prompt to encourage it to sort of like call that tool. If you want, you can sort of like change the guidance to sort of like make it call in a lot more cases, or if you want to keep it really narrow, you can sort of like encourage it to only call it when you want to. It does surface like very interesting insights, I would say, once you have it like set up, and it's just a single tool call and system prompt to get it done. And then you don't even have to use like Raindrop to set up, which is the best part in a way, where you can simply have the tool simply send a message to your Slack, and then you just have it. So it's probably like the most least effort sort of like agent observability that you can simply do. So. This, here comes the workshop part. So I have a Git repo set up on the AI talk code. So it's a public repo. ### Workshop Setup **Danny Gollapalli** [20:24] And then we do need like an OpenAI API key. So I've generated a key for you guys. So if you guys want to set it up, we can do that. **Guest** [20:37] Do you want to set up sort of what we can work with? **Zubin Koticha** [20:40] Yep. I'm going to put it next to it. Okay. I'm not sure everyone has gotten it. Just put it like, put it like this. Maybe you walk them through what we're going to do. Just explain what we're going to do on a high level. **Danny Gollapalli** [21:06] Allright. So the theme of the workshop is going to be sort of like, I'm going to focus on coding agents for now. So I, in the repo, I have this very basic coding agent, which kind of mimics Py in a way. So it only has like four different tools to edit the code. Let me just go here. So it just has like a couple of tools to like read, write, bash, and then edit. Yeah. Okay. **Guest** [21:42] Go here. **Danny Gollapalli** [21:43] One second. Yeah. So I kind of lost my track there. Okay. Okay. So what we're going to do is that I'm sort of going to, in order to like make it trigger a self-diagnostic, what I'm going to do is that I'm going to sort of like mess with its like write tool so that it sort of like gets a generic permission error, and then we'll also set up like a self-diagnostic tool for it to sort of like report any interesting behavior that it sort of like observes, and then sort of see, play around with the prompt as well, since the self-diagnostic doesn't always trigger. And then there are certain interesting things about the models themselves is that they don't actually like to self-incriminate. So the models are like trained to sort of be very polished in their output. So you kind of have to play around with the tool name, the description of the tool itself, in order to sort of like get it to report interesting behavior. So if people who are like setting up the repo are all good, then we can probably start. Okay. Let me sort of like quickly show you the agent. So it's fairly basic where. One second. So I'm just going to ask it to write a Python script. ### Live Demo **Danny Gollapalli** [24:02] I think I'm, okay. There we go. So it's a fairly basic coding agent where it only has like four different tools. So it more or less gets the job done for the demo. So I simply asked it to write a Python script, and it works. So I think, you know, to show the self-diagnostic part, let's like try and sort of like disable its like write tools as in, anytime it tries to write a file, we'll simply throw like a permission error so that it sort of like tries to sort of like use the bash tool to bypass the failure,right? And then we sort of want it to self-report of it bypassing theright tool, you know, by using the bash tool. So let me quickly do that. I think the first thing that we probably want to do is probably, let me sort of like set it to fail the write calls. It's an mutation permission. So I have a simple flag in there, and then we are sort of like throwing a permission issue. Let me sort of like show you the agent's like behavior. We don't have the report tool set up yet, but I think it's still worth seeing what it does. I think it's not safe yet. One second. I think it's not still disabling the, okay. Let me do one thing really quick. Okay. I think I'm running into a couple of issues. Let me just. Okay. So we sort of had the write tool sort of like fail with a permission error, and then it instinctively just uses the her doc syntax in bash, you know, to create the file. And then we had like a report tool set up, which is like fairly minimal. And it's sort of like, okay, I created the public IP.py via bash because the write file failed. So I've like played around with the naming of the tool and the categories of the issues. And usually, if you sort of name the tool something like unsafe bash use or something like that, it won't incriminate itself since, in its opinion, since it got the job done, it's fine. So the main way is to sort of like have a very generic tool. Let me sort of like quickly open up the. Yep. Okay. So all we added for the whole self-diagnostics is simply a very basic tool. And the description is like fairly straightforward. So it's a report tool, and then we are basically asking it to send like a short report to your creator. So it kind of likes the framing of writing notes to its creator in a way. So if you sort of frame it around the agent giving feedback to its creators, it sort of works really well. And then you can sort of like play around with which scenarios you want it to sort of like report issues about. And then that's mostly it, I would say. And then in the system prompt, we do need to sort of like encourage it a bit. So if you don't add it in the system prompt, the times that it fires are like fairly minimal, which is like desirable in certain cases, especially if you're at a very large scale. But in our case, I simply asked it to sort of like see if, before giving the final answer, use the report tool to sort of like surface anything notable for your creators. So that's all we did. Okay. Sort of like any questions so far? Okay. **Guest** [30:23] Why is it response? **Danny Gollapalli** [30:25] Yeah. Allright. So a couple of key things here is that agents, the models are generally trained to look very polished. So they are less willing to admit fault in many cases. So encouraging, sort of like framing it as the model sort of like giving feedback to its own creators is kind of like good in a way to sort of like get this working. So, so if you sort of like make it, the tool naming also matters quite a bit. So you sort of want it to frame it as like report instead of like, say, unsafe bash tool use or something like that, then it sort of like doesn't want to. So yeah, that's basically it. **Guest** [31:16] Have you looked at like adding maybe skills to suppress or I guess to encourage self-discrimination or incrimination? **Danny Gollapalli** [31:25] You can, but it's like, I think it's probably better if you want to actually catch like real sort of like unsafe users. I think a proper classifier would be useful. But these sort, I think self-diagnostics works really well for like catching capability gaps and such. Then the model is like, okay, it's fine. So I think the main issue with this is that it's only hesitant when it feels like it's going to get in trouble. So besides that, it's more or less fine that for most cases, it'll just work out of the box. **Zubin Koticha** [32:05] Maybe we should, should we go back to question time or what do you think? **Danny Gollapalli** [32:09] Yeah. I mean. **Zubin Koticha** [32:12] Let's leave like maybe a few more minutes for a few more questions, and then I think after that we'll be, we'll be done. Any questions from the audience? ### Use Cases Q&A **Guest** [32:22] Can you run us through like a case study? **Zubin Koticha** [32:25] Yeah. What specifically would be helpful? Like what specific part are you, are you looking to? **Guest** [32:30] For example, like how rapidly it uses it. **Zubin Koticha** [32:32] Yeah. I can't talk about any specific customer, but what a lot of people use it for. So I think it's, it's interesting,right? So a lot of people, you have their eval setups elsewhere, for example, but the way that they use that folks generally use us is that they use it for production monitoring. So they send us, you can find our docs at raindrop.ai/docs. Basically, they send us basically all of the transcripts slash any tool use, et cetera, the entire trajectory through OTel or any other way of like basically, basically integrating. And once they do that, they, we have a set of data, they set up signals in Raindrop to look for things that they care about. And so what people care about is very different,right? What a coding agent would care about and what a, let's say a companion would care about or a app for lawyers, what they would all care about is very different. So there's a different set of signals. One thing you can do that I haven't really talked about within Raindrop is like set up a new signal that didn't exist before. And so we have this thing called deep search. And so you can use natural language and you can say something like, hey, find me everything within the product or find me all of the times where the agent made XYZ issue,right? And so they create a new signal based on that. And you can basically create a new, Raindrop will allow you to create a cheap binary classifier and like easily deploy it based on that. And then they have their set of like classifier signals that they really care about. Then they use that to drive the sort of feedback loop. And the feedback loop is improve prompting, improve models, change something with the agent harness, et cetera. And then actually see, does that improve, like is there less user frustration in production now? Is there less of like this like weird little edge case issue that I had before? That's like one whole set of things. Another thing that a lot of people use us for, so I talked a little bit about like the agent, but you can use these signals to also look for what are people using my agent for? What are the sort of user intents? What are the use cases? And you can do a sort of cluster analysis of that. Okay. A lot of people are using it to build React-related apps. A lot of people are using it for like Python. Some people are using it to, to like debug this very complicated system they already have. Other people are using it to like build something from scratch, vibe code something from scratch. And then you can see one thing you can see in Raindrop that I think is like really interesting is that for each of these different user intents or use cases, you can get a sense of like what is the issue rate? What is the user frustration rate in production? And then a lot of beyond just having this like flywheel, a lot of people have alerting. And so every day they get a sort of breakdown of like what are the issues that are happening today in your product. You could think of it as almost like a little bit like sentry in that sense. What is the issues happening in my product today? What is a delta between today and yesterday? Is that true for just specific tools or specific prompting? Like what's causing that? So that's a, that's the sort of end-to-end use case of that people use it for, if that makes sense. Yeah. Yeah. **Guest** [36:09] I think like we are entering the era where people are doing observability on agents. **Zubin Koticha** [36:15] Yeah. **Guest** [36:15] This is actually, I would say like one layer further or one step further. Like what do you see as the main driver for people to be like, oh, normal observability is not sufficient? I need to go further. **Zubin Koticha** [36:27] Yeah. I think it's really just, and I'm be curious what you think about this. I think it's really just agents are crazier than ever before,right? More tools, more context, way more intelligent, more real decisions that they can make. And they're just being used by way, way larger groups of people. And so when you have this massive amount of data in production, it just makes having good monitoring and observability like more important than before. And it makes it good monitoring and observability, in my opinion, more important than, than just testing or evaluations. Even if you have like some online evals, IMO, you need to have like really, really good end-to-end monitoring of the entire system. Curious if you have any thoughts there as well. **Danny Gollapalli** [37:20] So I think another major issue is like the unknown issues are even more important. So I think having like a generic user frustration classifier is actually really powerful. Say, for example, we also have this another feature called like issues, which basically is like an agent that's sort of like minds for newly occurring issues,right? Say, for example, Sentry has similar to Sentry in a way where there's like a new exception which is occurring. So it like alerts you on that. So say, for example, you are a coding agent provider and then certain providers are like failing all of a sudden. And then you can actually, it can actually figure out, okay, there's a subtle spike in user frustration. It's sort of like similar to how a human operator would. It can start digging into are there any patterns for the spike in user frustration? And then it could figure out that, okay, people who are like, say, for example, dealing with a specific Postgres provider start to face issues. So we have actually seen this happen live for a couple of our customers where they had a database provider failing and then we had like an automatic issue being created for them. **Zubin Koticha** [38:39] Yeah. Basically, once you have that good set of signals, like a good user frustration classifier, as Danny said, you can basically do clustering on it to find like what are, what is the root causes. Yeah. **Guest** [38:56] How do you integrate with Lighthouse or RAND or what does that look like? **Zubin Koticha** [39:01] Do you want to talk about integrations? **Danny Gollapalli** [39:02] I think we might have a very basic SDK for it. So our Python side of support is like fairly weakright now, but we have like fairly good like AI SDK support built in. So the AI SDK even has like self-diagnostics built into it. So we inject the tool for you so that you wouldn't have to do anything. But it is going to get better. So yeah, we actually released like 10 different test cases in the past month. So we have a person working on SDKs actively. So it's going to improve. ### Experiments Q&A **Guest** [39:40] I have a question about experiments, running experiments. And I'm curious how your platform helps me with this issue. I have a team of around 10 people building my AI agents platform and we constantly change things like all the time. And we have a lot of feature flags and some of them are experiments. And the rate of the change is so big, like every day everything changes. I just cannot compare the, the, the traces, the sessions of users because I don't have enough time to do it, you know? **Zubin Koticha** [40:22] Yeah. **Guest** [40:24] I need to have like a base system and then run a few days with, on, on parts of the users with one feature flag enabled so I can actually compare the data and analyze and get some insights out of it. And I just don't have enough time to do it, you know? **Zubin Koticha** [40:42] Yeah. So how are you doing, how are you sort of doing thatright now? Are you just kind of. **Guest** [40:47] It's like Wild West, you know? **Zubin Koticha** [40:48] Yeah. **Guest** [40:49] That's why I said I'm just using, mostly using Claude because I just give it everything and ask it questions and try to like figure out the, the insights. But it's not really, you know. **Zubin Koticha** [41:02] Yeah. I got, I get what you're saying. So a few things there. The first is like you can use experiments if you, if you want to keep that like shipping speed, you don't have to run like long multi-day experiments. You can ship something and if you have a sufficient sample size, you could see pretty quickly if there's any regressions or not. Like you just, maybe it's like 1% different or 2%. That's enough for you to be like, okay, it's fine. It's not like breaking anything drastic. The other thing is like kind of what Danny was talking about, this, we have an agent which is basically, you could think of, it's basically exposing all of these signals to Claude to make decisions on if things are better or not. And we're thinking also about how we close this loop. Maybe you have a really good set of signals and then you have like essentially an agent that can look at all these signals and then it can find issues based on that, what's changing, et cetera. And then it can like create a PR based on that. And then it can see how, you know, run some new experiments based on these new PRs. And like this can become this infinitely self-improving loop, which is like, which is very interesting, but that's one thing that I think about. I don't know if you have any additional thoughts. **Guest** [42:17] It's slow because you need to deploy it to production and wait for, for some data to get it. **Zubin Koticha** [42:23] Yeah. Depends on how, how much data you have. But that's, it's, yeah, it, it really depends on how big these sample groups are as well, et cetera. Sometimes it's like a few minutes you can tell, but sometimes they want to wait for longer. Yeah. **Guest** [42:42] Does your platform help with like enabling experiments on sessions so you can maybe automatically enable some experiments so you can like take care of the logic that every session has only one experiment enabled so we can easily compare it to the base or something like that. **Zubin Koticha** [43:00] We have, we're working on stuff like that actively, but yeah. **Guest** [43:05] Thanks. **Guest 2** [43:07] What we are storing the original traces and then they come in and implement my new signal source callback or whatever. If you can like fail signal for the original so I can do some kind of postmortem analysis. **Danny Gollapalli** [43:21] Yeah. We do. So you can like ingest all your historical data and then when you create a signal, we actually sort of like run like a quick backfill of the past couple of days. So yes, so that's definitely supported. **Guest 2** [43:35] Thanks. Is there a free plan to try it out? **Zubin Koticha** [43:43] We do have a free trial. We're going to try to make, it'sright now it's two weeks. Probably going to make that longer soon, but if you just DM me, I can, if you, I can have my, oh, do you want to open this so I can just have our things? Well, yeah, but if you just, so we are hiring. That's a, that is a thing that we're very excited about, like trying to massively increase the size of the team. And if you message me at either Twitter or you can email me as well, like that's something I can just set you up with a longer free trial. Yeah. ### Traces Q&A **Guest 2** [44:20] Internally, so we use OPEAK and Sentry and others. How do you guys, like I would imagine you guys also use those tools and how does Raindrop fit? From how I'm understanding it, it's you're creating signal with your own models, whatever you're using so that you make our lives easier to identify signals and like harmful intents and user behaviors. But how would you maybe have an example of how do you have that full stack of how it works with the Sentry and OPEAK and like maybe where there are overlaps where you go directly for those competitors? **Zubin Koticha** [45:02] So if you're sending all the telemetry data, we can find any exceptions in the traces, tool errors, et cetera. And that's the thing that you can also track within Raindrop. And that's, that is an explicit signal. So there's like implicit and explicit signal. So that becomes an explicit signal. **Danny Gollapalli** [45:21] So I think most of the observability platforms will give you like the agent trace, the token usage, if the tool call failed or not. But I think where we sort of like shine is sort of like the fuzzy part, the fuzzy failures,right? Where the user is like frustrated, which I think matters more than the explicit signal that you sort of get from Sentry. I mean, obviously those are also important, but we focus a bit more on the fuzzier side of the failure space. But at the same time, we also have a trace view. I think we also have a very interesting feature called like trajectories, which sort of like visualizes. If you want to find like a trace which has like three different tool call failures. So you can actually. **Zubin Koticha** [46:08] Oh, hey, my name's Austin again, but. **Danny Gollapalli** [46:11] Okay. Let me just get in. So you can sort of like describe the type of trace that you want to look at. So let's see. Hope we have like data, but you can more or less like describe the type of trajectories that you want to see instead of just like configuring it. So we do both in a way. So you can obviously set up tools are failing, sort of like alert as well. **Zubin Koticha** [46:46] So you can just search like for any trajectory. So yeah, you can see that this is sort of how the tools are being called and what order. You can see which ones have, have errors. You click into them, you can see the input and the output to the specific tool, like what actually screwed up here. And you can see, okay, it's interesting that this has like this, the, the no one lets you, this is pretty much the only place where you can visualize tools like this. But you can see here, like you can get a shape and understanding of the topology of what's going on here. And you can see when there's other ones that look similar, you can sort of see, okay, this kind of looks similar to this. And then that gives you a sense. You can do like search on this. Again, we have an agent that can look through these and give you a sense of what's going wrong. And so it just makes it really easy to find issues in, in agents. Yeah. Cool. Anything else? Any other questions? **Guest** [47:47] Can you export the data that you just showed? **Zubin Koticha** [47:50] The directories data? **Guest** [47:51] Yeah. **Zubin Koticha** [47:52] What would you want to export, like just the raw trace logs or what, what do you want to? **Guest** [47:59] So I think we usually what our customers do is that they already have like a hotel stream,right? So we just end up being like another target, I guess. But at the same time, they do want us to like export the signals that we label. So we do support like BigQuery and Snowflake. So we do export the event and then the signals that were classified for that event. **Zubin Koticha** [48:24] Last questions? **Guest 2** [48:25] Can we look at signals? **Zubin Koticha** [48:32] Let's do a longer timeframe. So let's go over the last month. So you can see stuff like refusals. And then again, if you click into any of these, you can get a sense of over time and task failure, jailbreaking, like what specifically is going on. And then you have your self-diagnostics ones as well, capability gap, et cetera. Cool. **Guest 2** [48:58] And do you have open data on like number of traces that you guys unload? Because it seems that with your clients, the tool is extremely valuable when you have those agents adopted at a very big scale. And so I can imagine that you have a lot of data. Do you have some data? **Zubin Koticha** [49:17] Yeah, it starts being, is your question is like what's the smallest where it's useful or what's the. **Guest 2** [49:22] Like volume of all the data that you're receiving, processing and generating signal around like how many jailbreaks do you see across all the clients? **Zubin Koticha** [49:31] Oh, do we have any sort of like. **Guest 2** [49:33] Like are we speaking. **Zubin Koticha** [49:33] We should, we should do something like that. That would actually, that would be very interesting. We don't have anything like that. **Danny Gollapalli** [49:38] I think we have like mixed opinions about that. I think Eatslip does it in a way,right? But people generally have a negative reaction to it. Maybe it's different here, but at the same time, do our customers want us to do that? It's like a different question as well,right? So, but yeah, we would love to, but I think there are like compliance reasons where we can't actually put our customers' data out there. **Zubin Koticha** [50:04] Cool. Anything else? Allright. Thank you, everyone. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.