The State of MCP observability: Observable.tools — Alex Volkov and Benjamin Eckel, W&B and Dylibso

Hey folks, um, my name is Alex Volkov. I'm an AI Evangelist with Weights & Biases.

I'm Benjamin Eckel. I'm co-founder and CTO of Dylibso. We're creators of mcp.run.

Alright, and we're here to talk to you about MCP observability. Hey Ben, I wanted to ask you a question.

Yeah, sure.

You've worked at Datadog before, and you're somebody who runs multiple MCP servers and, uh, clients on production. Something that happened—advice that happened—sounding in my agent, uh, in production the other day.

Okay. Uh, yeah, I mean, we've been running MCP clients and servers in production since the beginning. Uh, yeah, but wait, aren't you, like, working at an observability company, Weights & Biases? And don't you work on, like, what's it called, Weave?

Yep, that's true. I work a lot with Weave, and—but since I started adding some powers to my agent via MCP, all that observability that I'm used to from just having my own code run end-to-end has gone a little bit dark.

Gotcha.

So this is what we're here to talk to you guys about. Uh, the rise of MCP is creating an observability blind spot. As AI agents become more, uh, prevalent, the problem can compound. With more and more tools via MCPs, the less the developers can know about the end-to-end happenings within their agent.

Yeah. Um, yeah, so on MCP run, we're running both clients and servers, and because it's a new ecosystem, we've had to, like, cobble together a lot of our own ways to do observability. And I've been looking around, and it seems like everyone is sort of doing this in isolation, sort of solving the same problems.

Um, so, you know, we wanted to bring the community together on this issue, and so today we're going to talk about the state of observability and the MCP ecosystem.

Yep. So why do we care about this, and why do we think that you guys should care about this? So if you don't have the ability to quickly understand why things went wrong on production, where they went wrong, and how, your ability to quickly respond is greatly diminished.

And we care deeply about—we both build tools that need MCP observability, and we support MCP, and we both care deeply about developer experience as well.

Yeah. It's—it's really important to me because enterprise engineering teams don't ship something to production unless they know for sure that they're going to be able to identify security and reliability problems before their customers do. Um, and that's why they invest a ton of money in observability platforms.

And, uh, so if you're going to ship MCP to these production environments, you must seamlessly integrate with these observability platforms.

Yep. So because we care deeply about, uh, developer experience and, uh, W&B, Weave, uh, I'm happy to announce here on stage that Weave supports MCP. Yay! As long as you're a developer of both the client and the server, all you need to do is set this MCP trace list operation environment variable on your client and server, and, uh, we'll show you the, the list tool calls, and we'll show you the, the duration of your MCP calls.

This works currently with our Python-based clients, and this is how it looks, super quick. With the red arrows, you can see the client traces, for example, and with the blue arrows, you can see we're pointing to the Calculate BMI tool and, and the other tool, and that's it.

Observability solved,right? Let's—let's get off the stage. We're done.

No, wait a second. So, uh, what about this, like, Calculate BMI tool, this, uh, MCP server? Why can't I see into that?

Um, you—yeah, we're working on this.

Uh, yeah, also this seems like—this is specific to Weave,right? Um, is there not, like, a vendor-neutral way to do this that's standardized?

Yeah, that'sright. Uh, this is a, uh, bespoke integration that we built into Weave, into our SDKs in Python. And while working on this, while our developers have been building this, like, uh, integration within our MCP tooling, I was advocating internally and externally that we should align with the open nature of MCP as a concept and created observable.tools.

Maybe some of you have seen this. This is a manifesto to drive a conversation that this is a problem that needs solving. And, uh, between observability providers, uh, such as us and other folks that's been on stage before and going to be on the eval shack tomorrow, uh, to do observability in a vendor-neutral and standardized way.

And so while working on observable tools, I realized—I did some search—realized that a vendor-neutral, scalable way to add observability exists, uh, and there could be a great way to marry the two open protocols to work together.

Yeah, exactly. Uh, fortunately, MCP-powered agents are really just another distributed system, and we've been doing that for decades. So OpenTelemetry is just the way that's—that we've, like, settled on doing that. Um, we're going to talk about OTel a little bit.

If you're not, uh, familiar with it, we need to learn about a few primitives first. So the main primitive that we need to learn about is the trace. So a trace is kind of like an atomic operation in your system.

It's made up of a tree-like structure of steps that we call spans. And a span represents the duration and some arbitrary metadata for each step. And what this step is exactly is completely up to you to define. It can be as high-level as, like, an HTTP request.

It can be as low-level as a tiny little function call. Um, here's an example of, like, a checkout experience, an API for a checkout. The size and position of each of these spans correspond to how long it took and where it sits in the call graph, respectively.

And just from this data, you can tell a lot about a system and how to observe it. Um, the other primitive you need to be aware of is sinks. So a sink is kind of like a centralized database where all your telemetry goes.

But often they come in the form of this, like, whole platform with, like, a UI and dashboards and alerting and monitoring and all those things.

So there's a lot of logos here, Ben. Uh.

Yeah.

Basically, a sink is an open standard way for folks, like collectors, to, like, receive those spans. As long as the developer instrumented their application code in a certain standard spec way, everybody can just receive those in, in the same unified way,right?

Exactly. Yeah. It's—if you squint, it's just kind of like a bunch of databases that all support the same schema and wire protocol.

And in fact.

And you can switch them out.

And in fact, they don't have to change much of their code or even change the code at all.

Right.

It could be just config,right?

Right.

Uh, by the way, LLM observability tools like W&B, Weave, and some friends, Simon from Lockfire here before, and some other friends, uh, all have switched to support OTel as well, OpenTelemetry. It's becoming, like, this global standard.

Great. Uh, yeah, another great thing about having a centralized sink, uh, is the last concept, distributed tracing. So going back to our checkout endpoint, if the, uh, fraud service sends its span to the same sink, then we can stitch back the together the traces and show the whole context.

So maybe you're kind of seeing where the MCP server stuff comes in here.

Yep. So, hey Ben, if it's possible via the integration to the open protocol, um, what if I want to use MCP servers that other people host, like GitHub, like Stripe, like other folks?

Yeah, that's a good question. So, um, with MCP-enabled agents, or really just any distributed system, there are kind of two scenarios. There's when the client and server are in different domains, and then there's when they're in the same domain.

And by domain here, I don't necessarily mean the literal definition. I just mean, like, the administration—the administrative domain of control,right? Like, do you—like, do you own this MCP server? Do you own this MCP client? Or is it a third-party thing?

So your GitHub/Stripe example is, like, a great example of, like, the different domain scenarios. So, um, this is a trace of an agent that is executing the prompt, "Read and summarize the top article on Hacker News." So it's going to reach out to this, like, remote fetch server to read Hacker News.

But it appears to us on the trace as a single service span because it's out—it runs outside of our domain of control. So it appears to a black box to us.

Um, but suppose we do own the server. Like, maybe it's running in a da—different data center than the client. Um, how do we get actually the whole context? Uh, it's pretty simple. So with distributed tracing and context propagation, we can have the remote fetch server send its spans to the same sink as the client, and the sink will just stitch together the missing, uh, parts of the trace back for us.

So in this graphic, you can see that we can now break into that fetch server, and we can see what it's doing. It's making some HTTP requests that's taking roughly 350 milliseconds, and then it's doing a little, uh, crunching to create some markdown.

Okay. So that—that's great in theory, and we went through this. We could have a whole hour talking about OTel. Not that we got an hour. Uh, but how do we can actually marry those two protocols together,right? Uh, is there a standard way that the MCP spec folk deploy away for us for observability?

Um, not quite. It was—it was, uh, pretty tricky to get—to get working. Um, it does work today, but, uh, it required a little bit more work than it should have. So in order to do this, we need to, as I said, propagate the trace context from the client to the server.

So here's a TypeScript example. And when we call a tool in the client, um, we're going to extract our current span, and we're going to, uh, pass it along to the server. And we achieve this by basically just shuttling the data through the protocol's meta payload.

And, uh, now that we're inside the server, uh, this would be, like, in the fetch server, we can pull that trace context out, inherit it as our current span, and then when we send our spans off to the sink, uh, it's as if it came from that parent span, and the sink can stitch it back together.

Ben, this is awesome. So you basically used an undocumented kind of property of the, uh, sending the payload together with the payload between clients and servers, um, to pass along that data that OTel needs to connect those things together,right?

Yeah, sort of. I just kind of had to abuse the lower-level interface reserved for the protocol. But higher-level ways should be provided through tooling, and that's something we should talk about a little bit later in the talk.

Yep. And so.

Oh, yeah. So by the way, this is, uh, this is not just, um, a screenshot. This is a working demo. So, um, it's a lot more code than what I showed in the slide. So if you want to actually go see how this works and adapt this for your needs, uh, go check out this GitHub link.

And I think, actually, you did that to, to get it to work with Weave,right?

Yeah. So now that we know how to pass context after you, you, you showed me the way, uh, let's see how amazing this solution actually is in practice. While Weave MCP, the thing I showed you guys before, was a bespoke solution baked into our Python SDK for Weave, the huge benefit of MCP, generally not only observability-related, is that servers and clients don't have to run on the same environment or share the same code or be from the same programming language.

So while we were working on the Python SDK, you built an agent in TypeScript. And so because W&B, W&B Weave supports OTel, OpenTelemetry, and it's an open protocol, uh, your TypeScript agent, it took me a few minutes to, by, uh, without changing much code, to just send those traces into Weave from a TypeScript agent and not necessarily from a Python agent.

So here, uh, here you could see in the green, the, the client traces are in the green, and then the server traces actually show what happens within those calls, uh, on kind of the, the server side as well.

Yeah, that's really cool. So how did—how did you actually get the traces into Weave?

So this is very, very simple, way simpler than before. Uh, we just define W&B Weave as the OTLP endpoint, a standard that you kind of, like, showed me around. Uh, and then folks can send their traces into wmbi.ai/otel.

And all you need to do in addition to this is authorize, so add authorization headers and specify which project you want to go into.

Cool.

Yep. So while we talked to you about observability, while I was working on this, I had a magic moment happening with MCP. I wanted to share this with everybody and.

Yeah, okay.

With you as well.

MCP story.

Yeah. So, um, I used Claude Opus 4 that just came out to Weaveify your agent that you built.

Yeah.

And to add this, uh, MCP observability. And W&B Weave, it's going to get a little meta. Stay with us. Uh, also has an MCP server.

Okay. What, what does it do?

So we have an MCP server that lets your agents or, or chats, et cetera, talk to your traces and see the data and summarize the data for you. Okay? So we have this MCP. It's been configured in my Windsurf.

Uh, and, and Claude, uh, code, uh, Opus 4, uh, was able to use this MCP server to kind of work through it. So here you see an example. Um, the agent basically started working on your code and then decided, "Okay, I'm going to run the code," and then said, "Okay, I'm going to go and actually see if the traces showed up at, at W&B Weave."

Then it noticed that they showed up, but they showed up incorrectly. So some input or output, a specific parameter that it needed to do, it didn't know how to do. It wasn't part of the documentation. And so, uh, the next moment just absolutely blew my mind.

This Opus 4 discovered that our MCP server exposes a support bot. So essentially, another agent, uh, decided to write a query for it, received the, theright information after a while, and acted upon this information, learned how to fix the thing that it needed to fix, fixed it, and then went back to Weave and, and, uh, to notice whether or not the fix was correct.

So my, uh, my coding agent talked to another agent via support, via MCP that it discovered on its own. I didn't even know that this ability exists. To work on your coding agent in, in things. The things got a little bit meta, and my head was, like, absolutely I was sitting like this while all this happened.

Didn't touch the keyboard once.

That's awesome. Yeah, it's pretty meta. Uh, yeah, before we go, I also wanted to have, uh, take a moment to have an announcement. So, um, mcp run will also be exporting telemetry to OTel-compatible sinks. Um, so as I mentioned before, we run both servers and clients.

Uh, so for servers, we have this concept called profiles, and these allow you to, like, slice and dice multiple MCP servers into one single virtual server. And on, on, uh, we also have the an MCP client called task.

And this is, like, a single prompt agent that could be triggered via URL or a schedule. And it also just sort of marries with the idea of profiles. Um, but yeah, soon you'll be able to get OTel out of both of these, and hopefully, you know, we'll, uh, connect up to Weights & Biases and have a little party.

Yeah, you can send those to Weave straight from mcp.run.

Okay. So, uh, to recap, um, observability is here at in MCP today, but it's not evenly distributed. Uh, OTel should get you most of the way there, but the community needs to come together, uh, create creating tooling and conventions to make it smoother.

Um, you shouldn't need to be an expert in observability to, like, get this stuff working.

So how do you get involved? Well, AI engineers, just start thinking about observability via MCP tooling and whether or not you're getting, uh, observability to the end-to-end of, of your execution chain. Um, for tool builders and, uh, platform providers, we should join and work on higher-level SDKs.

So, uh, Arise's open inference, for example, is a great start, but all of us should help with the instrumentation for our clients who use bespoke SDKs to work on conventions also together. Ben, can you explain semantic conventions super quick?

Yeah, sure. So as we learned earlier, um, spans, they carry user-defined attributes,right? So if they're user-defined, how does the sink know that a span is actually, say, an HTTP request with a 200 status code? Or how does it know that it's an MCP tool call that has an error?

Um, that's where semantic conventions come in. Um, and you can be a part of defining what the conventions are for agents that all observability platforms agree on. And if you're interested in this, I would suggest going to check out the, uh, GenAI Semantic Conventions effort by the OTel team.

And, um, yeah, lastly, for platform builders such as MCP run, um, you know, go add OTel support, help review RFCs, and finally, yeah, just come, like, talk to us about ideas because we're just everything's just kind of coming together.

Everything's so new and fresh, and we don't really know exactly what to do.

There's an additional track here at, at, uh, AI Engineer. This is called the hallway track. And I've learned more about the stuff that we were talking about, uh, out there.

Very good.

By actually talking to people who implement this than I learned while preparing, uh, before this talk. It's quite incredible. So, um, hey Ben.

Yeah, sure. Um, yeah, so again, I'm Ben. Uh, my call to action here would just be go check out mcp run. You can get a free account. Try it out. Uh, yeah, that's it.

And I'm Alex. Uh, uh, check out W&B Weave MCP OP to learn how to trace MCP, uh, with OTel. Uh, I'm also I did the observable.tools initiative. I would love for you to check out the manifesto to see if this resonates with you, to join forces to talk about observability.

And, uh, we yeah, please visit us at the booth. We have some very interesting surprises for you. We have a robotic dogright here, uh, that's observable. I also run the ThursdAI podcast. I want to send Swix a huge, huge shout-out for, uh, having, uh, uh, giving me the support to show up here and give if you guys are interested in AI news, we're going to record an episode tomorrow.

That's it. Thank you so much.