OpenLLMetry is all you need

Hey everyone, I'm Nir, the CEO of trace loop, and today I'm going to talk to you a bit about OpenLLMetry, which is a nice open-source project we built. You can probably hear from the name OpenLLMetry that it originates in another project called OpenTelemetry, and in case you're not familiar with OpenTelemetry, I'm going to spend the next couple of minutes explaining to you what OpenTelemetry is.

So, you know, we're not talking about GenAI yet, we're not talking about LLMs, just plain simple cloud observability. OpenTelemetry is an open-source project, it's maintained by the CNCF, it's one of the largest projects out there after Kubernetes that is maintained by the CNCF, and it standardizes a way to do cloud observability in your, you know, cloud environment.

Currently it's supported by every major observability platform, from Splunk, Datadog, Dynatrace, New Relic, Grafana, Honeycomb, and many, many others. So if you're using OpenTelemetry, you can use it in conjunction with any of these platforms pretty easily. But what is OpenTelemetry, exactly?

OpenTelemetry is a protocol, first and foremost, that standardizes the way to do logging metrics and traces in your cloud application. Logging, I think, I don't need to explain to you what is logging exactly, because if you've ever written some Python, you know, scripts and you've written print, then you've done some logging.

So logging is an arbitrary event that you can send anytime you want in the lifecycle lifetime of your application, and it's just emitted as is, and you can view it later on, possibly with some metadata. Metrics, on the other hand, is completely different.

Metrics is something you want to see on an aggregate level, you want to see how it behaves across days, or across users, or, you know, whatever you want. And when we're thinking about metrics in the traditional cloud world, we are probably talking about CPU usage, memory usage, or latency.

And if we want to, you know, talk a bit about GenAI, if you're thinking about which metrics we want to see when we're building a GenAI-based application, it's probably things like token usage, latency, error rate, and so on.

Lastly, OpenTelemetry defines tracing. Actually, this was the first thing that was defined with OpenTelemetry, but it's the, I would say, the least trivial one. So tracing is basically tracking of a multi-step process. So again, thinking about, you know, cloud environments, you have microservices, they're talking to each other, and you want to see some process that spans across multiple microservices, you want to use a trace for that.

And then you can see, you know, how a certain request coming from the user is processed across these microservices. Specifically for GenAI, I think tracing is actually pretty common, because we are using, you know, a lot of multi-step processes, whether it's chains, workflows, or even, you know, agents or multi-step processes that interact and run tools.

So, yeah, logging metrics and traces, this is what OpenTelemetry as a protocol defined. But it doesn't stop there, because, you know, what can you do with a protocol? So OpenTelemetry is also an ecosystem, and it contains SDKs, instrumentations, collectors.

SDKs are the way you can, you know, manually send out these logs, metrics, and traces from your application. OpenTelemetry currently has 11 different languages supported in 11 different languages of SDKs, from Python, TypeScript, Go, C++, and many others.

You know, every language you're probably using has an OpenTelemetry SDK. Instrumentations are a way to do this automatically. So remember, if you're using an SDK and you want to send out logs, metrics, and traces, you need to do it manually.

You need to actually send out a log, or emit a trace, or a span, or a metric. But instrumentations can do it automatically, and we're going to talk about it in a bit. And lastly, collectors allow you to do some processing to your observability data before you send it out to whatever observability platform you're using.

So what are instrumentations? Instrumentations are a way, as I said, to automatically get some observability data, some visibility into some part of your application. Let's say you're using an SQL server, then you can use an instrumentation for that SQL server, like Postgres, and get some logs, metrics, and traces automatically.

The way these instrumentations work is that they monkey-patch the client library that you're using within your application, and then emit all the data that you want to, probably want to see in your observability platform. Everything happens on the application side, and they're designed

magnificently, you know, on an engineering level, so that the latency impact is almost negligible, and you get a, you know, a nice view of everything that's happening in your system without doing anything.

Collectors are self-deployable components. You can deploy in your own, you know, cloud environment, on Kubernetes, or whatever you want, and can provide you some pre-processing before you send data out to whatever, you know, platform you're using. For example, if you want to filter out some data that is not important for you, or you want to obscure PII, or obscure sensitive data and hide it, you can use a collector to do it.

And those, you know, ready-made components that you can just deploy, and they're completely open-source, have a lot of these features just built in. And lastly, if you also want to send this out, send the, you know, the observability data out to multiple providers, you can also do it with a collector.

I think at this point you're probably asking me, "Hey, Nir, you talked a lot about OpenTelemetry, but we are in a GenAI conference. What does that have to do with GenAI?" This is where OpenLLMetry comes in. We took this amazing project called OpenTelemetry and extended it to support a lot of, you know, GenAI frameworks, foundation models, and vector databases that some of, you know, you can see some of the logos of the instrumentations we built here.

So we extended it to support all of these, you know, amazing products. And because we're relying on OpenTelemetry, you can then get observability in whatever platform you're using. You want to see, you know, traces within Datadog, just use OpenTelemetry.

You want to see them in Sentry, you want to see them in Grafana Tempo, or in Dynatrace, just, you know, just use OpenTelemetry, configure it correctly, and that's it. You get logs, metrics, and traces automatically in your favorite platform.

It's kind of nice.

We've worked a lot on building instrumentations with our community, and now we have more than 40 different providers. So we're talking about foundation models like OpenAI, Anthropic, Cohere, Gemini, Bedrock, and many others. We're also talking about vector databases like Pinecone, Chroma, and many others.

And we also have support for frameworks like LangChain, LlamaIndex, CrewAI, and Haystack. So you have, you know, the instrumentations that automatically emit logs, metrics, and traces, and then just connect it to whatever platform you want, and you get it out of the box.

And remember, because these are instrumentations, it's kind of done automatically. So it's pretty cool. It's like magic.

Just to give you an example of what an instrumentation for, let's say, Pinecone would look like. So if the Pinecone instrumentation will contain, you know, ways to see the queries going out to Pinecone, see indexing happening within Pinecone, and also ability to investigate vectors returned from Pinecone.

So you want to see, you know, the data, the distances, the vector distances that Pinecone returned, or scores, latencies. All of these are available in the standard OpenTelemetry format.

So OpenLLMetry, it's a great, you know, way to connect

LLM-based applications to whatever platform you're currently using. And because it's a standard protocol, you're never tied to a specific platform. You can easily switch between platforms. It's just a matter of, you know, a configuration change. Because all of the platforms that support OpenTelemetry support it with the same, with the exact same format.

That's it. If you have any questions, I'm available. I will be in the conference. I'm available in the conference, or otherwise at nir@traceloop.com. Thank you very much.