Honeycomb

OpenLLMetry is all you need

Feb 22, 2025 · 9:12

Nir, CEO of Trace Loop, introduces OpenLLMetry, an open-source project extending OpenTelemetry for tracing and monitoring GenAI applications. OpenTelemetry, maintained by CNCF, standardizes logging, metrics, and traces across cloud environments, supported by platforms like Datadog, New Relic, and Grafana. OpenLLMetry provides over 40 automatic instrumentations for foundation models (OpenAI, Anthropic, Cohere), vector databases (Pinecone, Chroma), and frameworks (LangChain, LlamaIndex, CrewAI). These instrumentations emit logs, metrics, and traces in OpenTelemetry format, allowing users to send data to any supported observability backend with a configuration change, avoiding vendor lock-in.

AI Engineering 201: The Rest of the Owl

Nov 8, 2023 · 56:57

Charles Frye, instructor of the Full Stack LLM Bootcamp, presents essential patterns for building language user interfaces (LUIs), arguing that while RAG chatbots are the 'to-do list app' of AI engineering, structured outputs via function calling (e.g., OpenAI's JSON schema, Instructor library) improve robustness, and agents with memory (like generative agents or Voyager in Minecraft) represent the true AI frontier. He emphasizes the need for hybrid search combining vector and keyword retrieval (citing Vespa, Postgres, Redis), and warns that monitoring and evaluation are the hardest engineering challenges: monitoring user behavior, latency quantiles (especially 99th percentile), and costs must be paired with observability tools like Honeycomb or Gantry, while evaluation often requires iterated decomposition or using LLMs as evaluators (GPT-4 as 90th percentile crowd worker). The episode concludes that shipping to learn — starting with production data to generate tests — is the dominant engineering mindset, and that the field is still filling in the gaps between inference and full product value.