Tracing, cost, and quality signals from production — where the real eval set comes from.
Sign in to track your progress across sections.
11 articles
Listed inLLM ObservabilityObservabilityon
Chip Huyen assembles the whole reference architecture step by step — context, guardrails, routing, caching, observability.
Listed inTracing & LoggingObservabilityon
One answer is many steps, and the step that broke is the one you did not log. What belongs on a span, and what should stay off it.
Listed inPostHogObservabilityon
A trace says what your system did. Whether the feature worked is a question about what the person did next, on the same events.
Listed inLangfuseObservabilityon
Open-source tracing, prompt management, and evaluation — including the data model behind a trace.
Listed inArize AIObservabilityon
Tracing and evaluation in one OpenTelemetry-native tool that can run locally. The drift and embedding analysis often attributed to it does not appear in the documentation.
Listed inLangSmithObservabilityon
A failing production trace becomes a saved test case, and that case runs against every later change. The documentation lists support for several frameworks and providers, so LangChain is not a prerequisite.
Listed inCost & Latency MonitoringObservabilityon
Output, cache reads, cache writes and batch requests are priced differently, so one token total cannot produce a bill. Record time to first token apart from total duration, and alert on ratios rather than on spend that grows with success.
Listed inLLM ObservabilityObservabilityon
Seeing what your system actually did, not what you assumed it did.
Listed inHeliconeObservabilityon
A proxy that adds logging, caching, and rate limiting with one URL change.
Listed inLangfuseObservabilityon
Open-source tracing, prompt management, and evals.
Listed inProduction MonitoringObservabilityon
Drift, failure rates, and alerting on quality instead of just uptime.