Everything between a demo that works and a feature you can leave running.
Sign in to track your progress across sections.
38 articles
Listed inDeterministic EvalsProduction AIon
An evaluation is deterministic when it is computed rather than judged by a model, which is not the same as exact rather than fuzzy. A similarity score is computed, so it sits between exact matching and a judge model, and it costs nothing to run.
Listed inRagasProduction AIon
Faithfulness, answer relevance, and context precision — how the metrics are computed and what they miss.
Listed inLLM EvaluationsProduction AIon
The open-source eval framework and registry — a good template for structuring your own graded test set.
Listed inLLM EvaluationsProduction AIon
Stop asserting one output and start measuring across many. Three layers, and the cheap one people skip on the way to a judge model.
Listed inDeepEvalProduction AIon
Evaluations in pytest. Almost every built-in metric calls a judge model, which decides what belongs in a blocking gate and what does not.
Listed inRegression TestingProduction AIon
Not "is this good" but "is this worse than last week". Build the set from failures you already had, and read the diff rather than the pass rate.
Listed inModel-Based EvalsProduction AIon
A judge answers what code cannot decide, and costs a model call per case while being able to be wrong. Validate it on ranking rather than absolute scores, and test it for grader hacking, position bias and verbosity bias.
Listed inEvaluation MetricsProduction AIon
Three questions need different machinery: is there a right answer, is there source material, or neither. Faithfulness needs no reference answer, which is what makes it computable on production traffic.
Listed inRagasProduction AIon
A metric library built around retrieved context, where `retrieved_contexts` is a first-class field rather than something you fold into a prompt. Whether you have a reference answer splits the metrics in two, and the half that needs none can run on production traffic you never labelled.
Listed inHuman EvalsProduction AIon
Measure whether two reviewers agree before you collect a single score, because a rubric they read differently turns every later number into noise. Comparison beats rating, since people drift on absolute scales and stay consistent picking a winner.
Listed inLLM ObservabilityProduction AIon
Chip Huyen assembles the whole reference architecture step by step — context, guardrails, routing, caching, observability.
Listed inTracing & LoggingProduction AIon
One answer is many steps, and the step that broke is the one you did not log. What belongs on a span, and what should stay off it.
Listed inPostHogProduction AIon
A trace says what your system did. Whether the feature worked is a question about what the person did next, on the same events.
Listed inLangfuseProduction AIon
Open-source tracing, prompt management, and evaluation — including the data model behind a trace.
Listed inArize AIProduction AIon
Tracing and evaluation in one OpenTelemetry-native tool that can run locally. The drift and embedding analysis often attributed to it does not appear in the documentation.
Listed inLangSmithProduction AIon
A failing production trace becomes a saved test case, and that case runs against every later change. The documentation lists support for several frameworks and providers, so LangChain is not a prerequisite.
Listed inCost & Latency MonitoringProduction AIon
Output, cache reads, cache writes and batch requests are priced differently, so one token total cannot produce a bill. Record time to first token apart from total duration, and alert on ratios rather than on spend that grows with success.
Listed inLLM ObservabilityProduction AIon
A 200 in 1.2 seconds tells you nothing about a response truncated mid-sentence. OpenTelemetry's GenAI conventions settle the attribute names, and warn that prompts and completions are likely to contain PII.
Listed inHeliconeProduction AIon
An OpenAI-compatible gateway rather than an SDK, which buys caching and per-customer rate limits the other tools cannot offer. The proxy sits in your request path and the async mode does not, and that choice decides whether Helicone going down affects your product.
Listed inLangfuseProduction AIon
Langfuse is the observability platform whose whole stack you can operate yourself, which decides the tool when prompts cannot leave your network. Generations carry fields that spans do not, so cost dashboards exist without you computing them.
Listed inProduction MonitoringProduction AIon
Collecting telemetry is not monitoring; monitoring is deciding what reaches a person who can act. Four signals carry information only an LLM system has, starting with a finish-reason mix that turns truncation into a quality regression returning HTTP 200.
Listed inPrompt Injection AttacksProduction AIon
Years of worked examples showing why filtering does not solve injection, and why the dual-LLM pattern is the closest thing to a fix.
Listed inSecurity and Privacy ConcernsProduction AIon
The industry checklist: injection, insecure output handling, supply chain, data leakage, and the rest — with mitigations.
Listed inConducting Adversarial TestingProduction AIon
Derive your attack surface from the OWASP taxonomy rather than a jailbreak list, then fix findings in the layer that can enforce them.
Listed inContent Moderation APIsProduction AIon
Scores are signals for your policy, not a blocking decision. Block on flagged alone and you suppress your own model declining to help.
Listed inBias and FairnessProduction AIon
You cannot inspect the training data, but your prompt, your examples and your retrieval are yours. Matched inputs that differ in one attribute test whether the system treats equivalent cases equivalently.
Listed inConstraining Inputs and OutputsProduction AIon
Length caps control cost, enums keep invalid values out, and schemas prevent parse failures. None of them makes a value true, and a rule stated in a prompt is a request rather than a constraint.
Listed inAI Safety and EthicsProduction AIon
You did not train the model and you still ship the feature, so the harms arrive at your product. Four decisions no library makes for you, including which outputs need a person first, where the guidance names code generation alongside high-stakes domains.
Listed inRobust Prompt EngineeringProduction AIon
A prompt tuned on twenty clean cases fails on the twenty-first because nobody said what to do with an empty field or contradictory input. Name the awkward cases explicitly, and keep limits in code, since a prompt asking for 200 words is a request rather than a cap.
Listed inAdding End-User IDsProduction AIon
Without a per-user identifier a provider's abuse systems see only your API key, so the remedy lands on your account instead of on one customer. The field is `safety_identifier`, the value is a hash rather than an email, and unauthenticated traffic still needs a session ID.
Listed inSecurity and Privacy ConcernsProduction AIon
Zero data retention exists and its carve-outs are where teams get caught: some models require 30-day retention, and flagged content can be held for two years regardless. Adopting ZDR also disables CORS, so a browser-calling architecture becomes a rewrite.
Listed inPrompt Injection AttacksProduction AIon
Your instructions and the user's input arrive in one token stream with nothing marking which is which. OWASP says fool-proof prevention may not exist, so the goal is an architecture where a successful injection cannot do much.
Listed inKnow Your Customers & Use CasesProduction AIon
Two scope questions asked before launch: who may use this, and what it is for. A signed-in user can be warned or removed while an anonymous one can only be blocked by IP, and a scope that lives only in a prompt is a suggestion.
Listed inCodexProduction AIon
The name meant a completion model once and a coding agent now. Sandboxing and approvals are the operating decision, not the surface you run it on.
Listed inWindsurfProduction AIon
Windsurf became Devin Desktop in June 2026 and Cascade was replaced. What the editor is now, and how it retrieves code from your repository.
Listed inReplitProduction AIon
The environment, database and hosting arrive with the agent, so the gap between an idea and a shareable URL closes by default. The documentation now names Chat, Build and Design, and what you trade is owning the runtime and the pipeline.
Listed inClaude CodeProduction AIon
Terminal, IDE, desktop, web and CI are one engine sharing your CLAUDE.md, settings and MCP servers, so moving between them costs no reconfiguration. Hooks are the mechanism teams reach for last and should reach for first, because a hook enforces where an instruction asks.
Listed inCursorProduction AIon
An editor built around a coding agent, where Tab's predictive editing is the feature no terminal agent has. Composer names both a mode and a selectable model, and the model list spanning three providers plus Cursor's own makes benchmarks harder to read.