Advanced
Production AI

Running it in production

What to record, what to alert on, and the inputs you cannot trust.

What you will be able to do

Run a model-backed feature you can debug, bill accurately and defend.

Prerequisites
A feature in production, and Knowing whether it works.
Length
6 lessons · 23 min
Start course

Syllabus

  1. 1LLM ObservabilityA 200 in 1.2 seconds tells you nothing about a response truncated mid-sentence. OpenTelemetry's GenAI conventions settle the attribute names, and warn that prompts and completions are likely to contain PII.3 min
  2. 2Tracing & LoggingOne answer is many steps, and the step that broke is the one you did not log. What belongs on a span, and what should stay off it.4 min
  3. 3Cost & Latency MonitoringOutput, cache reads, cache writes and batch requests are priced differently, so one token total cannot produce a bill. Record time to first token apart from total duration, and alert on ratios rather than on spend that grows with success.4 min
  4. 4Prompt Injection AttacksYour instructions and the user's input arrive in one token stream with nothing marking which is which. OWASP says fool-proof prevention may not exist, so the goal is an architecture where a successful injection cannot do much.4 min
  5. 5Constraining Inputs and OutputsLength caps control cost, enums keep invalid values out, and schemas prevent parse failures. None of them makes a value true, and a rule stated in a prompt is a request rather than a constraint.4 min
  6. 6Content Moderation APIsScores are signals for your policy, not a blocking decision. Block on flagged alone and you suppress your own model declining to help.4 min