The metrics, traces, and guardrails to monitor an LLM system in production, distilled onto one page.
Eval metrics that matter
| Layer | Metric | What it tells you |
|---|---|---|
| Retrieval | Recall@k | Did the right document even get retrieved? |
| Retrieval | Precision@k | How much noise is in the retrieved context? |
| Generation | Faithfulness / groundedness | Did the answer stick to the retrieved context? |
| Generation | Answer correctness | Is the answer actually right, per a labeled set? |
| Generation | Citation accuracy | Do the cited sources actually support the claim? |
| Agent | Task success rate | Did the agent complete the scripted scenario? |
| Agent | Tool-call correctness | Right tool, right arguments, right order? |
| System | Latency (p50/p95) | What users actually experience, not the average. |
| System | Cost per successful outcome | Not cost per call — cost per result that mattered. |
What to trace on every request
- Input (raw user query, sanitized of secrets/PII before storage)
- Retrieved context (chunks, source, scores)
- Full prompt sent to the model (after templating)
- Model, version, and parameters (temperature, max tokens)
- Raw output and post-processed output
- Tool calls: name, arguments, result, latency
- Total latency and token counts (input/output)
- A trace ID that ties all of the above to one user-facing request
If you can't reconstruct why the system produced a specific answer from your traces alone, you don't have observability — you have logs.
Guardrails checklist
- Input validation before the request reaches the model.
- Output validated against an expected schema/format before use.
- Tool-call arguments validated before execution — never trust the model's output directly against a system with side effects.
- Retrieved/tool content treated as data, never as instructions (prompt-injection defense).
- A hard fallback ("I don't know" / escalate to human) when confidence or retrieval quality is low.
- Rate limits and cost budgets with alerts, not just dashboards nobody watches.
Running evals in CI
- Keep a labeled eval set of real or representative queries with expected outcomes — start with 20–50 examples, grow it as you find failures.
- Run the eval suite on every change to prompts, retrieval config, or model version — treat prompt changes like code changes.
- Track eval scores over time; block deploys on regressions, don't just log them.
- Use LLM-as-judge for open-ended output, but calibrate the judge against a small set of human-labeled examples first — an uncalibrated judge just adds a second, unverified opinion.
Want the full walkthrough — building the eval harness, wiring tracing into a real RAG service, and setting up CI gates? That's covered in the AI Engineer Interview & Portfolio Kit.