// learn · production
Production AI engineering
A demo answers once; a production system answers every time — within a latency budget, a cost budget, and a safety policy, with the numbers to prove it. This is the path to operating LLM systems like an engineer, not a hobbyist.
New to the vocabulary? Start with LLMOps and evals in the glossary.
Structured traces on every call: prompt version, tokens, latency, cost.
Golden sets with pass rates — no prompt change ships without a number.
Input validation, output schemas, injection defense, least-privilege tools.
Cost ceilings and p95 latency SLOs per feature, with alerts.
Prompts, datasets, and configs in git, reviewed like code.
Canaries and rollbacks for model and prompt changes, not just code.
Quality, cost, and drift dashboards; sample real traffic for review.
Every production failure becomes an eval case — the suite grows forever.
None of these require new infrastructure on day one — a log line per call, a JSONL eval set, and a budget check are enough to start. The LLMOps guide for DevOps engineers walks the maturity path from there.
- 01LLMOps for DevOps engineersYour CI/CD and observability instincts, applied to models.
- 02Production-ready GenAI architectureThe layers between a demo and a system.
- 03LLM Eval & Observability Cheat SheetThe metrics and traces to wire first — free download.
- 04GenAI system design interviewHow production trade-offs show up in interviews.
- 05AI engineer interview questionsEvals, cost, and system-design questions you'll be asked.
Frequently asked questions
What's the difference between MLOps and LLMOps?
MLOps centers on training pipelines, feature stores, and model retraining. LLMOps rarely retrains anything — you iterate on prompts, retrieval, and guardrails instead. The new surfaces are prompts-as-config, nondeterministic outputs, token-based cost, and quality evals in place of accuracy metrics.
Do I need Kubernetes to run LLM apps in production?
No. A container on any platform with env-driven secrets, a health check, and structured logs covers most teams. Reach for Kubernetes when scale or team size demands it — the LLMOps disciplines (traces, evals, budgets) matter long before the orchestration does.
How do you test a nondeterministic system?
With golden datasets and pass-rate thresholds instead of exact assertions: grade outputs with exact match where possible, model-graded rubrics where not, and require a minimum pass rate in CI before any prompt or model change ships. Add sampled human review of production traffic on top.
What does production AI actually cost?
Almost entirely tokens: input tokens dominate in RAG systems because you pay for retrieved context on every call. Set a per-feature monthly budget, cache repeat queries, route cheap paths to smaller models, and alert on daily spend — a cost ceiling in code beats a surprise invoice.
Production AI Notes
One practical AI engineering email each week
One concept, one architecture, one project idea, and one interview question — written for developers who want to build and ship real AI systems.