AI engineer interviews test one thing: can you build and operate LLM systems in production. Not transformer math, not backpropagation derivations, not paper trivia. Interviewers want evidence that you can design a retrieval pipeline, debug it when it hallucinates, keep an agent from doing something stupid, and reason about cost and latency like an engineer rather than a demo-builder.
In practice you will face four formats:
- Technical Q&A — rapid-fire and deep-dive questions on LLMs, RAG, agents, and production concerns. This post's worked examples target this round.
- System design — "design a support copilot" style prompts where the shape of your answer matters more than any single choice.
- Take-home project — build a small but production-shaped app in a few days.
- Behavioral / project deep-dive — narrate something you built and defend the trade-offs.
Below: ten representative questions with model answers, a worked system-design drill, what take-home reviewers actually check, and a practice plan. Depth over count — ten answers you can deliver confidently beat sixty you've skimmed.
The questions, with model answers
LLM fundamentals
1. "What do temperature, top-p, and max tokens actually do? What would you tune for a support bot that hallucinates?"
Temperature scales the logits before sampling — lower values make output more deterministic; top-p restricts sampling to the smallest token set covering that probability mass; max tokens caps output length. For a hallucinating support bot I'd set temperature near zero for consistency, but the honest answer is that decoding params don't fix hallucination — grounding does. I'd fix retrieval and add citations first.
What they're probing: whether you know the mechanics cold and know their limits. Candidates who answer "lower the temperature" to a grounding problem fail the follow-up.
2. "When do you fine-tune instead of prompting or RAG?"
Fine-tune to change behavior — output format, tone, style, consistent tool-calling patterns — not to inject knowledge. Knowledge that changes belongs in retrieval; knowledge that's stable and phrasing-specific can go in the prompt. Fine-tuning is the last tool I reach for because it adds training pipelines, versioning, and eval burden.
What they're probing: judgment against hype. "I'd fine-tune" as a first answer signals you haven't operated a system where the underlying data changes weekly.
RAG and retrieval
3. "Your RAG app returns confident but wrong answers. How do you debug it?"
Start with retrieval, not the model. Log the retrieved context for failing queries — is the right chunk even being fetched? If not, look at chunking, embedding quality, metadata filters, and reranking. If the right chunk is retrieved and the answer is still wrong, the problem is grounding or the prompt: check whether the answer cites the context and whether the prompt instructs the model to abstain when context is insufficient.
What they're probing: debugging order. Retrieval-first is the mark of someone who has actually operated these systems; model-first is the mark of someone who has only built demos.
4. "How do you choose chunk size and overlap?"
Empirically, not by superstition. Start with a reasonable default — a few hundred tokens with 10–15% overlap — then build a small set of representative queries and measure retrieval hit rate across chunk sizes. The trade-off: small chunks embed precisely but lose context; large chunks preserve context but dilute the embedding and blow up token cost. For structured docs I'd split on structure (headings, sections) before fixed sizes.
What they're probing: whether your defaults are measured or cargo-culted, and whether you can articulate the precision-versus-context trade-off.
5. "How would you evaluate a RAG system?"
Build a golden set of representative questions with known-good answers and source passages — 50 to 200 is enough to start. Score retrieval (was the right passage fetched?), faithfulness (does the answer follow from the context?), and answer relevance, using a mix of deterministic checks and an LLM judge validated against human labels. Run it in CI on every prompt, model, or index change and track scores over time to catch regressions.
What they're probing: whether you treat evals as a first-class engineering artifact. "I'd eyeball some outputs" is a red flag in 2026.
Agents and tool use
6. "When would you use an agent instead of a workflow?"
Only when the steps genuinely can't be predetermined. A workflow — a fixed graph of LLM calls and code — is cheaper, more reliable, and far easier to test, so it's the default. I reach for an agent when the task is open-ended, branching, and multi-tool: deep research, multi-step debugging, exploratory data work. Even then, I bound it: max iterations, scoped tools, and guardrails on every action.
What they're probing: reliability-first thinking. "Agents for everything" tells them you'll ship demos that fall over in production.
7. "How do you make an agent's tool calls safe in production?"
Least privilege everywhere: each tool gets the minimum scopes it needs, writes go through an allowlist of actions with validated parameters, and destructive or expensive operations require human approval. I'd treat all retrieved content the agent reads as untrusted input, add rate limits and per-run budgets, and log every tool call for audit. Then eval the failure modes: does it refuse out-of-scope requests, does it recover from tool errors?
What they're probing: whether you think about agents as systems with a security surface, not just prompts with functions attached.
Production, LLMOps, and evals
8. "How do you defend a RAG app against prompt injection?"
Assume retrieved content is hostile. Defenses in depth: keep user and retrieved content clearly separated in the prompt with instructions that tool use is never triggered by document text; constrain output format and validate it; give tools least privilege so injected instructions can't reach dangerous capabilities; and for high-stakes actions, require confirmation. You can't eliminate injection — you shrink the blast radius.
What they're probing: security realism. Anyone claiming a prompt-level fix ("just tell the model to ignore injections") hasn't thought about it seriously.
9. "Your token bill tripled after launch. What do you do?"
First, trace usage per request — which feature, which prompts, which users drive the spend. Common culprits: retrieved context padded far beyond what's used, full conversation history resent every turn, and a frontier model doing work a small model could do. Fixes: semantic caching for repeated queries, trimming and summarizing context, routing simple requests to a smaller model, and setting per-request budgets. Cost is an architecture decision, not an ops afterthought.
What they're probing: observability instincts and whether cost reasoning is in your design vocabulary at all.
10. "How do you know a prompt change is safe to ship?"
The same way you know any code change is safe: tests. Run the golden-set eval offline before merging — if faithfulness or task success regresses beyond a threshold, it doesn't ship. Then roll out behind a flag or to a small traffic slice, watch production metrics (task completion, escalation rate, latency, cost), and roll back on regression. Prompt changes are code changes and deserve the same gates.
What they're probing: whether you apply normal engineering discipline — CI, canaries, rollback — to LLM systems, which is exactly what separates an AI engineer from someone who vibed a demo.
These ten are the shape of the round, not the whole pool. The full bank — 60 questions across fundamentals, RAG, agents, evals, and system design, with answer frameworks — lives in the AI Engineer Interview & Portfolio Kit.
The system design round: a worked drill
Prompt: "Design a RAG support assistant that answers customer questions from our help center." Here is the answer shape interviewers want — notice it's a sequence of decisions, not a diagram recital.
1. Clarify requirements first. How many documents (say 40k help-center articles), how often they change (daily), expected traffic (50k conversations a month), latency target (p95 under 3 seconds), languages, and what "success" means — deflection rate, CSAT, escalation accuracy. Asking these questions is part of the test.
2. Walk the architecture in layers. Ingestion: pull articles from the CMS on a webhook, chunk on document structure, embed, and upsert into a vector store with metadata (product area, plan tier, last-updated). Retrieval: hybrid search (vector plus keyword) with metadata filters, then a reranker over the top candidates. Generation: prompt with retrieved chunks, instructions to cite sources and abstain when unsure. Serving: streaming responses, semantic cache in front, and tracing on every request.
3. State trade-offs explicitly. Hybrid search adds an index to maintain but rescues keyword-heavy queries like error codes. Reranking adds ~100ms but buys precision you can measure. A managed vector store costs more per month than pgvector on your existing Postgres but removes an ops burden at this scale — either answer is fine if you justify it with the requirements.
4. Evals and failure modes. Golden set from real historical tickets; score retrieval hit rate, faithfulness, and abstention correctness; gate prompt and index changes on it in CI. Name the failure modes unprompted: stale articles, out-of-scope questions, prompt injection via pasted ticket text, language mismatches.
5. Close with cost and latency. Do rough math out loud: ~3k input and 400 output tokens per conversation on a mid-tier model is well under a cent per conversation, so the monthly model bill lands in the low hundreds of dollars — embedding and reranking infra is the same order. Cache the top repeated queries and that drops further. This kind of envelope math is what "production judgment" sounds like.
For the full framework — including how this maps to the classic ML design rounds — see the GenAI system design interview guide.
The take-home project: what reviewers actually check
Reviewers spend ten minutes on your repo. In that time they check four things, in this order:
- README. Can they run it in under five minutes? Setup steps, architecture diagram, and — the differentiator — a "trade-offs and limitations" section.
- Architecture. Separation between ingestion, retrieval, generation, and serving. Config and prompts out of the code. No API keys in the repo.
- Tests and evals. Even a small eval set with a script that scores it counts for more than any feature. It signals you know what production means.
- Trade-off notes. A short doc explaining why you picked this vector store, this chunking, this model — and what you'd do with more time. This is the senior signal.
Depth beats breadth every time: one polished RAG or agent project beats five tutorials cloned from YouTube. The projects hub has production-shaped specs, and the projects that get you hired breakdown shows what hiring managers respond to.
The behavioral round: narrating a project
Every project story should follow the same spine: problem → constraints → trade-offs → results. Interviewers are listening for the constraints and trade-offs — anyone can build with unlimited time and budget.
An example script:
"Our support team was drowning in repetitive tickets, so I built a RAG assistant over 8,000 help-center articles. The hard constraint was accuracy — a wrong answer erodes more trust than no answer — and a latency budget of three seconds. I chose hybrid search with a reranker over pure vector search because error-code queries kept failing in evals; it cost us about 100ms, which fit the budget. I also made the assistant abstain and route to a human when retrieval confidence was low. After two months it resolved 38% of tickets end-to-end with a CSAT score equal to the human team's, and the eval suite I built caught two regressions from prompt changes before they shipped."
Notice what's in there: a real constraint, a named trade-off with numbers, a failure mode handled, and a measurable result. Rehearse two stories like this until you can tell them without notes.
How to practice
Six weeks of deliberate practice beats six months of passive reading:
- Weeks 1–3: build or polish the project you'll narrate. Answers grounded in a real system are immediately distinguishable from memorized ones.
- Weeks 3–6: two mock interviews a week. Set a timer, answer out loud — not in your head — and record yourself. Score each answer against a rubric: did I clarify requirements, name trade-offs, mention evals, reason about cost?
- Throughout: drill one question category per day using the worked examples above as the bar for answer quality.
The interview prep hub has mock interview structures, scoring rubrics, and a prep timeline that maps to this loop — use it as the backbone of the six weeks. To start drilling today, the free interview kit has a 30-question sample from the full bank. And if you're earlier in the transition, the roadmap sequences the skills so your project stories come from real builds, not tutorials.
FAQ
How much LeetCode do I need?
Some, but it's not the focus. Most AI engineer loops include at most one standard coding round at easy-to-medium difficulty — strings, dicts, basic data structures, maybe a small class design. The differentiating rounds are LLM system design and the take-home. Spend 80% of prep there and keep coding fundamentals warm with a couple of problems a week.
Do I need ML theory?
Conceptual, yes; mathematical, rarely. You should be able to explain what an embedding is, why transformers attend, what fine-tuning changes, and what a token is. Nobody at a GenAI product company will ask you to derive attention or prove convergence. If a loop does demand deep math, it's an ML research role wearing an AI engineer title — the career path guide explains the distinction.
What if I've never shipped AI at work — what do I say?
Say it plainly, then point at what you built anyway. "My current role doesn't touch LLMs, so I built this RAG system on my own — here's the architecture, here are the eval results, here's what broke." Interviewers count a serious self-directed project with evals and trade-off notes as real experience, because it is. What they reject is tutorial projects presented as experience — the difference is the depth of your answers about it.
How long should I prepare?
With solid software engineering fundamentals and at least one real project: 4–8 weeks of focused prep. Starting from zero on LLMs, add the time to build a project first — that's the long pole, typically 6–10 weeks alongside a job. The prep itself (drills, mocks, system design practice) compresses well; fake experience doesn't.
Are remote AI interviews different?
Two practical differences. Live system-design rounds happen over a shared whiteboard or doc — practice structuring diagrams with text and boxes under time pressure, it's a distinct skill. And take-homes are more common in remote loops, so the repo quality bar above matters even more. Everything else — the questions, the rubric, the judgment being tested — is the same.