LLM Engineer Interview Preparation Plan — From Transformers to Production

The "LLM engineer" role is real now — and the LLM engineer interview is one of the hardest to prep for because it spans transformer internals, fine-tuning, production serving, and agent systems simultaneously. The role is distinct from a data scientist (who builds models), an ML engineer (who productionizes them), and a software engineer (who builds applications). An LLM engineer sits at the intersection of all three.

If you're preparing for this role and you think it's about "writing good prompts," stop. The bar is much higher. This guide covers what interviewers actually test and a month-by-month plan to get there.

What LLM Engineers Actually Do

The day-to-day varies by company, but the core responsibilities are:

Model selection and evaluation — choosing between GPT-4o, Claude 3.5 Sonnet, Llama 3.1 70B, Mistral, and Gemini for a given task. Understanding the trade-offs: cost, latency, context length, multilingual support, instruction following quality. Running evals to make the decision empirically rather than on vibes.

Fine-tuning — adapting a base model or instruction-tuned model to a specific domain or task. Knowing when fine-tuning is worth it versus prompt engineering, when LoRA is enough versus full fine-tuning, and how to evaluate whether the fine-tuned model is actually better.

Inference serving — deploying models in a way that meets latency and cost requirements. Understanding batching, quantization, and serving frameworks.

Evaluation frameworks — building systems to measure LLM performance consistently. This is unglamorous but is what separates companies that improve their LLM products systematically from those that rely on gut feel.

Prompt engineering at scale — not just writing prompts, but building prompt versioning systems, A/B testing prompts, and understanding how prompt changes affect performance across diverse inputs.

Transformer Internals You Must Know

You don't need to implement a transformer from scratch in an interview. You do need to explain it clearly and reason about its properties.

Self-attention — the core operation:

For a sequence of tokens, each token attends to all other tokens. The attention score between token i and token j is:

score(i, j) = softmax(Q_i · K_j / √d_k)

Where Q is the query (what am I looking for?), K is the key (what do I contain?), and V is the value (what do I return?). The output for token i is a weighted sum of all value vectors, weighted by the attention scores.

Why scale by √d_k? As d_k grows, dot products grow in magnitude, pushing softmax into regions with very small gradients. Scaling by √d_k keeps the dot products in a reasonable range.

Multi-head attention: Instead of one set of Q, K, V projections, run h parallel attention operations ("heads") with smaller dimensions (d_k / h), then concatenate and project. Different heads can learn to attend to different relationships — one head might learn syntax, another semantics.

Positional encoding: Transformers process tokens in parallel — there's no inherent notion of sequence order. Positional encodings inject position information. Original transformers used sinusoidal encodings. Modern LLMs use RoPE (Rotary Position Embedding) — it encodes relative position by rotating query and key vectors, which works better for long sequences and generalizes beyond training context lengths.

The KV cache — why LLM inference is different from training:

During training, you process the entire sequence at once and compute attention in parallel (causal masking prevents looking at future tokens). During inference, you generate one token at a time — autoregressive. For each new token, you'd normally recompute attention over all previous tokens: expensive.

KV cache: store the Key and Value matrices computed for all previous tokens. When generating the next token, only compute Q, K, V for the new token, then attend over the new K/V plus all cached K/V pairs. This reduces inference from O(n²) to O(n) per token — at the cost of GPU memory.

Context window limits: Most LLMs today support 128K–1M token contexts. The memory cost of KV cache scales as: 2 × num_layers × seq_len × num_heads × head_dim × bytes_per_element. For a 70B model at 32K context in BF16: ~18GB just for KV cache. This is why very long contexts are expensive.

Mixture of Experts (MoE): A critical architecture to understand in 2026 — GPT-4, Mixtral 8x7B, Grok-2, and Gemini 1.5 all reportedly use MoE. Instead of a single FFN (feed-forward network) in each transformer layer, MoE has multiple FFN "experts" and a lightweight router that selects the top-K experts for each token. Mixtral 8x7B has 8 experts per layer and routes each token to 2 experts — so while the model has 46.7B total parameters, only ~13B are active per token. The benefit: model capacity scales with total parameters (quality), but compute per forward pass scales with active parameters (cost). The engineering challenges: expert load balancing (some experts get overloaded, some underused — requires auxiliary loss terms), all-to-all communication in distributed training (tokens must be routed to the GPU holding their assigned expert), and memory fragmentation. Interview tip: if asked "how would you train a model that's larger than your GPU cluster can fit but you need to keep inference costs low?" — MoE is the answer.

The Fine-Tuning Landscape

Fine-tuning adapts a pre-trained model to a specific task, domain, or behavior. The choices matter more than most candidates realize.

When should you fine-tune vs prompt engineer?

Fine-tune when: the task requires consistent output formatting that's hard to enforce via prompts, you have a specialized domain with terminology the base model doesn't know, you need to remove certain behaviors (safety fine-tuning), or you need to match a specific communication style consistently.

Prompt engineer first when: you don't have > 1000 high-quality labeled examples, the task changes frequently, or you need to ship fast and iterate. Prompt engineering is much cheaper and faster to iterate on.

Full fine-tuning: Update all model weights on your dataset. Best quality, highest cost. Requires GPU memory proportional to model size × 4 (weights + optimizer states + gradients). A 7B model in FP32 full fine-tuning needs ~112GB GPU RAM — not practical without a multi-GPU cluster.

LoRA (Low-Rank Adaptation): Instead of updating all weights, add small trainable rank-r matrices alongside the frozen base weights. For a weight matrix W ∈ ℝ^(d×k), add ΔW = BA where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with r ≪ min(d, k). Typically r=8 or r=16. Trains < 1% of parameters while achieving near-full fine-tune quality on most tasks. Merges back into the base model at inference with zero overhead.

QLoRA: Apply LoRA to a quantized (4-bit NF4) base model. The base model is frozen and quantized (uses ~1/4 the memory), the LoRA adapters train in BF16. This allows fine-tuning a 70B model on a single 80GB A100. Dettmers et al. (2023) showed quality loss from quantization is minimal for fine-tuning.

Instruction tuning (SFT): Supervised fine-tuning on (instruction, response) pairs. This is how base models become chat models. LLaMA 3 base → LLaMA 3 Instruct via instruction tuning on millions of examples.

DPO (Direct Preference Optimization): An alternative to RLHF that's easier to implement. Given pairs of (chosen, rejected) responses for the same prompt, train the model to increase the likelihood of chosen responses and decrease rejected. No reward model needed. Increasingly preferred over RLHF for fine-tuning chat behavior.

RLHF and Alignment

RLHF (Reinforcement Learning from Human Feedback) is how models like GPT-4 and Claude learned to be helpful, harmless, and honest. Understanding it is expected for LLM engineer roles.

The RLHF pipeline:

1. SFT (Supervised Fine-Tuning): Fine-tune the base model on high-quality demonstrations of the desired behavior.
2. Reward Model Training: Collect human preference data — for the same prompt, which of two responses do humans prefer? Train a separate reward model to predict human preference scores.
3. PPO (Proximal Policy Optimization): Use RL to fine-tune the SFT model to maximize the reward model's score, with a KL penalty to prevent the model from drifting too far from the SFT baseline (which would cause it to "hack" the reward model).

Constitutional AI (Anthropic): Instead of human feedback for the reward model, use AI feedback — Claude critiques and revises its own responses according to a set of principles ("the Constitution"). This scales better than human labeling and enables fine-grained control over model behavior.

Why is RLHF hard?

• Reward hacking: the model finds ways to score high on the reward model without actually being better
• Reward model collapse: the reward model's scores stop correlating with human preferences over time
• Mode collapse: the model converges to a narrow set of responses that score well
• High compute cost: PPO requires multiple forward passes per step and is notoriously unstable

DPO was developed specifically to sidestep these problems — it achieves similar alignment without the instability of RL.

Inference and Serving

Getting a model into production with acceptable latency and cost is an engineering discipline of its own.

Key latency components:

• Time to First Token (TTFT): latency from request receipt to first generated token. Affected by: KV cache pre-fill time (processing the input prompt), queue depth (how many requests are ahead of you). Target: < 500ms for interactive applications.
• Time Per Output Token (TPOT): time to generate each subsequent token. Affected by: batch size (larger batches = better GPU utilization but higher TPOT), model size, quantization. Target varies by use case.

Continuous batching (vLLM's approach):

Traditional serving: wait for a fixed batch to fill, process it, return results, start next batch. Problem: short requests finish early but must wait for long ones to complete, wasting GPU capacity.

Continuous batching: as soon as one request in the batch finishes (reaches end-of-sequence), immediately add a new request to fill its slot. The batch is always full — no wasted GPU cycles between batches.

Quantization for serving:

INT8 inference (using bitsandbytes or TensorRT-LLM) reduces memory by 2× vs BF16 with minimal quality loss. INT4/GPTQ reduces by 4× with noticeable but often acceptable quality loss for non-critical applications. The tradeoff: dequantization adds CPU/GPU overhead per layer, which can offset some of the bandwidth savings.

Prompt caching: Both Anthropic and OpenAI support caching the KV cache for repeated prompt prefixes. If your system prompt is 2,000 tokens and you make 10,000 API calls per day, you're paying for those 2,000 tokens 10,000 times. With prompt caching enabled, the first call computes and caches the KV representation of the system prompt; subsequent calls within the cache TTL (5 minutes for Anthropic, 1 hour for OpenAI) reuse it — reducing cost and latency for the cached portion by ~90%. This is a significant practical LLM engineering lever that most candidates don't know about. Interviewers at companies with high API costs ask about it directly.

Serving frameworks:

• vLLM: best throughput for general LLM serving, excellent continuous batching, PagedAttention
• TGI (HuggingFace Text Generation Inference): good integration with HuggingFace Hub models
• TensorRT-LLM (NVIDIA): highest throughput on NVIDIA hardware, but requires compilation overhead
• Ollama: for local/edge deployment, not production serving

Evaluation

"How do you know if your LLM application got better?" is one of the most important questions in the field, and it's asked in interviews because most teams handle it poorly.

Standard benchmarks: MMLU (knowledge), HumanEval (coding), MT-Bench (instruction following), HELM (comprehensive evaluation across tasks). These measure general capability but rarely correlate with your specific use case.

LLM-as-judge: Use a capable LLM (GPT-4o, Claude Opus) to grade outputs on a rubric. Scales well, but has biases: LLMs prefer longer responses, responses similar to their training data, and their own outputs when judging between two models ("self-preference bias"). Mitigation: blind evaluation, swap order of compared responses, use multiple judges.

Task-specific evals: For structured tasks, build deterministic evals — parse the model's output and check it against expected values. For a coding assistant: run the generated code and check if tests pass. For a SQL assistant: run the generated query and compare results. These are high-signal and fast.

Regression testing: Any time you change the model (new version, new fine-tune, new prompt), run your full eval suite before deploying. Improvement on the primary metric should not come at the cost of regressions on other metrics.

12 LLM Interview Questions with Answers

Q1: Explain self-attention in your own words.

Self-attention computes, for each token, a weighted combination of all tokens' values — where the weight between token i and token j represents how relevant token j is to understanding token i. The weights are computed by comparing a query vector (derived from token i) against key vectors (derived from all tokens j), then normalizing with softmax. This allows every token to "look at" every other token in the sequence, enabling the model to capture long-range dependencies that RNNs struggle with.

Q2: What is the difference between LoRA and full fine-tuning?

Full fine-tuning updates all model weights — it's more expressive but requires memory for the full model's optimizer states (8× model size for Adam in FP32). LoRA keeps the original weights frozen and adds small trainable rank-decomposition matrices alongside them. For a 7B model, LoRA trains ~50M parameters instead of 7B — 100× fewer parameters, far less memory and compute. LoRA models can be merged back into the base weights at inference with zero overhead. Quality is close to full fine-tuning for most tasks; it degrades on tasks that require large shifts in model behavior.

Q3: What is QLoRA and when would you use it?

QLoRA applies LoRA adapters on top of a base model quantized to 4-bit NF4 format. The base model stays frozen in 4-bit; the adapters train in BF16. A 70B model in FP16 would need 140GB — impossible on a single GPU. In 4-bit NF4, it needs ~35GB plus ~1GB for the adapters — fits on a single 40GB A100. Use QLoRA when you want to fine-tune a very large model on limited hardware and can accept minor quality loss from quantization.

Q4: How does KV cache enable autoregressive inference?

During generation, the model produces one token at a time. To compute attention for the new token, it needs the Key and Value vectors from all previous tokens. Without caching, these would be recomputed from scratch at every step — O(n) compute per token, O(n²) total. KV cache stores K and V for all previous positions; the new token only computes its own K and V, then attends over the cached + new K/V. This reduces per-token compute to O(1) (constant new computation) at the cost of O(n × d) memory per sequence.

Q5: What is the RLHF pipeline?

Three stages: (1) SFT — fine-tune the base model on high-quality demonstrations to create a strong starting point. (2) Reward model — collect human preference pairs (A vs B responses, A is better) and train a reward model to predict preference scores. (3) PPO — use reinforcement learning to fine-tune the SFT model to maximize the reward model's scores, with a KL divergence penalty to keep the model close to the SFT baseline and prevent reward hacking.

Q6: What is DPO and how does it differ from RLHF?

DPO (Direct Preference Optimization) achieves the same goal as RLHF without training a separate reward model. Given preference pairs (chosen, rejected), DPO directly trains the policy model to increase the probability ratio of chosen over rejected responses. The objective is derived mathematically from the RLHF objective, eliminating the RL training loop. DPO is simpler to implement, more stable to train, and achieves similar alignment quality. Most fine-tuning pipelines now use DPO or its variants (IPO, ORPO) instead of PPO.

Q7: How does Flash Attention reduce memory usage?

Standard attention materializes the full N×N attention matrix in GPU memory (HBM) — O(N²) memory. Flash Attention (Dao et al., 2022) tiles the attention computation into blocks that fit in SRAM (GPU shared memory). It uses a mathematically equivalent but numerically stable online softmax to compute the attention output block by block, never storing the full attention matrix. Memory complexity drops from O(N²) to O(N). The GPU reads from HBM fewer times, dramatically improving throughput for long sequences.

Q8: Explain speculative decoding.

A small, fast draft model generates K candidate tokens. The large target model verifies all K candidates in a single forward pass (it evaluates them in parallel, not sequentially). For each candidate: if the target model would have chosen the same token, accept it. At the first rejected token, sample from the corrected distribution and discard subsequent candidates. The speedup comes from verifying K tokens in one forward pass instead of K sequential passes. Typical speedup: 2–3× with well-matched draft/target model pairs.

Q9: How would you evaluate whether your fine-tuned model is actually better than the base model?

Never rely on a single metric. Run your task-specific eval suite: for coding, pass@k on HumanEval or your internal test suite; for summarization, ROUGE + LLM-as-judge on factual accuracy; for chat, MT-Bench plus blind A/B preference testing. Also run a regression check on general benchmarks (MMLU) to ensure fine-tuning didn't hurt general capability. Finally, check for safety regression — fine-tuning can inadvertently weaken safety guardrails. Shadow deploy and compare on real traffic before full rollout.

Q10: What causes training instability in LLMs and how do you address it?

Common causes: (1) Learning rate too high — use warmup (linear ramp from 0 to peak LR over first 1–3% of steps) and cosine decay. (2) Gradient explosion — use gradient clipping (typical max norm: 1.0). (3) Loss spikes — often caused by batches with very long sequences; use sequence length bucketing and mixed-length batching carefully. (4) BF16 overflow — BF16 has lower precision than FP16; avoid operations that produce very large intermediate values.

Q11: How do you serve an LLM efficiently at 10,000 requests per minute?

Estimate throughput: at 200 output tokens per request, 10K RPM = 167K tokens/sec. A single A100 with a 7B model in INT8 can produce ~4,000 tokens/sec with continuous batching and batch size 64. You need ~42 A100 replicas for steady state. Add 20% headroom for spikes: ~50 A100s. Use a load balancer with consistent hashing (same user → same replica to leverage KV cache for multi-turn). Implement request queuing with priority tiers (paying users first). Monitor queue depth and p99 TTFT — auto-scale if queue depth > 10s of throughput.

Q12: What is positional encoding and why do modern LLMs use RoPE instead of sinusoidal?

Positional encoding adds position information to token embeddings since transformers have no inherent sequence order. Original transformers (Vaswani et al.) used fixed sinusoidal encodings. RoPE (Su et al., 2021) rotates query and key vectors by a rotation matrix that depends on position. This encodes relative positions directly into the attention score — the dot product of Q_i and K_j automatically captures the relative position (i - j). RoPE generalizes better to longer sequences than training time (the rotation structure extrapolates), which is why LLaMA, Mistral, Gemma, and most modern open-source models use it.

4-Month Preparation Plan

Practice on KnowEaseAI

The gap between understanding LLM concepts and answering interview questions about them is larger than it looks. Interview questions test whether you can reason under pressure, apply concepts to new scenarios, and explain trade-offs clearly — not just recall definitions.