AI Inference & Serving Interview Prep
Test & Track Your Readiness

Prefill vs decode phase, autoregressive generation, memory bandwidth vs compute bound regimes, roofline model for LLMs, FLOP counts, and why inference is harder than training.

KV cache sizing formulas, activation memory, parameter memory in fp16/int8/int4, batch size vs sequence length tradeoffs, context length cost curves, and hands-on memory budget worksheets for common models.

Static batching problems, continuous batching (iteration-level scheduling), dynamic batching, PagedAttention mechanics, chunked prefill, the scheduler as the core of a serving engine, and batching for different SLA profiles.

INT8/INT4 quantization theory, post-training quantization vs quantization-aware training, GPTQ algorithm, AWQ activation-aware quantization, GGUF format for CPU/edge, bitsandbytes, calibration datasets, accuracy-compression tradeoffs.

Draft-verify mechanism, acceptance rate math, draft model selection, self-speculative decoding (MEDUSA heads), speculative decoding for different GPU regimes, Lookahead decoding, and when speculation hurts.

vLLM PagedAttention implementation, block manager, prefix caching (hash-based), chunked prefill, async engine, sampling parameters, multi-modal support in vLLM, and extending vLLM with custom attention backends.

TensorRT-LLM plugin system, in-flight batching, fp8 quantization, INT4 AWQ in TRT-LLM, Triton Inference Server integration, engine building pipeline, multi-GPU tensor parallel in TRT-LLM, and Dynamo serving.

Tensor parallelism (column/row linear splits), pipeline parallelism (micro-batching, bubble overhead), sequence parallelism, data parallelism for inference replicas, expert parallelism for MoE models, and Megatron-LM partitioning strategies.

KV cache eviction policies, prefix caching with radix trees, shared prefix across requests, cross-request prefix reuse, paged eviction, KV cache offloading to CPU/NVMe, and KV cache compression (quantized KV).

A100 vs H100 vs H200 vs B100 specs, NVLink vs PCIe bandwidth, memory hierarchy (HBM3, L2, SM registers), CUDA cores vs Tensor Cores, SXM vs PCIe form factors, and sizing hardware for LLM serving workloads.

vLLM vs TGI (Hugging Face) vs TensorRT-LLM vs Triton vs Ollama vs llama.cpp vs SGLang — feature matrix, latency benchmarks, quantization support, hardware requirements, multi-modal support, and when to use each.

P50/P90/P99 latency, TTFT vs TPOT, throughput (tokens/sec), the latency-throughput Pareto frontier, SLA design for interactive vs batch workloads, max batch size for a given P99, and load testing methodology.

ONNX export pipeline, CoreML conversion, TFLite, GGUF for CPU inference, Apple Neural Engine, quantization for mobile (INT4/INT2), model pruning, knowledge distillation for edge, and ARM NEON vs GPU on mobile.

Embedding inference is different: batch strategies, dynamic batching for variable-length inputs, ANN index updates, embedding cache, cross-encoder reranker serving, hardware sizing for embedding workloads, and combined retrieval+generation serving.

OpenTelemetry gen_ai.* semantic conventions, tracing prefill/decode phases, token-level latency instrumentation, cost attribution per request/user/feature, GPU utilization metrics, and building inference dashboards in Grafana.

Running tensor-parallel models across multiple GPUs, load balancing strategies (round-robin vs least-loaded vs session affinity), replica autoscaling, failover and model health checks, NVLink topology impact on serving.

Cost per token analysis, spot instance inference, preemptible GPU pools, mixed-precision inference, model tiering (large + small model routing), prompt caching economics, and the cost structure of hosted vs self-hosted inference.

Blue-green model deployment, canary releases for model updates, rollback strategies, model artifact versioning, serving health checks, capacity planning, incident response for serving outages, and the LLM serving reliability ladder.