Evals Engineering Interview Prep
Test & Track Your Readiness

The eval pyramid, offline vs online evals, the 'eval flywheel', why traditional software testing fails for LLMs, anatomy of an eval suite, and how to design an eval-first development culture.

Golden set construction, data contamination detection, dynamic benchmarks, slice-based evaluation, distribution shift, human vs synthetic labels, and canonical benchmark critique (MMLU, HumanEval).

Limitations of n-gram metrics, BERTScore mechanics, task-specific metric design, composite scoring, inter-rater reliability, calibration, and when to build custom metrics vs use off-the-shelf.

Using LLMs to evaluate LLMs: positional bias, verbosity bias, self-consistency, pairwise vs pointwise scoring, calibration against human labels, judge model selection, and when LLM-as-judge fails.

RAGAS metrics (faithfulness, answer relevancy, context recall, context precision), DeepEval framework internals, TruLens, Promptfoo, running evals at scale, custom metric plugins, and framework comparison.

RLHF data collection architecture, preference annotation interfaces, inter-annotator agreement (Cohen's kappa, Krippendorff's alpha), annotator calibration, adversarial quality control, and scaling annotation.

Generating eval datasets with LLMs, quality control via LLM-as-judge, diversity sampling, adversarial synthetic data, contamination risks in LLM-generated evals, and domain-specific synthetic benchmark creation.

Eval-driven development (EDD), snapshot eval testing, CI/CD pipelines for LLM changes, regression budgets, flaky eval management, eval caching, pre-commit eval hooks, and golden test suites.

Harm taxonomies (HELM, ToxiGen), jailbreak evaluation, refusal calibration, adversarial prompt datasets, automated red-teaming with Garak, Constitutional AI eval, bias benchmarks, and safety eval reporting.

Tool use correctness, task completion rate, loop termination quality, trajectory-level evaluation, SWE-Bench methodology, TAU-bench, multi-agent eval challenges, and process reward models vs outcome reward models.

End-to-end RAG eval: retrieval quality (MRR, NDCG, recall@k), generation faithfulness, answer correctness, context utilization, chunk-level attribution, pipeline-level vs component-level eval, and RAGAS in production.

Vision-language eval benchmarks (MMMU, MMBench, SEED-Bench), perceptual quality metrics (FID, IS, CLIP score), visual QA grading, image generation evaluation, audio ASR/TTS metrics, and eval challenges unique to multimodal.

A/B testing for model changes, shadow deployment eval, continuous eval pipelines, statistical significance, user feedback loops, embedding drift detection, and building a live eval dashboard.

Architecture of internal eval platforms, Braintrust deep dive, Weights & Biases Weave, MLflow eval tracking, dataset versioning, eval result storage, async eval workers, cost management, and multi-team eval governance.

Real production eval failures: when LLM judges disagree with users, regression blindspots, benchmark gaming, silent model degradation, the 'eval gap', and lessons from eval systems at Anthropic, OpenAI, Google, and startups.