How to Evaluate LLM Applications — RAGAS, LLM-as-Judge, and Production Eval Pipelines

Your RAG system works in development. It answers questions correctly on your test queries. You deploy it. Three weeks later, a user reports the chatbot cited a return policy of 14 days when it's 30. Your faithfulness score was 0.92 on the development set. What happened?

This is the most common failure pattern in production LLM applications — not dramatic hallucinations obvious testing would catch, but quiet failures on real-world inputs your development set didn't represent. The evaluation framework you build determines whether you catch these before users do or after.

Evaluation for LLM applications has three distinct problems most teams conflate: measuring quality before deployment, catching regressions when you change something, and detecting drift after deployment. These require different tooling, different metrics, and different thresholds. Treating them as one problem leads to either excessive manual review that doesn't scale, or automated evals that miss the failures that matter.

The 3-Layer Evaluation Stack

Good evaluation isn't a single pipeline. It's three pipelines serving three different purposes, each with a different trigger and a different gate.

Layer 1 — Offline Eval (pre-deployment)
  ├─ Fixed labeled dataset
  ├─ Runs before every deployment
  └─ Gate: score below threshold = no deploy

Layer 2 — CI Regression (on every PR)
  ├─ Golden dataset (200–500 examples)
  ├─ Triggers on: prompt changes, model version bumps,
  │  retrieval config changes, chunking strategy changes
  └─ Gate: >2% drop on primary metric = block merge

Layer 3 — Production Monitoring (continuous)
  ├─ 5–10% sample of live traffic
  ├─ Scored by automated evaluator
  └─ Alert: weekly metric drift OR individual outlier rate spike

Most teams implement Layer 1 and skip Layers 2 and 3. Every failure Layer 1 misses is a failure that reaches users.

RAGAS — What Each Metric Actually Measures

RAGAS (Retrieval-Augmented Generation Assessment Suite) is the de-facto evaluation framework for RAG systems in production. It computes eight metrics, but four carry 90% of the diagnostic value.

Faithfulness — is the generated answer grounded in the retrieved context?

The mechanism matters more than the name. RAGAS doesn't compute semantic similarity between answer and context. It decomposes the generated answer into individual factual claims using an LLM call, then checks each claim against the retrieved context with a second LLM call.

answer = "The return policy is 30 days and applies to all products."

# Step 1: RAGAS decomposes answer into atomic claims
claims = [
    "The return policy is 30 days",         # claim A
    "The policy applies to all products"     # claim B
]

# Step 2: For each claim, ask: can this be inferred from context?
# context = ["Returns accepted within 30 days for most items..."]

# claim A → SUPPORTED    (score: 1)
# claim B → NOT SUPPORTED — context says "most items", not "all products"
#                            (score: 0)

faithfulness = supported_claims / total_claims
# faithfulness = 1 / 2 = 0.50

A faithfulness score of 0.50 here is correct — and the claim decomposition surfaces the exact bug. A similarity score would have missed it entirely.

Context Precision — of all the chunks retrieved, how many were actually relevant?

This diagnoses your retriever, not your generator. High context precision means you're retrieving signal. Low context precision (retrieving 8 chunks when only 1 was needed) inflates token cost, dilutes generator attention, and increases hallucination risk.

retrieved_chunks = [chunk_1, chunk_2, chunk_3, chunk_4, chunk_5]
relevant_chunks  = [chunk_1, chunk_3]  # only these contributed to the answer

# Precision@K — weighted by rank position
# Higher-ranked relevant chunks score more than lower-ranked ones
context_precision = (precision_at_1 + precision_at_2 + ...) / total_retrieved
# Simplified: 2 relevant out of 5 retrieved = 0.40 baseline

Context Recall — did the retriever find all the information needed?

Requires a reference answer (ground truth). Checks whether all facts in the ground truth are findable in the retrieved context. Without labeled data, skip this metric and weight context precision more heavily.

Answer Relevance — does the answer actually address the question?

RAGAS generates reverse questions from the answer and measures cosine similarity to the original question. An answer that correctly uses context but answers a different question scores low here. Catches the subtle failure where retrieval and generation both work but the answer is off-topic.

Production thresholds

| Metric | Alert | Investigation | |---|---|---| | Faithfulness | < 0.75 | < 0.65 | | Context Precision | < 0.70 | < 0.60 | | Context Recall | < 0.80 | < 0.70 | | Answer Relevance | < 0.80 | < 0.70 |

These are starting points. Calibrate by manually rating 50 real queries and finding the score that separates acceptable from unacceptable in your specific domain.

LLM-as-Judge — When It Works and When It Lies

RAGAS uses LLM-as-judge internally. You'll use it for custom dimensions: tone compliance, domain-specific factual accuracy, output format correctness, safety checks. It's more consistent than human annotation at scale and costs cents per evaluation. It also has four well-documented failure modes you need to engineer around.

Positional bias: When comparing two responses, LLM judges prefer the first option at statistically significant rates, independent of quality. In A/B prompt testing, always randomize which variant appears first and average scores across both orderings.

Verbosity preference: Longer answers score higher regardless of information density. A judge rating "helpfulness" will rate a 300-word response higher than a 150-word response with identical content. Fix with an explicit rubric that includes examples of short high-quality responses.

Self-preference: A model from the same family scores its own outputs 5–8 points higher than a cross-family judge would. When using Claude to judge Claude outputs, validate your judge calibration against human scores quarterly — or use a different model family as the judge entirely.

Sycophancy on ground truth: If your judge sees the expected answer, it confirms it rather than evaluating independently.

# Anti-pattern — exposes ground truth, produces inflated scores
prompt = f"""
Expected answer: {ground_truth}
Model response: {response}
Does the response match? Score 0-1.
"""

# Correct pattern — judge evaluates independently
prompt = f"""
Question: {question}
Context provided to model: {context}
Model response: {response}

Evaluate ONLY whether the response is factually grounded
in the provided context. Do not use external knowledge.
Think step by step, then give a score from 0.0 to 1.0.
Format: reasoning: <your reasoning> | score: <0.0-1.0>
"""

The "think step by step before scoring" instruction meaningfully reduces positional bias and verbosity preference — the chain-of-thought forces the judge to reason about the content rather than pattern-match on surface features.

Building the CI Regression Harness

The regression harness is the Layer 2 gate — the automated check that runs on every PR touching a prompt, model version, or retrieval config. Its job is to catch regressions before they deploy, not to evaluate every edge case.

The golden dataset is the foundation. It should cover: your most common query types (sample from production logs weighted by frequency), edge cases that caused past failures (every production incident becomes a test case), and adversarial inputs (partial queries, mixed languages, queries with numbers that require precise retrieval).

200–500 examples is the practical range. Below 200, statistical noise dominates and a real regression looks like random variance. Above 500, the cost per CI run becomes a friction that makes engineers skip it.

class RegressionHarness:
    def __init__(self, golden_dataset, scorers, thresholds):
        self.dataset = golden_dataset   # [{query, context, expected?}]
        self.scorers = scorers          # RAGAS + custom judges
        self.thresholds = thresholds    # {metric: min_acceptable_mean}

    def run(self, system_fn) -> HarnessResult:
        results = []
        for example in self.dataset:
            response = system_fn(example["query"])
            scores = {
                scorer.name: scorer.score(
                    query=example["query"],
                    context=example["context"],
                    response=response,
                    reference=example.get("expected")  # optional
                )
                for scorer in self.scorers
            }
            results.append(scores)

        aggregate = compute_percentiles(results)  # mean, p50, p90 per metric
        failures = [
            metric for metric, threshold in self.thresholds.items()
            if aggregate[metric]["mean"] < threshold
        ]
        return HarnessResult(aggregate=aggregate, failures=failures)

    def compare(self, baseline: HarnessResult,
                candidate: HarnessResult,
                max_regression: float = 0.02) -> list[tuple]:
        """Returns list of (metric, delta) for any metric that regressed."""
        regressions = []
        for metric in baseline.aggregate:
            delta = (candidate.aggregate[metric]["mean"]
                     - baseline.aggregate[metric]["mean"])
            if delta < -max_regression:
                regressions.append((metric, round(delta, 4)))
        return regressions  # empty = pass, merge allowed

Wire compare() into your PR pipeline. A drop of >2% on any metric surfaces the specific metric name and delta — not just "eval failed." Engineers fix regressions faster when they know exactly which metric broke and by how much.

Production Monitoring — Catching Drift Before Users Do

Production monitoring is different from the regression harness. You're not comparing against a baseline — you're watching for continuous drift in a live system that changes even when you change nothing (model provider updates, input distribution shift, document index staleness).

Three signals to monitor weekly:

Score drift: Mean faithfulness and answer relevance across your 5–10% sample. A 3-point drop over two weeks with no deployment is almost always either input distribution shift (users started asking different questions) or a silent model update from your API provider. Both require investigation.

Embedding drift: Compute embeddings of incoming queries weekly. Track mean cosine distance to your training distribution. A shift of >0.15 from baseline indicates users are asking meaningfully different questions than you designed for — time to expand your golden dataset and revisit chunking strategy.

Outlier rate: Flag individual responses where faithfulness < 0.50 or answer relevance < 0.60. If >5% of sampled responses hit these thresholds, there's a systematic failure — a document type the chunker handles poorly, a query category the retriever consistently misses.

# Runs as a weekly scheduled job
def weekly_eval_job(sample_log, evaluator, baseline_scores, alert_fn):
    sampled = sample_log.get_week_sample(rate=0.05)
    current = evaluator.batch_score(sampled)

    # 1. Score drift check
    for metric, baseline_mean in baseline_scores.items():
        delta = current.metrics[metric].mean - baseline_mean
        if delta < -0.03:
            alert_fn(f"DRIFT: {metric} dropped {delta:+.3f} vs baseline")

    # 2. Outlier rate check
    hard_failures = [
        r for r in current.individual
        if r.faithfulness < 0.50 or r.answer_relevance < 0.60
    ]
    outlier_rate = len(hard_failures) / len(sampled)
    if outlier_rate > 0.05:
        alert_fn(
            f"OUTLIER RATE: {outlier_rate:.1%} of sampled responses "
            f"below hard threshold — {len(hard_failures)} cases queued for review"
        )

Evaluating Agents — Different Dimensions, Different Stakes

Everything above applies to RAG pipelines. Agents require additional dimensions because you're evaluating behavior over a sequence of actions, not a single input-output pair.

Task completion rate replaces faithfulness as the primary metric. Did the agent accomplish what was asked? Requires a binary or partial-credit scoring function specific to your task domain.

Safety rate is non-negotiable for any agent with write access. Did the agent take an irreversible action — send a message, make a purchase, delete a record — without explicit user confirmation? Run safety checks on 100% of traffic, not a sample. A 0.1% safety failure rate is 10 unsafe actions per 10,000 runs.

Step efficiency: Track mean tool calls per successful task completion. An agent calling a search tool 9 times when 2 would suffice is burning cost and signals a prompt or tool definition problem. Alert on upward drift in this metric week-over-week.

Cost per task: At 50,000 agent runs per day, a 15% increase in mean steps from a prompt change costs thousands of dollars before anyone notices if cost is not monitored per-run.

AgentEvalDimensions = {
    "task_completion_rate":  {"sample_rate": 1.0,   "alert_threshold": 0.90},
    "safety_rate":           {"sample_rate": 1.0,   "alert_threshold": 0.999},
    "mean_steps_per_task":   {"sample_rate": 0.20,  "alert_direction": "up"},
    "cost_per_task_usd":     {"sample_rate": 0.20,  "alert_direction": "up"},
}

Eval Anti-Patterns Worth Knowing

Evaluating only on development queries. The queries you used to build the system are the ones it performs best on. Production traffic has a different distribution. If you can't eval on real user data, build synthetic variations: paraphrase golden queries, add typos, try partial queries, mix languages. Cover the input space, not just the happy path.

Same model family for generation and judging. Claude judging Claude outputs inflates scores due to self-preference. Use a different family as your judge, or rely on RAGAS's non-judge metrics (context precision, faithfulness via claim decomposition) for the dimensions you care about most.

Treating a passing eval as a deploy signal. A passing regression harness means "no known failure modes regressed." It doesn't mean "no new failure modes exist." Pair automated evals with a 1% random human review rate to catch failure patterns your harness doesn't yet cover — and add every discovered failure to the golden dataset immediately.

Setting thresholds without calibration. A faithfulness threshold of 0.75 is a reasonable default. In a medical information system, it should be 0.92. In a casual recommendation chatbot, 0.68 might be acceptable. Calibrate against human ratings in your domain before setting production alert thresholds.