LLM as judge, what the numbers actually say

An automated LLM judge decided whether our production prompts shipped. I had implemented it because industry write-ups reported that LLM evaluators agreed with human annotators roughly 80% to 85% of the time.

Six weeks later, a reported accuracy improvement turned out to be the evaluator flipping verdicts on a rerun.

A judge that agrees with human raters 85% of the time is not 85% accurate. It is 85% concordant with a group of humans that only agree with each other roughly 80% of the time on subjective criteria.

Read the primary papers and the headline agreement percentage turns out to be hiding the statistical adjustment that matters.

Chapter 0: What agreement metrics actually measure

The 85% agreement figure originates from the MT-Bench and Chatbot Arena benchmark (Zheng et al., NeurIPS 2023). The dataset spans 80 multi-turn queries across 8 categories with 3,000 expert votes and 30,000 crowdsourced pairwise judgments.

Excluding ties, GPT-4 pairwise judgments agreed with human experts 85% of the time.

The accompanying baseline metric is frequently omitted: human-to-human agreement was 81% to 82%.

The judge matches the concordance rate between two independent human annotators. It does not measure ground-truth correctness, because open-ended generation has no deterministic label to be correct against. Two domain experts disagree on roughly one subjective rating in five. The judge models the human preference distribution, and its ceiling is the clarity of your rubric.

Step 1: Apply chance correction via Cohen's Kappa

Raw percentage agreement inflates apparent reliability when the underlying label distribution is imbalanced.

In large-scale empirical studies evaluating LLM judges across multiple providers and benchmarks (such as arXiv 2606.19544), kappa deflation is consistent across models. Between raw percentage agreement and Cohen's kappa on MT-Bench, agreement scores drop by 33 to 41 percentage points.

Raw agreement credits instances where both the evaluator and the human select the winning candidate simply because one option dominates the dataset. Cohen's kappa ($\kappa$) subtracts the probability of agreement occurring by chance:

where $P_o$ is observed agreement and $P_e$ is expected chance agreement.

Krippendorff's alpha ($\alpha$) extends this formulation to multi-annotator panels and missing data, setting standard statistical reliability thresholds at $\alpha \ge 0.800$.

The diagnostic check: if your evaluation pipeline reports only raw percentage agreement, you are measuring label class imbalance rather than true scoring reliability.

Step 2: Report metrics per evaluation criterion

Aggregating scores across different evaluation criteria conceals severe per-task variance.

In clinical and technical evaluation studies, aggregate kappa scores of 0.74 frequently mask individual sub-criteria spanning from 0.34 (barely above chance) to 0.97 (high consistency).

# agreement.py
from collections import Counter
 
def cohens_kappa(a: list[str], b: list[str]) -> float:
    """Chance-corrected agreement between two raters over identical items."""
    if len(a) != len(b) or not a:
        raise ValueError("Rater vectors must be non-empty and of equal length.")
 
    n = len(a)
    observed = sum(1 for x, y in zip(a, b) if x == y) / n
    ca, cb = Counter(a), Counter(b)
    expected = sum((ca[k] / n) * (cb[k] / n) for k in set(a) | set(b))
    if expected == 1.0:
        return 1.0
    return (observed - expected) / (1 - expected)
 
def per_criterion_report(labels: dict[str, tuple[list[str], list[str]]]) -> dict:
    """Computes kappa and raw agreement per criterion separately."""
    out = {}
    for criterion, (human, judge) in labels.items():
        raw = sum(1 for x, y in zip(human, judge) if x == y) / len(human)
        kappa = cohens_kappa(human, judge)
        out[criterion] = {
            "raw_agreement": round(raw, 3),
            "kappa": round(kappa, 3),
            "deflation_gap": round((raw - kappa) * 100, 1),
        }
    return out

Disaggregating tells you which criterion the judge is unreliable on, which is the one to rewrite.

The diagnostic check: if you report one number for the whole rubric, you cannot tell a judge that is wrong everywhere from a judge that is wrong on one criterion.

Step 3: Evaluate pairwise candidates in both presentation orders

Position bias occurs when the evaluator systematically prefers the first or second candidate passage regardless of content.

Empirical measurements of consistency when swapping candidate positions ($A/B$ vs. $B/A$):

A consistency score of 50% indicates that reversing presentation order flips the verdict half the time.

# position_bias.py
from dataclasses import dataclass
from itertools import product
 
@dataclass
class PairJudgment:
    task_id: str
    order: str  # "ab" or "ba"
    winner: str  # "a", "b", or "tie"
 
def calculate_consistency(judgments: list[PairJudgment]) -> float:
    """Calculates fraction of tasks where swapping presentation order did not alter winner."""
    by_task: dict[str, dict[str, str]] = {}
    for j in judgments:
        by_task.setdefault(j.task_id, {})[j.order] = j.winner
 
    complete = [v for v in by_task.values() if "ab" in v and "ba" in v]
    if not complete:
        raise ValueError("No tasks evaluated in both presentation orders.")
 
    agree = sum(1 for v in complete if v["ab"] == v["ba"])
    return agree / len(complete)
 
def generate_swapped_pairs(tasks: list[str], answers: dict[str, tuple[str, str]]):
    """Emits every candidate pair in both presentation orders."""
    for task_id, order in product(tasks, ("ab", "ba")):
        a, b = answers[task_id]
        yield task_id, order, (a, b) if order == "ab" else (b, a)

The diagnostic check: evaluate every test pair in both orders. If the verdicts disagree, mark the comparison as unresolved rather than taking an arbitrary single-order result.

Step 4: Supply explicit reference answers

Putting a verified reference answer in the judge prompt cuts grading error rates:

Grounding the evaluation against a known golden response shifts the task from unconstrained open-ended generation to direct semantic verification.

Step 5: Embed adversarial canary probes

Research on reward model vulnerabilities demonstrates that content-free text strings (such as a single colon, a period, or standard boilerplate prefixes like "Thought process:") can elicit false-positive passing scores from uncalibrated evaluators.

Include content-free test cases in evaluation suites to catch prompt degradation:

# canaries.py
CANARY_PROBES = [
    ":",
    ".",
    "Thought process:",
    "Let's solve this problem step by step.",
]
 
def audit_canary_responses(judge_fn, test_tasks: list[dict]) -> dict:
    """Verifies whether the evaluator awards passing scores to empty or meaningless inputs."""
    false_positives = []
    for task in test_tasks:
        for probe in CANARY_PROBES:
            verdict = judge_fn(question=task["question"], candidate_answer=probe)
            if verdict.get("passed"):
                false_positives.append({"task_id": task["id"], "probe": probe})
 
    return {
        "total_probes": len(test_tasks) * len(CANARY_PROBES),
        "false_positive_count": len(false_positives),
        "suite_valid": len(false_positives) == 0,
    }

If the evaluator passes any empty canary probe, the scoring threshold is invalid and should not gate deployment pipelines.

Step 6: Multi-model panels and cost routing

Using a panel of diverse smaller models (PoLL architecture) from disjoint model families (such as mixing Claude-3.5-Haiku, GPT-4o-mini, and Llama-3.1-8B) prevents shared family biases while reducing evaluation costs relative to a single frontier model.

Run the deterministic verifications first: JSON schema validators, unit tests, regex matchers. Send the judge only the properties that no compiler can check.

When this is the wrong choice

• The property has a deterministic check. Schema validity, a passing test, an exact-match answer. A validator answers those exactly, for free, every time. A judge answers them at a cost, with the position bias and rerun variance described above.
• The rubric is still moving. Kappa measures agreement against a fixed rubric. Score against a rubric you rewrote last week and the change you observe is rubric drift, not a model improvement, which is the same mistake as reading a rerun flip as progress.
• You can only afford one presentation order. Order-swapped evaluation doubles judge calls. At the consistency rates in the table above, single-order verdicts on close pairs are near noise, so a smaller human-reviewed sample buys more information than a larger automated one.