LLM as judge, what the numbers actually say

The judge decided whether a change shipped, which is a lot of weight to hang on a component I'd adopted because the blog posts said it agreed with humans about 80% of the time. It carried that weight for about six weeks. Then I reported an improvement that turned out to be the judge flipping on rerun, after the number had already been passed on to someone who took it at face value, and correcting that in writing is a specific kind of unpleasant. That sent me back to the primary sources, where the headline figure turns out to be doing far less work than the way it gets quoted suggests.

Why the 80% figure is where the thinking usually stops

The number comes from MT-Bench and Chatbot Arena, Zheng et al., NeurIPS 2023 Datasets and Benchmarks track, arXiv 2306.05685. Worth knowing the shape of the data before quoting the result: MT-Bench is 80 multi-turn questions across 8 categories with 3,000 expert votes from 58 expert labelers, and Chatbot Arena contributed 30,000 conversations judged by 2,114 unique crowdsourced judges.

Excluding ties, GPT-4 pairwise agreed with human experts 85% of the time, and single-answer grading on the second turn agreed 84%. Those are the numbers everyone repeats. The number almost nobody repeats sits right next to them in the same paper: human-human agreement was 81 to 82%.

That comparison is the whole point, and getting it backwards is the most common error in judge writing. GPT-4 matches human-to-human agreement. It does not match ground truth, because on these tasks there is no ground truth to match. Two qualified humans disagree about one answer in five. The judge is not approximating a correct answer, it is approximating a distribution of human opinion, and the ceiling on that is your annotation guideline. If your guideline is vague, your humans will disagree, and a judge that perfectly reproduced your humans would still look unreliable.

I found this clarifying and slightly deflating. A large chunk of "the judge is unreliable" turns out to be "my rubric is underspecified".

The biases, each with its measured number

The same paper measures three biases, and the measurements are more interesting than the category names.

Position bias, measured as consistency after swapping the order of the two answers: GPT-4 was consistent 65.0% of the time with the default prompt and 66.2% with a renamed prompt. GPT-3.5 scored 46.2% and 51.2%. Claude-v1 scored 23.8% and 56.2%.

Read that last pair again. Claude-v1's consistency more than doubled from merely renaming the assistants in the prompt. Nothing about the answers changed, nothing about the task changed, and the judge became a different judge. That is the sharpest prompt-sensitivity result in the paper and the reason I now treat any judge prompt as a component with its own version number.

Verbosity bias, tested with a repetitive-list attack where an answer is made longer without adding information, across 23 answers: Claude-v1 fell for it 91.3% of the time, GPT-3.5 also 91.3%, and GPT-4 8.7%. The spread there is enormous, and it says the bias is a property of specific models rather than of the judging paradigm.

Self-enhancement bias: GPT-4 favoured its own outputs with roughly a 10% higher win rate, Claude-v1 by roughly 25%. The paper is careful to say this is suggestive rather than causally isolated, since the models also differ in ways that could explain part of it, and I'll be equally careful here.

# position_bias.py
from dataclasses import dataclass
from itertools import product
 
 
@dataclass
class PairJudgment:
    """One judgment plus the order the answers were shown in."""
 
    task_id: str
    order: str  # "ab" or "ba"
    winner: str  # "a", "b", or "tie"
 
 
def consistency(judgments: list[PairJudgment]) -> float:
    """Fraction of tasks where swapping the order did not change the winner.
 
    This is the metric MT-Bench reports. It is not accuracy: a judge can be
    perfectly consistent and perfectly wrong. Consistency below roughly 0.7
    means the judge cannot resolve small differences at all, because the
    presentation order is a larger signal than the quality difference.
    """
    by_task: dict[str, dict[str, str]] = {}
    for j in judgments:
        by_task.setdefault(j.task_id, {})[j.order] = j.winner
 
    complete = [v for v in by_task.values() if "ab" in v and "ba" in v]
    if not complete:
        raise ValueError("no task was judged in both orders")
 
    agree = sum(1 for v in complete if v["ab"] == v["ba"])
    return agree / len(complete)
 
 
def build_swapped_runs(tasks: list[str], answers: dict[str, tuple[str, str]]):
    """Emit every task twice, once in each presentation order.
 
    Running only one order is the single most common evaluation bug I see.
    It hides position bias completely, because there is nothing to compare.
    """
    for task_id, order in product(tasks, ("ab", "ba")):
        a, b = answers[task_id]
        yield task_id, order, (a, b) if order == "ab" else (b, a)

The mitigation ladder that actually has numbers behind it

Table 4 of the same paper is the most practically useful thing in it. On math questions, across 20 judgments, the default prompt produced a 70% failure rate. Adding chain-of-thought brought it to 30%. Adding reference-guided grading, where the judge sees a reference answer, brought it to 15%.

Seventy to thirty to fifteen. That progression is the cleanest published argument for reference-guided grading, and it's why I now consider a judge without a reference answer to be a fallback rather than a design. Generating a reference is work, but it is work you do once per task, not once per evaluation run.

Reliability without validity, which is the part that changed my mind

The most differentiating material I found is a June 2026 preprint, "Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models", arXiv 2606.19544. Flag for anyone about to cite it: this is a v1 preprint and has not been peer reviewed. Treat the findings as strong signal, not settled fact.

The scale is what makes it worth reading. 21 judges from 9 providers, evaluated across 3 benchmarks and 3 protocols, 118 runs, roughly 541,000 individual judgments.

The headline finding, and the reason I rebuilt my evaluation: kappa deflation is universal. Between exact-match agreement and Cohen's kappa on MT-Bench, the gap runs 33 to 41 percentage points. Every claim of the form "our judge agrees with humans 85% of the time" overstates reliability by 30 to 40 points once you correct for chance agreement. Raw agreement counts the cases where the judge and the human both said "A wins" because A wins most of the time in your dataset. Kappa does not.

If you take one thing from this post, take that. The 85% is real, and the reliability it implies is not.

Three more findings from the same preprint, each of which contradicts something I believed.

Judge rankings shift by up to 14 positions across benchmarks. A judge ranked near the top on one benchmark can land near the bottom on another. Picking a judge off a leaderboard is therefore unsound, and I had done exactly that.

The consistency-bias paradox: production judges with test-retest reliability above 0.95 simultaneously show position bias above 0.10. The most reproducible judges are among the least valid ones. A judge that returns the same answer every time is reproducing its own biases faithfully, which is what reproducibility measures and all it measures. I had been using rerun stability as my proxy for quality, and signing off on results with it.

And a contrarian one worth carrying: verbosity bias measured below 0.011 across the cohort under a single pairwise rubric. Against the 2023 numbers where two of three models fell for the repetitive-list attack more than 90% of the time, that suggests verbosity bias has largely been trained out, while position bias has not. The standard bias listicle that treats them as equally live concerns is out of date on one of them.

Content-free answers that score well

Separate from the bias literature, "One Token to Fool LLM-as-a-Judge" (arXiv 2507.08794) demonstrates that content-free "master keys" elicit false-positive rewards. The examples are as bare as they sound: a lone colon, a full stop, the phrase "Thought process:", and "Let's solve this problem step by step." No answer, no reasoning, no content. Rewards anyway, across GPT-o1, Claude-4, and multiple model scales.

The paper's mitigation is Master-RM, a reward model trained with truncated outputs as adversarial negatives so that a plausible-looking prefix with nothing behind it gets scored as the failure it is. The abstract gives no per-model false-positive rates, so I'm not going to put a figure on how often any specific model falls for it.

What I do now is keep a small canary set of content-free answers in every judge evaluation. If the judge passes any of them, the judge is broken, and nothing else I measure that day matters.

# canaries.py
MASTER_KEYS = [
    ":",
    ".",
    "Thought process:",
    "Let's solve this problem step by step.",
]
 
 
def canary_report(judge, tasks: list[dict]) -> dict:
    """Score content-free answers and count how many the judge passes.
 
    Any pass here is disqualifying. A judge that rewards an empty prefix is
    not measuring answer quality, it is measuring surface plausibility, and
    every other number it produces inherits that.
    """
    failures = []
    for task in tasks:
        for key in MASTER_KEYS:
            verdict = judge(question=task["question"], answer=key)
            if verdict.get("pass"):
                failures.append({"task": task["id"], "key": key})
 
    total = len(tasks) * len(MASTER_KEYS)
    return {
        "trials": total,
        "false_positives": len(failures),
        "examples": failures[:10],
        "usable": len(failures) == 0,
    }

Reporting agreement without fooling yourself

Cohen's kappa is the minimum bar, and the Landis and Koch (1977) bands are the usual reading: 0.21 to 0.40 fair, 0.41 to 0.60 moderate, 0.61 to 0.80 substantial, 0.81 to 1.00 almost perfect. Those bands are conventions rather than laws, and Landis and Koch themselves offered them as arbitrary but useful. Krippendorff's alpha handles more than two raters and missing data, with the conventional cutoffs of 0.800 for reliable and 0.667 for tentative conclusions only.

Reference points make those numbers concrete. MT-Bench human-human agreement of 81 to 82% raw is the realistic ceiling for a subjective task. MAST's human annotators reached kappa 0.88 on a failure taxonomy, which is what a tight guideline looks like. And a clinical LLM-jury study reported kappa 0.74 against human consensus, while per-criterion kappa ranged from 0.34 to 0.97.

That last range is the reason I report per criterion and have stopped reporting an aggregate at all. A headline 0.74 hides a criterion sitting at 0.34, which in Landis and Koch terms is fair agreement, meaning barely better than the coin. Averaging is how you ship a judge that is excellent at four things and useless at the fifth without ever finding out which is which.

# agreement.py
from collections import Counter
 
 
def cohens_kappa(a: list[str], b: list[str]) -> float:
    """Chance-corrected agreement between two raters over the same items.
 
    The gap between this and raw agreement is the whole point. On skewed
    label distributions raw agreement can look excellent while kappa sits
    near zero, because both raters are mostly just guessing the majority.
    """
    if len(a) != len(b) or not a:
        raise ValueError("rater vectors must be non-empty and equal length")
 
    n = len(a)
    observed = sum(1 for x, y in zip(a, b) if x == y) / n
    ca, cb = Counter(a), Counter(b)
    expected = sum((ca[k] / n) * (cb[k] / n) for k in set(a) | set(b))
    if expected == 1.0:
        return 1.0
    return (observed - expected) / (1 - expected)
 
 
def per_criterion_report(labels: dict[str, tuple[list[str], list[str]]]) -> dict:
    """Kappa for every criterion separately, plus raw agreement alongside.
 
    Never collapse this to a mean. The point of the table is to expose the
    criterion where the judge falls apart, and a mean is designed to hide it.
    """
    out = {}
    for criterion, (human, judge) in labels.items():
        raw = sum(1 for x, y in zip(human, judge) if x == y) / len(human)
        kappa = cohens_kappa(human, judge)
        out[criterion] = {
            "raw_agreement": round(raw, 3),
            "kappa": round(kappa, 3),
            "deflation_points": round((raw - kappa) * 100, 1),
        }
    return out

That deflation_points field is there because I wanted to see the 33 to 41 point gap on my own data rather than take it on faith. I saw it.

Cheaper and better options than one big judge

Replacing a single large judge with a panel of smaller ones is the alternative with the best cost story. PoLL, "Replacing Judges with Juries" (arXiv 2404.18796), uses a panel drawn from disjoint model families, which is the load-bearing detail: models from the same family share biases, so a panel of three siblings mostly amplifies one opinion. The panel is reported as more than 7 times less expensive than a single large judge and less prone to intra-model bias.

A few other reference points worth knowing by name. G-Eval, with a GPT-4 backbone, reports Spearman 0.514 with humans on summarization, and its own authors flag that it shows a preference for LLM-generated text. Prometheus, a 13B fine-tuned evaluator, reports Pearson 0.897 with human evaluators across 45 rubrics, against GPT-4's 0.882 and ChatGPT's 0.392 on the same comparison, which is a striking result for a model that size. And JUDGE-BENCH, spanning 20 datasets and 11 LLMs, lands on the conclusion that works as the counterweight to every 80% headline: variance across models and datasets is substantial, and judges "should be carefully validated against human judgments before being used as evaluators".

Rubric shape, where I have an opinion rather than a citation

My practitioner preference is binary pass/fail over Likert scales, and the sourcing here is weaker than everything above, so treat it as opinion. Likert without worked exemplars collapses toward central scores, and judges do not share a latent notion of what separates a 3 from a 4. Two judges scoring the same answer 3 and 4 are not disagreeing about quality, they are disagreeing about the scale.

The honest tension: G-Eval's probability-weighted normalization exists precisely because coarse integer scores are tie-heavy and give you no ranking resolution. If you need to rank twenty systems, binary labels will produce a wall of ties. So "always go binary" is not universal. It's a tradeoff between decision-usefulness, which favours binary, and ranking resolution, which favours graded scores with a normalization scheme behind them. Pick based on what you'll do with the output.

When to not use a judge at all

The clearest signal is a deterministic check existing. Anthropic orders verification as rules-based first and describes the judge as something that "isn't considered very robust". If a linter, a schema validator, or an exact-match test can answer the question, that answer is free and correct, and a judge is a worse version of it.

Beyond that, four situations where I'd hold off. Before validating against human labels on your own data, because published agreement numbers transfer poorly and the ranking shifts across benchmarks say so directly. Reference-free evaluation of output from the same model family, given the self-enhancement numbers. As a reward signal for RL without adversarial hardening, given the master-key result. And on high-stakes criteria where your own human annotators show low agreement, because a judge cannot be more reliable than the guideline it is imitating.

There's a fifth that took me longest to accept: chasing small effect sizes. A judge whose verdict flips when you swap the answer order cannot resolve a 2% regression. The position bias alone is larger than the effect you're measuring. I spent weeks reporting on differences my instrument could not resolve, which is worse than reporting nothing.

What I changed after reading all of this

The judge is still in the pipeline and it still gates changes. What changed is what I let it decide on its own. It runs after the deterministic checks, with a reference answer, on binary criteria, in both presentation orders, with a canary set attached, and I report kappa per criterion rather than an aggregate agreement percentage.

If you want one concrete thing to do this week: take 100 items you already have human labels for, run your judge over each one twice with the answer order swapped, and compute both raw agreement and Cohen's kappa per criterion. The distance between those two columns is the number nobody puts in a blog post, and it's the one that tells you whether your evaluation has been measuring anything at all.