# Calibrate an LLM Judge Before You Trust Its Scores > A groundedness score can look precise while the grader marks a contradicted answer as supported. This guide writes a rubric with a precedence rule, builds a blinded labelled set, and measures agreement with Cohen's kappa and a confusion matrix, using a deterministic stub judge and a synthetic 30-item set. It adds swap, padding and planted-bias tests, shows why 30 perfect items still cannot prove 95% reliability, and lists when a judge should not make the call. - Author: [Shashank Rawlani](https://shashank.rawlani.com) - Published: 2026-11-06T13:30:00.000Z - Updated: 2026-11-06T13:30:00.000Z - Category: AI in Testing - Canonical source: https://automationtester.in/blog/ai-in-testing/calibrate-llm-as-judge-scores - Tags: llm-as-judge, llm-evaluation, rag-evaluation, cohens-kappa, evaluation-bias, ai-testing **Quick answer:** Calibrate a model grader the way you would calibrate any instrument before reporting its reading. Write a rubric with defined labels, label a set of answers by hand without knowing which model wrote them, and compare the judge with those labels using a confusion matrix and a chance-corrected statistic such as Cohen's kappa, not one accuracy number. Then break the judge on purpose: swap answer order, pad answers, plant a known bias to prove your test notices it. Route scores the data cannot vouch for to a person. If no score band clears your bar, the judge does not ship. A dashboard says groundedness is 0.82 this week, up from 0.79. Nobody on the team can say what a point of that number is worth. A customer-facing answer claims digital goods are refundable within 30 days, the policy text says they are not, and the grader marked the answer supported. The score is precise to two decimals and wrong about the one thing it exists to catch. This is Part 2 of the LLM and RAG Evaluation series. [Part 1](/blog/ai-in-testing/build-llm-eval-dataset-product-risks) covers where the cases come from. This part covers the instrument that grades them, and it stays on one question: how much of a judge's output should you believe, and how do you find out? **Read the evidence for what it is.** Every number in the worked example below comes from a deterministic stub judge and a 30-item synthetic labelled set constructed for this article. The stub scores word overlap and calls no model, so its failures are not a forecast of what any real model does. What carries over is the harness: the metrics, the invariance tests and the threshold arithmetic. Code ran on Python 3.14.5, with Cohen's kappa cross-checked against scikit-learn 1.9.1 (the current release on PyPI, checked 2026-11-02). Figures quoted from the MT-bench and position-bias papers describe the models those authors tested in 2023, not today's. ## What does the rubric have to say before a model applies it? "Rate the quality from 1 to 10" is a rubric that lets the model pick its own definition of quality on every call. A groundedness judge needs labels a human can apply to the same item and reach the same answer. The ones used here have a precedence rule, because the interesting disagreements are about which defect wins: ``` You are grading one answer against the CONTEXT only. Split the ANSWER into factual claims. For each claim decide: supported - the CONTEXT states it, in the same form (numbers, limits, who/when match) unsupported - the CONTEXT does not mention it contradicted - the CONTEXT states something incompatible with it Final label: SUPPORTED - every claim supported PARTIAL - at least one claim unsupported, none contradicted CONTRADICTED - at least one claim contradicted, whatever else is true Ignore style, length and tone. Do not use outside knowledge. Return JSON: {"claims":[{"text":..., "verdict":...}], "label":...} ``` Three decisions are baked in. Claim-by-claim reasoning is demanded before the label, which is the same decomposition the RAGAS paper uses for its faithfulness metric: extract statements from the answer, check each against the context, and score the share that are supported. A contradiction outranks everything else in the answer, because a response that is 90% accurate and states the wrong refund window is a bad response. And the output is a label, which makes it comparable with a human's label cell by cell. The three labels are also ordered (supported, partial, contradicted), which matters later when choosing a kappa. If a person reading this rubric cannot label ten items and agree with a colleague, the model will not do better. Find that out first, on paper. ## How do you build labels the judge cannot lean on? Each labelled item is a context, an answer and a label. Two people label independently, then settle disagreements into one adjudicated label that becomes the reference. Keep both sets of raw labels, because the disagreement between the two people is the ceiling for the judge: a model cannot sensibly be asked to agree with humans more than they agree with each other. Blind the labellers and the judge. Neither should see which model, prompt version or system produced an answer. The MT-bench paper saw some judges win more often for their own model than humans did (10% higher for GPT-4, 25% higher for Claude-v1), though GPT-3.5 did not favour itself and the authors say their limited data cannot settle whether self-enhancement bias exists. Unsettled is reason enough to keep the generating model's name out of the grading call. This is also checkable, so check it. The harness renders each judge prompt from the context and the answer only, then asserts no identifying string leaked in: ``` def render_prompt(context, answer): return RUBRIC + "\n\nCONTEXT:\n" + context + "\n\nANSWER:\n" + answer p = render_prompt(CONTEXTS["refund"], ITEMS[0][1]) leaks = sum(n in p.lower() for n in ("gpt", "claude", "llama", "gemini")) assert leaks == 0 ``` The worked set has five invented policy contexts (refunds, shipping, warranty, account lockout, data export) with six answers each: two supported, two partial, two contradicted. The contradicted answers are the dangerous ones by design. They reuse the context's own wording and change one fact, such as 30 days becoming 60, or 5 failed sign-ins becoming 10. Real traffic is not balanced like this, so on your own data report results per class, never only overall. ## Why does one accuracy number hide the failure that matters? Run the stub judge, which maps the share of an answer's content words found in the context to a label, over the 30 items and compare it with the adjudicated labels. The metric code is short enough to own: ``` def kappa(a, b, weights=None): n = len(a) m = confusion(a, b) # 3x3 counts, rows = a, cols = b w = lambda i, j: 0 if i == j else (1 if weights is None else abs(i - j) / 2) po = sum(w(i, j) * m[i][j] for i in range(3) for j in range(3)) / n ra = [sum(r) for r in m] cb = [sum(m[i][j] for i in range(3)) for j in range(3)] pe = sum(w(i, j) * ra[i] * cb[j] for i in range(3) for j in range(3)) / n / n return 1 - po / pe ``` scikit-learn documents the same quantity as (p_o - p_e) / (1 - p_e), where p_o is observed agreement and p_e is the agreement expected if both raters labelled at random with their own label frequencies, and notes that 0.0 means no agreement beyond chance. Swapping the two raters does not change it. It also offers linear and quadratic weights, which suit ordered labels like these, where calling a contradiction "partial" is a smaller miss than calling it "supported". Output from Python 3.14.5, identical to scikit-learn 1.9.1's `cohen_kappa_score` on the same labels: ``` human A vs human B raw agreement 0.900 kappa 0.850 weighted kappa 0.889 judge vs gold accuracy 0.467 kappa 0.200 weighted kappa 0.077 ``` | Gold label (rows) / judge label (columns) | Supported | Partial | Contradicted | | --- | --- | --- | --- | | Supported (10) | 7 | 3 | 0 | | Partial (10) | 1 | 7 | 2 | | Contradicted (10) | 8 | 2 | 0 | The two humans disagreed on three of 30 items and still reached kappa 0.85. The judge's accuracy of 0.467 already looks poor, but the table says more: it never produced the correct label for a contradicted answer, and eight of ten were graded supported. Every one of those eight scored between 0.86 and 1.00, because a lexical judge sees all the right words. A real model judge will not fail this way, but it can fail on the same cell, and a mean score cannot show you that. Read the off-diagonal cells, and read the items in them: `Express shipping costs 20 dollars and takes 1 to 2 business days` against a context that says 12 is the kind of sentence that tells you what the grader is actually matching. ## Which biases can you test for, and which test sees which? The MT-bench paper names three: position bias (favouring an answer because of where it sits), verbosity bias (favouring the longer answer), and self-enhancement bias (favouring the grader's own output). A separate paper on evaluation fairness showed how far position alone can move a ranking: with ChatGPT as evaluator, Vicuna-13B beat ChatGPT on 66 of 80 queries just by changing the order the two answers appeared in. Each bias needs its own test, and a test for one is blind to the others: | Bias | Test that exposes it | What that test cannot see | | --- | --- | --- | | Position | Grade each pair twice with the order swapped; count pairs where the winner changes | Anything that does not depend on order, including length preference | | Verbosity | Add text that carries no new information and check whether the label moves | Order effects; padding that happens to be an inert sentence may not trigger it | | Self-preference | Compare judge win rates per generator with human win rates for the same pairs | Needs enough human-labelled pairs per generator to mean anything | | Identity leakage | Assert that no model or prompt name reaches the judge call | Style cues that identify a model by themselves | The swap test is cheap. The paper's conservative recipe is to call the judge twice with the order swapped and declare a winner only if it wins both ways, otherwise call a tie. In the paper's own measurements, GPT-4 stayed consistent after swapping on 65.0% of cases with its default prompt, and Claude-v1 on 23.8%. Here is the harness version, on the 60 same-context pairs with different gold labels: ``` for name, kw in (("unbiased", {}), ("planted first-position +0.10", {"first_bonus": 0.10}), ("planted length +0.01/word", {"length_bonus": 0.01})): cons = 0 for a, b in pairs: x = pair_prefers_first(CONTEXTS[a[0]], a[1], b[1], **kw) y = pair_prefers_first(CONTEXTS[a[0]], b[1], a[1], **kw) cons += (x != y) # first-wins in one order and second-wins in the other = same winner print(name, cons / len(pairs)) ``` ``` unbiased pairs 60 same winner after swap: 57 (95.0%) planted first-position +0.10 pairs 60 same winner after swap: 47 (78.3%) planted length +0.01/word pairs 60 same winner after swap: 59 (98.3%) ``` Three things come out of that. First, plant a bias and confirm the test catches it: the +0.10 first-position bonus drops consistency from 95.0% to 78.3%, which proves the test has teeth. A test that has never failed has not been shown to work. Second, the "unbiased" stub is not perfectly consistent: three pairs score exactly equal, and the tie-break (`>=`) hands them to whichever answer comes first. Ties resolved by position are position bias, and a real judge returning "tie" or an arbitrary pick has the same exposure. Third, the planted length bonus is nearly invisible to the swap test (98.3%), because a length preference is the same in both orders. That is why verbosity needs its own test. For verbosity, the paper's "repetitive list" attack rephrases a numbered list and prepends the rephrasing so the answer has the same content at double the length. Claude-v1 and GPT-3.5 were fooled on 91.3% of 23 answers; GPT-4 on 8.7%. The harness equivalent is a padding test with two variants, because they fail differently: ``` PAD = " To summarise, the information above reflects the policy as described in the context." flips = [i for i, it in enumerate(ITEMS) if label(support_score(CONTEXTS[it[0]], it[1] + PAD)) != J[i]] print("labels changed by padding:", len(flips), "of", len(ITEMS)) # true-but-irrelevant padding: append the context's first sentence to every answer # and count non-supported answers promoted to SUPPORTED ``` ``` labels changed by padding: 28 of 30 non-supported answers promoted to S by the padding: 4 of 20 ``` The stub moves in the opposite direction from the paper's judges when the padding adds unfamiliar words (28 of 30 labels changed, all of them downward), and moves up when the padding is lifted from the context (four non-supported answers, including one contradicted answer, were promoted to supported). Either direction means the label depends on text that has nothing to do with correctness. The pass condition for this test is not "no change in score", which a model will rarely meet. It is "no change in label", and you set the tolerated flip rate before you run it. ## How much data does a review threshold need? The aim is a rule of the form: above this score, accept without a person; below it, queue for review. The stub's score by gold label shows why that is hard to place. Supported answers scored 0.80 to 1.00, and contradicted answers scored between 0.57 and 1.00, with eight of the ten at 0.86 or above. At a cut-off of 0.95, the bucket the judge called supported held 10 items, 7 truly supported, a precision of 0.70. The sample size matters as much as the score. The 95% Wilson interval for that 7-of-10 bucket runs from 0.40 to 0.89. And the arithmetic works against you even when the judge is perfect: ``` def auto_accept_threshold(scores, gold, target=0.95): """Lowest cut-off whose 'supported' bucket has a Wilson lower bound >= target. None means the judge may not auto-accept anything.""" for t in sorted(set(scores)): bucket = [g for s, g in zip(scores, gold) if s >= t] k = sum(g == "S" for g in bucket) if bucket and wilson(k, len(bucket))[0] >= target: return t return None ``` ``` auto-accept threshold on the 30 items: None Wilson lower bound, 30/30 supported: 0.89 Wilson lower bound, 100/100 supported: 0.96 Wilson lower bound, 58/60 supported: 0.89 ``` Thirty hand-checked items that the judge got entirely right still cannot show a 95% reliability claim; roughly a hundred error-free items in the bucket can. So the honest statement from a small labelled set is usually a confidence interval and a queue, not a threshold. A judge that is right 58 times in 60 and a judge that is right 30 times in 30 are indistinguishable at this sample size. The queue itself should be driven by more than the score. The position-bias paper proposes measuring how much a judge's verdict changes across orderings, using an entropy of those verdicts as a difficulty signal, and asking for human help on the unstable items. That is the useful shape: send an item to a person when scores sit in the zone your data cannot vouch for, when the swapped or repeated judgments disagree, or when the label is CONTRADICTED on a path where a wrong answer is costly. Treat the disagreement queue as a deliverable, and read its items every release. They are the cheapest source of new labelled cases. ## When should the judge not make the call? There are four situations where the calibration result should be "do not use this judge here", and the work above is how you find out. The first is when the judge has to solve the problem to grade it. In the MT-bench paper's math test, GPT-4 as judge accepted an incorrect answer 14 of 20 times with the default prompt, 6 of 20 with chain-of-thought, and 3 of 20 when given a reference answer. A judge that cannot do the task should be handed a reference or replaced by a deterministic check. The second is self-grading: if the model that wrote an answer also grades it, add a human-labelled slice for that pairing and compare. The third is small differences. A move from 0.79 to 0.82 with no interval and no agreement figure is noise until shown otherwise, and the labelled set above is why you can say so. The fourth is any gate where one wrong pass has a large cost: a policy answer, a refund amount, a medical or legal claim. The measured error in the contradicted row of the confusion matrix is the number to weigh against that cost. Agreement figures from papers are not a licence either. MT-bench reports GPT-4 agreeing with human experts 85% of the time on non-tie pairwise comparisons, against 81% agreement among the humans themselves, on its own benchmark. The RAGAS paper reports 0.95 agreement with human annotators for faithfulness on its WikiEval set, but the comparison is pairwise (which of two answers is more faithful) on 50 recent Wikipedia pages, a different task from labelling one answer in your domain. Both are reasons to expect a judge can work, and neither stands in for measuring yours. Calibration also expires. The judge is a model plus a prompt plus a rubric, and a change to any of the three is a change to the instrument. Keep the labelled set and the harness in the repository, pin the judge version and prompt next to the scores, and re-run the agreement and invariance checks whenever one changes. Structure checks that do not need a model, such as validating the judge's JSON output against a schema before anything is scored, belong ahead of the semantic grading; the [JSON Schema Validator](/tools/json-schema) covers that step. ## Apply this now - Write the rubric as labels with a precedence rule, and have two people label the same ten items with it. If they cannot agree, fix the rubric before involving a model. - Build a labelled set of at least 30 items per label you care about, drawn from real failures and from the risks in [your eval dataset](/blog/ai-in-testing/build-llm-eval-dataset-product-risks). Keep both raw human labels and the adjudicated one. - Strip model, prompt and system identity from the judge call, and add an assertion that fails the build if any of them appears in the rendered prompt. - Report the confusion matrix and a weighted kappa for ordered labels, next to the human-to-human kappa as the ceiling. Look at the items in the off-diagonal cells. - Run a swap test, a padding test and a planted-bias check. Record the tolerated flip rate beforehand, and treat a test that cannot fail as not yet written. - Compute a Wilson interval for every bucket you want to auto-accept. If its lower bound is below your target, put that bucket in the human queue and keep labelling. - Store the judge version, prompt hash and labelled-set version with every reported score, and re-run the harness when any of them changes. The evidence to capture is the confusion matrix, both kappas, the three invariance results with their thresholds, and the list of disagreement-queue items from the latest run. ## Questions readers ask ### How many labelled items do I need? Enough that the intervals on the decisions you care about are narrower than the gap you are trying to detect. The worked set shows the floor: 30 flawless items still give a lower bound of 0.89 on reliability. For a per-class claim you need to count items per class, not overall, and the contradicted class is usually the one with the fewest. ### Will a stronger judge model fix it? Perhaps, and the harness is how you would know. The MT-bench paper found a large gap between judges on the same attacks (8.7% versus 91.3% failure on the repetitive-list test), so judges differ. It also found the best-performing judge still failed on math without a reference answer. Measure the one you plan to use, on your labels. ### Can the model under test grade itself? You can, but the MT-bench data showed some judges scoring their own model above the human win rate while saying the evidence was too limited to conclude. So compare its verdicts against human labels for that specific pairing before trusting the numbers. Grading with a different model family, and swapping order, removes two known sources of tilt at the cost of two more calls. ### Why kappa rather than percent agreement? If one label dominates, two raters agree often by chance. Kappa subtracts the agreement expected from each rater's label frequencies, so 0.0 means no better than chance. Percent agreement is still worth printing beside it, because it is easier to read. ## Primary references - [Zheng et al., MT-Bench and Chatbot Arena paper (arXiv 2306.05685)](https://arxiv.org/abs/2306.05685). Names position, verbosity and self-enhancement bias; reports position consistency after swapping (65.0% GPT-4, 23.8% Claude-v1 with default prompts), the repetitive-list failure rates (91.3% for Claude-v1 and GPT-3.5, 8.7% for GPT-4 on 23 answers), the self-preference win-rate gaps (10% GPT-4, 25% Claude-v1) together with the authors' statement that their data cannot determine whether the bias exists, the math-judging failure rates by prompt (14/20, 6/20, 3/20), the conservative swap-twice mitigation, and 85% GPT-4 to human agreement against 81% among humans. - [Wang et al., position bias in LLM evaluators (arXiv 2305.17926)](https://arxiv.org/abs/2305.17926). That changing answer order alone let Vicuna-13B beat ChatGPT on 66 of 80 queries, and the three calibration strategies: multiple evidence, balanced position aggregation, and a diversity-entropy signal for asking humans for help. - [Es et al., RAGAS paper (arXiv 2309.15217)](https://arxiv.org/abs/2309.15217). Faithfulness defined as the share of answer statements supported by the context, and the WikiEval agreement table (0.95 faithfulness, 0.78 answer relevance, 0.70 context relevance, pairwise accuracy against human annotators). - [scikit-learn documentation for cohen_kappa_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html). The kappa formula, that it is symmetric between raters, the linear and quadratic weighting options, and the meaning of 0.0 as no agreement beyond chance. ## Continue reading on AutomationTester.in - [Use AI for Performance Diagnosis Without Inventing Causes](https://automationtester.in/blog/ai-in-testing/ai-performance-diagnosis-evidence-boundaries) - [Review AI-Generated JUnit Tests Before They Raise Coverage](https://automationtester.in/blog/ai-in-testing/review-ai-generated-junit-tests-before-coverage) - [Catch promptfoo Config Mistakes That validate Lets Through](https://automationtester.in/blog/tools-reviews/catch-promptfoo-config-mistakes-before-eval) - [Playwright MCP Is Not Your Regression Test Suite](https://automationtester.in/blog/ai-in-testing/playwright-mcp-vs-regression-test-suite) - [MCP Observability: Audit Tool Calls Without Logging Secrets](https://automationtester.in/blog/ai-in-testing/mcp-observability-audit-without-secrets) Source: [Calibrate an LLM Judge Before You Trust Its Scores](https://automationtester.in/blog/ai-in-testing/calibrate-llm-as-judge-scores) by Shashank Rawlani.