Earlier this year Anthropic published a number that should have landed harder than it did. A model, Opus 4.5, scored 42% on a benchmark called CORE-Bench. Then, without touching the model, they fixed some rigid grading, some ambiguous task specs, and some harness bugs, and the same model scored 95%. Fifty-three points. The model did not get smarter overnight. The ruler changed.
That is the whole problem in one anecdote. We treat an eval score as a measurement of the model. It is actually the product of two things: the model's behaviour, and the instrument that judged it. And that instrument, more and more, is itself a model, an LLM-as-judge, or a benchmark harness with a grader buried inside it. Almost nobody measures the instrument.
A score is two claims #
Write it out and it is obvious. A passing score asserts two things: that the model produced an acceptable output, and that your grader is correct that it was acceptable. The first claim gets all the attention: leaderboards, launch posts, benchmark wars. The second gets almost none. Yet you cannot tell "the model regressed" from "the grader drifted" unless you have versioned and frozen the grader, and hardly anyone ships a grader with a version number. Your judge is a dependency that changes underneath you without a changelog.
The good news, which nobody wants to hear first #
Modern judges are good, and getting better, which ruins the easy hot take. The canonical study of LLM-as-judge found a strong model agreed with human graders more often than the humans agreed with each other.
the agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).Zheng et al., Judging LLM-as-a-Judge with MT-Bench (2023)
I wanted to see how that holds up on today's models, so I ran a small probe: ten question-and-answer pairs, graded PASS or FAIL by the three newest Gemini judges at temperature zero, all on one shared rubric. They agreed on all ten. I ran the newest of them twelve times on each item, same input every time, and it never once contradicted itself. The old fear that judges are slot machines, that even temperature zero flips verdicts, is largely obsolete on frontier models.
The unreliability moved into the criteria #
If the judges agree and never waver, the unreliability has not gone away; it has moved into the criteria. Watch what happens when I hold the same flagship judge and the same answers fixed, and change the rubric by one word, from "correct and adequately responsive" to "correct, complete, and precise":
It is no longer in the model, or even the judge. It moved into the criteria. And nobody versions the criteria.
One adjective, and the flagship model reverses its verdict. That is not the model being unreliable. That is the criteria being underspecified, and the score being a function of wording nobody wrote down carefully. Researchers have a name for this. They call it criteria drift: you cannot actually pin down your evaluation criteria until you start looking at outputs, so the criteria keep moving as you judge.
it is impossible to completely determine evaluation criteria prior to human judging of LLM outputs.Shankar et al., Who Validates the Validators? (2024)
And notice which item flipped. Not the arithmetic, not the capital of Australia. The subjective one. That is exactly the problem, because the clear-cut cases never needed a judge in the first place. You reach for an LLM grader precisely for the fuzzy, subjective residue, and that residue is where the grader is least anchored and most sensitive to how you phrased the ask.
The harness moves the number more than the model #
Zoom out from one grader to a whole benchmark and it gets worse, because a benchmark is not just a grader. It is a harness: prompts, tools, context limits, retries, the environment. Any one of those can swing the score by as much as a whole model generation would.
a composite of models, harnesses, contexts, environments, and feedback signals, any one of which can move the benchmark score by margins comparable to those between adjacent model generations.Coding Benchmarks Are Misaligned with Agentic Software Engineering (2026)
The CORE-Bench jump from 42 to 95 was exactly this: grading, specs, and harness, not the model. And it is not only agentic benchmarks. An audit of Chatbot Arena found two identical checkpoints, the same weights submitted twice, landing 17 rating points apart, with four other models ranked in between the two copies of the same thing. The measurement had more spread than the models it was ranking.
Even agreement lies #
Suppose you do the responsible thing and try to validate your grader by measuring how often it agrees with a human. On real production traffic almost everything passes, and on lopsided data raw agreement is a liar:
Ninety percent agreement, and a grader barely better than a rubber stamp that says PASS to everything. Cohen's kappa corrects for how often you would agree purely by chance, and on pass-heavy data chance agreement is enormous, so the corrected number collapses. Clinicians named this the kappa paradox back in 1990: high agreement, near-zero reliability. Which means even the teams diligent enough to validate their validator usually do it with a statistic that flatters them.
And it gets worse as the models get better #
One more twist, and it is the one that should worry anyone betting on models simply outgrowing this. Optimization pressure always flows into the gap between your metric and what you actually wanted. That is just Goodhart's law, and stronger models are better at finding the gap. Cursor reported that the smarter the model, the more it games the eval:
reward hacking is far more common with newer, more sophisticated models than with older ones.Cursor, Reward hacking is swamping model intelligence gains (2026)
So a stronger model does not make your unvalidated eval more trustworthy. It makes it less. The thing being measured is actively adversarial to the ruler, and it is improving faster than the ruler is.
This is not an argument for despair #
It would be easy to read all this as "scores are meaningless," and that is wrong, and I want to be careful because it is the lazy conclusion. Judges can match human agreement, as MT-Bench showed, once you validate and bias-correct them. Fixing a vague grader is often cheap: a handful of labelled examples and a couple of iterations. The claim is not that measurement is impossible. It is that the instrument is unvalidated by default, and validating it is both tractable and neglected.
The moves are unglamorous and mostly borrowed from measurement disciplines older than machine learning. Version and freeze the grader, so that a score change means the model moved and not the ruler. Report agreement as kappa, not raw percent, so lopsided data cannot flatter you. Decompose a holistic "quality: 4 out of 5" into atomic yes-or-no checks, because one blurry score hides broken outputs and binary checks expose them. And wherever an outcome is checkable against the real world, check it instead of judging it: the most reliable eval is the one that needs no opinion. Agent benchmarks like tau-bench score the final database state rather than the prose, and are immune to grader noise for exactly that reason. Reserve the fallible judge for the genuinely subjective residue, and then treat that judge as the fallible instrument it is.
Who evaluates the evaluator? #
There is a wall at the end of this, and honesty means walking up to it. If you validate your grader with another grader, what validates that one? The regress bottoms out only at something expensive and unscalable: careful human judgment, or a real-world outcome you can actually observe. You cannot automate your way to the bottom. Which means there is no product, no framework, no clever meta-judge that makes this go away. There is only the discipline of treating the thing that produces your scores as a measured, versioned, noise-quantified instrument, instead of an oracle.
The industry spent three years getting very good at building the thing being measured, and almost no time getting good at the measurement. A passing score is a claim about your model and a claim about your grader. We have been arguing the first at the top of our lungs and taking the second on faith.
Full disclosure: I am building in evals, so read this as a practitioner's obsession, not a detached take.
Further reading #
- ▸Anthropic, Demystifying Evals for AI Agents (2026). The CORE-Bench 42 to 95 story, and how much of a score is grading and harness.
- ▸Zheng et al., Judging LLM-as-a-Judge with MT-Bench (2023). Where the 85% vs 81% human-agreement result comes from.
- ▸Shankar et al., Who Validates the Validators? (2024). Criteria drift, and why you cannot fully specify criteria up front.
- ▸Singh et al., The Leaderboard Illusion (2025). Identical checkpoints, different Arena scores.
- ▸Coding Benchmarks Are Misaligned with Agentic Software Engineering (2026). Why harness variance rivals a model generation.
- ▸Cursor, Reward hacking is swamping model intelligence gains (2026).
- ▸Feinstein and Cicchetti, High agreement but low kappa (1990). The kappa paradox, from clinical statistics.
- ▸Yao et al., tau-bench (2024). Grading the end state instead of judging the prose.