Somebody checked the exam.

A team at Edinburgh took MMLU, the benchmark quoted in almost every model announcement, and re-annotated 5,700 of its questions by hand across all 57 subjects. They estimate 6.49 percent of the questions contain errors. In the virology section, 57 percent of the questions they analyzed were wrong.

Not questions the models got wrong. Questions where the exam itself was wrong.

When they corrected them, model performance no longer matched what had originally been reported. The leaderboard had been measuring something other than what everyone thought.

You have almost certainly approved an AI decision on the strength of a number like that one. Not MMLU specifically. An accuracy figure in a vendor deck, a pass rate from a pilot, a score in a slide that ended an argument about whether the thing works.

The exam has errors

Start with what a benchmark is. It is a set of questions with agreed answers, and a score is the fraction your system gets right.

That definition holds up only if the agreed answers are right. Where they are not, the score stops being a measure of the model and becomes a measure of how well the model reproduces somebody's mistakes.

Six and a half percent is not a rounding error at the top of a leaderboard, where models are separated by one or two points. Two systems a point apart on a test with a 6 percent error rate are not distinguishable. They just look distinguishable.

The virology figure is the one that should stay with you. Fifty-seven percent. On the questions they examined in that subject, the exam was wrong more often than it was right. Any model scoring well on that section was being rewarded for agreeing with the errors.

The grader disagrees with itself

Human grading does not scale, so the industry graded the tests with another model. The practice has a name, LLM-as-a-judge, and it is now standard.

Rajarshi Haldar and Julia Hockenmaier at Illinois asked a question the practice had skipped. Not whether the judge agrees with humans. Whether the judge agrees with itself.

They ran judges three times over the same work and measured agreement using a statistic that corrects for lucky guesses. Agreement was low for every model they tested. The best of them reached roughly zero, which on that scale means no better than chance.

Read that twice. The grader, shown identical work, awarded different marks. The authors describe the ratings as almost arbitrary in the worst case.

They also checked whether more runs would settle it. Three runs, five runs, ten runs, no meaningful change. This is not noise you can average away. It is a property of the instrument.

Now think about where that instrument sits. Somebody in your organization is grading AI output with AI. Maybe it is a vendor benchmarking their own product. Maybe it is your team scoring a pilot. Either way a number comes out, the number goes in a slide, and the slide settles a debate.

The number moved when nobody touched anything.

The setup moves the score more than the model does

A third group, led by Yusuke Yamauchi, took apart the design choices inside these evaluations. What criteria you write, how you word them, whether you ask the judge to reason first.

Their finding is that the criteria carry the reliability. Vague criteria produce unreliable grading no matter which model grades. That is intuitive once stated and almost never audited, because criteria live in a prompt that nobody reviews the way they would review a control.

Their second finding is stranger and worth sitting with. Turning sampling off, so the judge behaves the same way every time, made alignment with human preference worse, not better.

So the obvious fix makes it more consistent and less right. You can have a grader that repeats itself or one that agrees with people. Choosing is now a design decision somebody in your organization is making by accident.

What you are buying when you buy a score

Put the three findings in a line and the picture is uncomfortable.

The questions may be wrong 6 percent of the time. The grader disagrees with itself at close to chance. And the grading setup, which nobody reviews, moves the result more than the choice of model does.

Call the thing you are left holding borrowed confidence. You did not run the evaluation. You did not write the criteria. You did not check the answer key. You are trusting a number produced by a process you have never seen, and the number is doing work inside a decision you signed.

Picture how it actually happens. A vendor reports 94 percent on a document extraction task. Your team runs a two-week pilot and gets 91. Close enough, so the program is approved and the number becomes the baseline everyone manages against.

Nobody asks which documents. Nobody asks who decided what a correct extraction was. Nobody runs it twice.

This is not an argument that benchmarks are worthless. They are useful for the thing they were built for, which is comparing research systems at a coarse grain. It is an argument that a benchmark score is evidence about a model in a laboratory, and your question is about a process in your company.

Those are different questions. Only one of them is on the slide.

What a test you could defend looks like

The alternative is narrower and much less impressive, which is why nobody puts it in a deck.

Take 200 real items out of your own process. Not synthetic examples. The actual invoices, claims, tickets or contracts, including the ugly ones, chosen so that the mix matches what a Tuesday actually looks like.

Have a person who does the work today write the correct answer for each. That is your answer key, and it is worth more than any public benchmark because it is the only one that describes your business.

Then define pass and fail as a rule rather than a judgment. Not whether the summary was good. Whether it picked the right cost center. Whether it flagged the missing purchase order. Whether the total reconciled. Rules can be checked by software, and software agrees with itself.

Run it more than once. If the same item gets two different answers across runs, you have found variance before it found you.

Two hundred items sounds small next to a benchmark with thousands. It is worth more anyway. Public benchmarks are broad and generic by design, because they have to work for everyone. Yours only has to work for you, and it can contain the seven cases that actually cause your team trouble.

Keep it. Rerun it when the model version changes, when the vendor ships an update, when somebody edits a prompt. That is the difference between a test and a demo. A test is something you can run again next quarter and compare.

The test to run this week

Find the most influential AI accuracy number in your organization. The one that appears in the board update or the business case.

Ask four questions about it. Who wrote the questions. Who wrote the answers. Who or what did the grading. And how many times was it run.

If you cannot get all four answers within a day, that number is not evidence yet. It is a claim wearing the clothes of evidence, and you have been treating it as settled.

The uncomfortable version of the question is the useful one. If that score turned out to be five points lower, would you reverse the decision you made on it? If the answer is no, the number was decoration. If the answer is yes, it deserves a real test, and you now know what one looks like.

Sources