They asked one model the same question one thousand times. Same prompt. Same model. Temperature zero, the setting that exists so the answer comes back the same way twice.

They got 80 different answers.

The prompt was "Tell me about Richard Feynman." Every one of the thousand runs produced the same opening, all the way through "Feynman was born on May 11, 1918, in". Then 992 of them continued "Queens, New York" and 8 continued "New York City". Token 103. Both are true. Only one of them is the answer you filed.

Nothing in that experiment was random. Nobody rolled a die. Temperature zero did exactly what it promised, which is to take the highest-probability token every single time. The probabilities moved.

What moved them was the other people using the model.

Feynman is a harmless example. Replace him with an invoice, an insurance claim or a credit decision, and the same mechanism is running inside a process that somebody signs their name to.

Nothing here was random

Floating point addition is not associative. Add a long column of numbers in one order and you can get a slightly different total than if you add the same numbers in another order. The gap shows up in the last few decimal places. It is tiny, it is not a bug, and it is a property of how computers hold fractions.

Inference servers batch incoming requests together, because that is what makes them affordable. The batch shape depends on how much traffic arrived in the same few milliseconds. Change the size of the batch and the chips add their numbers up in a different order.

Different order, different ninth decimal, occasionally a different winning token.

So your request was pooled with strangers. The size of that pool moved a number you will never see. That number picked one word over another. Call it the batch lottery. You did not buy a ticket and you cannot see the draw.

The useful part is where this puts the problem. The nondeterminism is not in the model. It is in the load.

The lottery moves your accuracy score too

This is not confined to prose about physicists.

A team led by Jiayi Yuan measured what happens to reported results. Running a reasoning model and always taking the highest-probability answer, they found up to 9 percent variation in accuracy and a 9,000 token difference in the length of the answer, which runs to thousands of words. The cause was the number of graphics cards, the model of card, and how many test items were processed together. Same weights. Same prompts. No code change.

Read that as an executive. The accuracy figure in your evaluation deck is not a property of the model you licensed. It is a property of the model, the hardware, and how busy that hardware was in the hour you measured. Benchmark on a quiet Sunday and a heavy Monday and you can be nine points apart while believing you ran the same test.

How much room there is for this depends on how finely the model stores its numbers. Store them finely and the arithmetic is close to stable. Store them coarsely and there is far more room for the order of the sums to change the outcome. Coarse storage is faster and cheaper, so it is what most production systems use. The setting that saves you money is the setting that wobbles most.

Where the variance concentrates is the worst possible place

A separate group went looking for where nondeterminism actually lands, and the answer should stop you.

Tairan Fu and colleagues measured the effect on token probabilities rather than on finished text. The variance is significant for probabilities between 0.1 and 0.9. It is much smaller when probabilities sit close to 0 or close to 1.

Translate that. Where the model is confident, it is stable. Where the model is unsure, it wobbles.

The unsure cases are the exceptions. The ambiguous invoice. The contract clause that does not match the template. The transaction that could be coded two ways. Those are precisely the decisions you would want a system to be consistent about, and they are the ones the batch lottery reaches first.

Your happy path is reproducible. Your hard cases are not.

An invoice arrives with a line item that could reasonably be coded to two different cost centers, and the model is about 60 percent sure it belongs in the first. On a quiet morning that request runs in a small batch and the answer comes back one way. At month end, when everyone in the company is pushing work through the same endpoint, the same document in a much larger batch can come back the other way. Nobody changed the prompt, the model or the policy. The only thing that changed was how many colleagues hit the service in the same instant.

What reproducibility was quietly holding up

Consider what breaks when an output cannot be regenerated on demand.

  • An auditor asks how a figure was produced and expects the process to produce it again.

  • A regression test asserts that yesterday's behavior still holds today.

  • A dispute with a customer or a regulator turns on showing what the system did, not describing it.

  • An incident review needs the failure to reproduce before anyone can fix it.

Every one of those assumes you can run it back. None of them survive an answer that arrives differently depending on server load.

This is why the finding matters more to a controller than to a researcher. A researcher loses some benchmark comparability. A controller loses the ability to say what happened.

The question that eventually arrives is never abstract. Somebody asks why this invoice was coded this way, or why that customer was declined, and the honest answer cannot be that the system was busy at the time. An explanation that depends on infrastructure load is not an explanation anyone will accept from you twice.

Determinism is purchasable, and it is not free

Thinking Machines Lab, which published the thousand-run result in September 2025, shipped a fix in the same post. Batch-invariant kernels make the arithmetic independent of batch size, which produces bit-identical output across repeated runs.

It costs speed. On their test, the default path finished in 26 seconds, the deterministic path in 55, and 42 seconds once the attention kernel improved. Roughly 60 percent slower, on an implementation the authors call not yet optimized.

That is a real price. It is also, for the first time, a decision rather than a constraint. Anyone telling you repeatability is impossible is describing last year.

Where to put the model instead

Two conclusions follow. Only one of them is comfortable.

The uncomfortable one is that a probabilistic system makes a poor system of record. Not because it is inaccurate. Because it is not repeatable, and a record you cannot reproduce is a claim.

The other is more useful. Interpretation and execution do not have to run on the same machinery. A model is an excellent instrument for reading a messy document and a poor one for being the ledger that records what it read. Let the probabilistic part interpret. Hand a structured result to something that executes the same way every time and writes down what it did.

In practice that means the model returns a structured result rather than a decision. Fields, values, and a confidence for each one, handed to a layer that applies the rule, records the inputs it acted on and produces the same output whenever those inputs recur. The interpretation can vary between runs, because it is a reading rather than a ruling. What gets written down does not.

The variance stays where you can tolerate it and stops leaking into the place you cannot.

Most enterprise deployments have not made that split. They pointed a nondeterministic system at a process whose entire job is producing the same answer twice.

The test to run this week

Take one AI output that already moves a decision. A classification, an extraction, a figure that feeds a report.

Run it 100 times, unchanged, at the temperature you use in production. Count the distinct results.

If you get one, you have found a genuinely stable case or you are not looking at where the output gets rounded. If you get more than one, you now know the size of a variance you have been treating as zero.

Then ask the harder question. Who in your organization currently believes that number is fixed?

Sources