Thirteen models, all advertising support for at least 128,000 tokens of context. A researcher measured what they could actually do with it.
At 32,000 tokens, 11 of the 13 dropped below half of their own short-context baseline.
GPT-4o was among the better performers. It fell from 99.3 percent to 69.7 percent as the context grew.
Those two numbers are the same model, the same task, the same day. The only variable was how much text surrounded the answer.
Nothing was broken. Every one of those models accepts a 128,000 token input, processes it, and returns an answer.
The window is real. Finding things inside it is the part that degrades.
A context window is a capacity, not a competence
The number in the product announcement describes how much text the system will accept. People read it as how much text the system can use, and those are different quantities.
The gap matters because of how enterprise systems are being built. Someone reads that a model handles 128,000 tokens, decides a 200-page contract fits comfortably, and designs a workflow on the assumption that anything in the prompt is available to the answer.
That assumption is doing enormous load-bearing work. It underwrites contract review, policy question answering, document comparison, and most of what gets demonstrated as "ask your documents anything". If it is only true for the first and last few thousand tokens, those systems are correct on the easy questions and quietly unreliable on the rest.
That work by Ali Modarressi and colleagues, published in February 2025 as NoLiMa, tested exactly that assumption. The trick in their design is that the question and the answer share almost no words, so the model has to infer a connection rather than match a phrase. Real questions work that way. Keyword overlap is the easy case.
Take the harder case and the advertised window shrinks to a fraction of itself.
Two numbers, then. The one on the spec sheet, and the one your workflow actually gets. Only the first is published.
The middle is where answers go missing
This was visible earlier, and more specifically, than most people realize.
In 2023 Nelson Liu and colleagues published Lost in the Middle. Their finding is that performance is highest when the relevant information sits at the beginning or the end of the input, and degrades significantly when the model has to reach for something in the middle.
The curve is a U. Strong at the start, weak in the middle, strong again at the end.
They also found it in models specifically built for long context. That rules out the comfortable explanation, which is that it was a limitation of the smaller models of 2023 and has since been engineered away. NoLiMa measured the same shape two years later on models ten times the size.
Think about what that means for a 200-page contract. Page one gets read. The last page gets read. Pages 40 through 160 are present, billed, and unreliable.
That is a bad property for a contract. The termination clause is rarely on page one.
It gets stranger than a position effect
In July 2025 Chroma ran 18 models through a wider set of long-context tasks, and the results complicate the tidy U-shape.
Their summary is blunt. Models do not use their context uniformly, and performance grows increasingly unreliable as input length grows.
Two findings from that work are worth carrying around. A single distractor, one plausible but wrong passage, measurably reduced accuracy. And models performed better on shuffled haystacks than on logically structured ones.
Read that second one twice. Organizing the context coherently made the results worse.
Nobody should build a strategy on one counterintuitive result. It does say something about how little is understood about what happens inside the window.
And it has a practical edge. If logical structure is not helping, the taxonomy work your knowledge base team spent two years on is not the lever anyone thought it was. That is worth knowing before the next reorganization of the document store.
This breaks the plan most teams are on
The dominant architecture right now is to retrieve a lot and let the model sort it out. Cheap context encouraged it. If the window is enormous, why be selective?
These three results say selectivity is the whole game.
Three findings, one direction. Position matters, so where a passage sits changes whether it is used. Distractors cost accuracy, so every extra passage has a price. And degradation rises with length, so the total matters independently of what is in it.
Retrieving 40 passages when 4 would do does not improve the answer. It adds 36 distractors, buries some of them in the middle, and pushes the total length toward the region where accuracy falls away. You paid more tokens for a worse answer. No error was raised. Nothing in the logs says anything happened.
There is a related idea that long context makes retrieval obsolete, since you can simply put everything in the prompt. The measurements point the other way.
Long context makes retrieval quality more important, not less. The penalty for retrieving badly used to be visible, because a short prompt with the wrong passage produced an obviously wrong answer. Now the wrong passage sits among 39 others inside a window nobody reads, and the answer comes back fluent.
What actually helps
Retrieve less and rank harder. The work moves from filling the window to deciding what deserves to be in it. That is an engineering problem with 30 years of known answers, most of them from search.
Put the decisive material at the start or the end of the prompt. It is a crude fix, it follows directly from the U-shape, and almost nobody does it deliberately.
Check where your framework puts retrieved chunks by default. Most concatenate them in relevance order or in document order, which means the ranking that decides position was not designed with the U-shape in mind.
Measure at your real context length, not at a demo length. A system evaluated on 2,000 token inputs and deployed against 60,000 token inputs has not been evaluated. It has been previewed.
Cut distractors on purpose. A retrieval step that returns near-misses is actively harmful, not neutral.
Precision beats recall here. That inverts the instinct most search engineers were trained with, where returning one more possibly-relevant result costs nothing. In this architecture it costs accuracy.
The test to run this week
Take a question your system answers today from retrieved documents. Note the answer.
Now run it three times.
Once with only the passage that contains the answer. Once with that passage plus ten more. Once with that passage buried in the middle of everything you can retrieve.
Compare the three answers. The input contained the correct passage every time.
If they differ, you have measured your own context degradation. You now know something no specification sheet will tell you, because no vendor publishes the usable length.
If they are identical, run it again at 60,000 tokens rather than 6,000. The effect is a function of length, so a short test will show you nothing.
Then ask the question that follows, which is about your own reporting. What is the longest input your system sees in production, and has anyone tested it at that length?
If the evaluation ran at 4,000 tokens and production runs at 40,000, the accuracy figure in your business case describes a system you are not operating.
Sources
Liu, N. F. et al., Lost in the Middle: How Language Models Use Long Contexts, arXiv:2307.03172, TACL, July 2023. Performance is highest when relevant information appears at the beginning or end of the input and degrades significantly when the model must retrieve from the middle, including in models built for long contexts.
Modarressi, A. et al., NoLiMa: Long-Context Evaluation Beyond Literal Matching, arXiv:2502.05167, February 2025. Of 13 models advertising at least 128K context, 11 fell below half their short-length baseline at 32K tokens. GPT-4o declined from 99.3% to 69.7%.
Hong, K., Troynikov, A. and Huber, J., Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma, July 2025. Across 18 models, performance grows increasingly unreliable as input length rises. A single distractor reduced accuracy, and models scored higher on shuffled haystacks than logically structured ones. Company-published.