Where something sits in a long prompt changes whether it gets used

The intuitive model of a long prompt is a reader who takes in all of it. The evidence is that position matters: information at the start and end of a long input was used well, and information in the middle was used markedly worse — in the worst cases measured, worse than giving the model nothing at all.

Read the paper. What follows is a reading of it, not a replacement; the link is above.

What the study measured

Liu and colleagues, in a paper published in the Transactions of the Association for Computational Linguistics, gave models a question and a set of retrieved documents including the one that held the answer, and moved that document through every position in the set. They tested GPT-3.5-Turbo and Claude 1.3 with their extended-context versions, MPT-30B and LongChat-13B, with GPT-4 on a subset, and a second task that asked models to retrieve a value by its key from a long list.

What it found

Accuracy followed a U shape. It was highest when the answer was at the start or the end of the input, and lowest in the middle. With twenty documents, GPT-3.5-Turbo answered 75.8% correctly when the answer came first and 53.8% when it sat in the middle — a fall of 22 points from moving one document. Claude 1.3’s fall was smaller, from 59.9% to 55.9%.

In the middle, the documents made things worse. Asked the same questions with no documents at all, GPT-3.5-Turbo scored 56.1%. With the answer buried in the middle of twenty documents, it scored below that. Handing it the right material, in the wrong place, did more harm than handing it nothing.

A bigger context window did not mean better use of context. The extended-context versions of these models performed almost identically to the standard versions on inputs that fitted both. Being able to accept more did not make them use it better.

More documents bought very little. Going from twenty retrieved documents to fifty raised accuracy by about one and a half points for GPT-3.5-Turbo and one point for Claude 1.3, at the cost of a much longer input. The authors suggest ranking the relevant material first, or cutting the list, rather than adding to it.

Repeating the question helped retrieval, not reasoning. Putting the question both before and after the documents brought key–value retrieval close to perfect, but barely changed results on the question-answering task.

What the vendor recommends now

Anthropic’s current prompting guide, written for current models, recommends putting long documents and data at the top of the prompt, above the question, instructions and examples. It adds that questions at the end improved response quality by “up to 30 percent” in its own tests, especially with several long documents. That figure is the vendor’s and its method is not published, so treat it as a direction rather than a measurement — but it is the same direction as the study.

What follows for how you work

Put the material first and the question last. Long documents, data and reference text at the top; the task and the question at the end. Both the study’s U shape and the vendor’s guidance point the same way.

Do not bury what matters. In a long instruction, the rules that must not be missed belong at the start or the end, not in the middle of paragraph nine.

Curate rather than accumulate. The most useful result here is that relevant material in the wrong position can leave a model worse off than no material at all. Adding everything that might help is not neutral. Rank the relevant first and cut the rest — the same argument as What earns its place, measured on someone else’s benchmark.

Prefer short files fetched when needed. A context file loaded every session is a long input whose middle is where the easily missed rules live. Keeping each file short, and pointing to the rest so it is read only when it applies, is what a pointer rootnote is for — and this is evidence for why it matters.

Do not treat a large context window as permission. The ability to accept a long input is not evidence that all of it will be used.

How far this reaches

The models tested date from 2023. Current models may handle long inputs considerably better — nothing cited here measures them — and the specific figures — the 22-point fall, the drop below no documents — should not be read as descriptions of them. That the vendor’s guidance for current models still recommends this ordering suggests position has not stopped mattering entirely, but it is not a measurement of how much.

The tasks were retrieval-shaped. Finding one relevant document among many is not the same as following a long set of instructions, and the study does not measure the second directly. The recommendation to keep context files short and well ordered is our inference from it, not its finding.

The sources were retrieved on the date shown in this document’s header.