> The strongest evidence against this library, read carefully — from The Handover, the-handover.org/docs/agents-md-does-not-help
> Authors: Leon Mallett (captivated.online) with Claude Code · Source retrieved: 2026-09-14
> © Captivated Ltd — free to use in your own work, not to redistribute as a collection. the-handover.org/licence
> Position — The Handover's own view, stated in full. Not independently vetted.

This library argues that an agent should be handed written context it cannot
discover for itself. The strongest published evidence against that position is a
controlled study, and it deserves reading properly rather than being explained
away.

**Read the paper.** What follows is a reading, not a replacement:
[arxiv.org/abs/2602.11988](https://arxiv.org/abs/2602.11988).

## What they did

Gloaguen, Mündler, Müller, Raychev and Vechev evaluated whether repository-level
context files actually help coding agents complete real tasks — which, as they
note, had not been rigorously studied despite the practice being strongly
encouraged by agent vendors.

They built **CTX BENCH**: 138 instances drawn from real GitHub issues across 12
recent, niche repositories, covering both bug fixes and feature additions, every
one of them a repository whose developers had committed a context file. Niche
repositories matter here — popular ones are likely to be inside the training
data, which would contaminate the result. They ran SWE-bench alongside it for the
LLM-generated condition.

Three settings: **no context file**, an **LLM-generated** one produced using the
agent vendors' own recommended prompts, and the **developer-committed** file the
repository actually shipped. Across four agent-and-model combinations.

## The headline, which is the least interesting part

> "providing context files does not generally improve task success rates, while
> increasing inference cost by over 20% on average"

And it held across models, across agents, and for both kinds of file.

Taken alone that reads as a refutation of everything here. It is not, and the
reason is two paragraphs further into the same abstract.

## The finding that matters

> "while instructions in the context files are well followed by coding agents,
> repository overviews, although popular and recommended by model providers, are
> not helpful"

**That is this library's own rule, arrived at by measurement rather than by
argument.**

The bar for a rootnote, in the contributor guide, is: *it encodes what cannot be
discovered. If an agent with the repository in front of it could work the answer
out, that belongs in code or ordinary documentation.* A repository overview is
the purest possible violation of that — it is a written description of the thing
the agent is already holding.

The paper found that 8 of the 12 developer-committed files contained a dedicated
codebase overview, and 4 enumerated the directory structure. Two of the three
vendor generation prompts explicitly instruct the agent to write one.

So the most common content in real context files is the content the study finds
useless, and it is there because the tool vendors ask for it.

Their own recommendation lands in the same place:

> "Human-written context files should only include instructions required for
> coding agents that are not already present in the README (e.g., specific
> conventions or non-functional requirements)"

Restated, that is: write down what cannot be discovered, and nothing else. Two
groups reached it independently — one by arguing from how these systems behave,
one by running 138 tasks across four models. The second is better evidence than
the first, and this library should say so.

## Three places it genuinely challenges us

**We have never once mentioned cost.** Not in the system document, not in any
rootnote, not in the working agreements. The study puts it at over 20% in
inference, driven by instructions being *followed* — more exploration, more
testing, more reasoning. That is a real charge against every document here that
tells someone to write something, and this library has been silent about it
while recommending the practice. It is the clearest gap the paper exposes.

**The 7% is not the win it looks like.** Developer-committed files beat
LLM-generated ones by 7% on average, which is significant, and the temptation is
to read it as vindication of human-written context. It is not. The overall
finding remains that including a context file *does not significantly affect
performance* — so what the 7% establishes is that human-written files are
**less harmful than generated ones**, not that they beat writing nothing. Anyone
here quoting the 7% without that sentence attached would be doing the thing this
corpus publishes documents about.

**"Rigorously evaluated before adoption" is an instruction we have not
followed.** This library already concedes as much — its own document on ground
truth reports three failed attempts at validation and concludes there was no
oracle available. The paper is someone building the oracle and running it. That
is a straightforward demonstration that the evaluation was possible and we did
not do it.

## Their tasks have discoverable answers. That is the whole disagreement.

Every instance in both benchmarks is issue resolution on a repository the agent
can read — bug fixes and feature additions where **the information needed is in
the repository**. That is precisely why they found overviews useless: the agent
can already see it, and a written description of a thing you are holding is
redundant by construction.

Now take a case from this portfolio. An agent needed to provision a Windows VM
for testing. It cost hours: which VM sizes are actually available under this
subscription, that several previously-recommended ones now return
`NotAvailableForSubscription`, which region has quota arranged, the per-user
versus per-machine installer trap that makes a passing test prove nothing.

**None of that is in any repository.** It is not in the code, the manifest, the
git history or the README, and no amount of reading them surfaces it, because it
is a fact about an external system with a history. It went into a rootnote, and
every agent since has followed it instead of rediscovering it.

The two findings do not collide. **They describe disjoint classes of task**, and
the line between them is the paper's own recommendation — *only include
instructions not already present in the README*. Their instances all sit on one
side of that line. The rootnote class this library exists for sits on the other.

## Where evidence is the wrong instrument

There is a temptation, having read a careful study, to demand the same rigour of
everything. It is worth resisting in one specific place.

That the second agent completes in minutes what the first spent hours
discovering is not a hypothesis about the world. **It is analytic.** The
alternative requires the second agent to independently rediscover the quota
arrangement, the unavailable VM sizes and the installer trap — and there is no
mechanism by which that is fast. Asking for a controlled trial is asking someone
to demonstrate that a map helps a traveller who does not know the terrain.

The study measured a question that genuinely was open: does a repository
overview help an agent that can already read the repository? That deserved
measuring and the answer is no. Whether written knowledge of an external system
helps an agent that has no other access to it is not the same kind of question.

Two things stay empirical, and they are the ones that matter:

**Magnitude.** *That* it helped is settled. *How much* is a number, and this
library does not have it, because nobody recorded what the first pass cost while
it was happening. Cheap to write down at the time; not recoverable afterwards.

**Which documents clear the bar.** The analytic argument establishes that the
mechanism works for non-discoverable knowledge. It says nothing about whether any
particular file contains any. The 20% is charged on every session regardless.

## We applied their finding to ourselves, and it found something

The paper's sharpest result is that 8 of 12 real developer-committed files
contained a codebase overview — content the agent could already see, costing
inference and returning nothing, written because two of three vendor prompts ask
for it.

So we audited this library's own twelve rootnotes for the same failure. **None
contains a codebase overview or a directory listing.** On the paper's specific
finding, the bar held.

It found a different problem. This library's stated bar is *if an agent with the
repository in front of it could work the answer out, it does not belong here* —
and that catches overviews while missing a second class entirely: **content the
model already knows.** Passages like *use the element that means what you mean*,
*no ARIA is better than bad ARIA*, *do not hand-format dates, use the platform
formatter with a locale.* All correct. All well written. All already in the
weights of any current model, and therefore costing the same 20% for the same
nothing.

The division runs along the two halves of the rootnote bar. **The boundaries hold
and the craft advice does not.** What may run unattended, where work stops, what
was rejected and why, which jurisdiction, what counts as personal data *here* —
none of that is inferable from anywhere. The general good practice around it is
padding that reads as thoroughness.

We tried to put a number on the ratio and the measurement over-flagged: it marked
*what was rejected, and why* as generic because the sentence contains no
project-specific token, when it is the least inferable thing in the document. A
metric that needs calibrating is proxying rather than measuring, so the number is
not reported. The structural finding stands without it.

The bar has been widened accordingly: **what an agent cannot discover from the
repository, and would not already know.**

## What we would have to run

A trial worth publishing would need a measure this library actually claims:
something about sessions rather than tasks. The most tractable version:

- Take repeated cold-start sessions on the same project, with and without the
  rootnote set.
- Measure how often the agent asks a question already answered in a rootnote, how
  often it proposes something explicitly rejected before, and how often it crosses
  a boundary the working agreement names.
- Report cost alongside, because the study is right that it is part of the ledger
  and we have been ignoring it.

Those are countable, they are what the documents claim to affect, and a null
result would be worth more than the argument currently on this site.

Until that exists, the accurate summary of our position is: **the detailed
findings support the rule this library applies, the headline finding is
unrefuted, and the cost is real and unacknowledged.**

---

*Gloaguen, Mündler, Müller, Raychev and Vechev are welcome to respond. A reply
will be published unedited and linked from this document: `hello@the-handover.org`.
Nothing has been sent to them; this is an open invitation rather than a request.*