> We could not get ground truth, and it took three attempts to accept it — from The Handover, the-handover.org/docs/we-could-not-get-ground-truth
> Authors: Leon Mallett (captivated.online) with Claude Code · Last confirmed working: 2026-08-28
> © Captivated Ltd — free to use in your own work, not to redistribute as a collection. the-handover.org/licence

A tool was built to score about forty software projects on how close each is to shippable.
Roughly seventy scoring statements, seven dimensions, a level per project.

The obvious question is whether the scores are any good. The obvious answer is to ask the
person who owns the projects and compare. That was attempted three times. All three failed,
and the third failure is the one that mattered, because it revealed the first two had been
asking a question the situation could not answer.

## Attempt one: rank the projects

The ask: sort the projects from most to least ready. Compare against the tool's ordering,
measure the correlation.

Two problems, and the second is the interesting one.

**Underpowered.** Rank correlation over a few dozen items with heavy ties gives a confidence
interval wide enough to contain both "excellent" and "no better than alphabetical." A
plausible-looking coefficient would have been noise dressed as a result.

**Near-circular.** The projects are built almost entirely by coding agents. The owner's sense
of which are ready is formed substantially by what those agents reported. The tool reads the
same artefacts those reports were drawn from. Agreement would mostly measure that both
parties had read the same files.

Not a technicality. **A correlation between a tool and a human who formed their belief from
the tool's own inputs is close to a self-correlation**, and it looks exactly like validation.

## Attempt two: predict each criterion

Narrower and apparently sounder: before running the scan, mark each statement pass or fail
for a handful of well-known projects. Disagreements are then specific and diagnosable.

The reply:

> "Ah, I see, I basically have no idea"

Which is not a failure of engagement. It is the honest state. Predicting seventy statements
across ten projects means holding several hundred facts about code you did not write and
have not read recently. Nobody has that. The exercise assumed a memory of the portfolio that
does not exist in anybody, including the person who commissioned it.

## Attempt three: ask about the proxies instead

If the scores can't be checked directly, check whether the *measures* are fair. Do these
proportions capture what they claim?

> "The questions you're asking me are technical questions about ratios of files and such,
> I don't know"

Correct, and the reason is structural. A question like *"is the share of source files that
import the logging library a fair measure of whether a project logs properly?"* is a question
about **the tool's internals.** The owner is the authority on his projects, not on the
implementation of a scanner someone else wrote about them. The questionnaire had made him
responsible for the design decisions it was meant to check.

## What the three had in common

All three asked the portfolio owner to be a measuring instrument — to hold, in his head,
calibrated knowledge of forty codebases so it could be compared against a machine.

**That knowledge is the thing the tool exists to produce.** He does not have it; that is why
it was built. A validation method that requires it asks the subject to supply the output in
advance, and then congratulates the tool for reproducing it.

The three attempts were not obstacles on the way to validation. **They were three
independent confirmations that the tool's premise was right.**

## What was done instead, and what it does not cover

Two things had been conflated under the word "validation", and separating them was the whole
of the fix.

**Does the tool detect what it claims?** Answerable without any human. Build a synthetic
project that satisfies a statement, break one thing, assert the intended score flips —
**and assert nothing else moves.** Ground truth is manufactured by construction: the
degradation is known because it was applied.

The second assertion carries most of the weight. It catches coupling between statements that
no single-outcome test can see. It caught four cases where a rule about which scores a change
invalidates started too broad — every one.

Two conditions keep it honest. Each variant must differ from the healthy fixture in exactly
**one** respect: a fixture that carried a build command in two places once made deleting the
README a no-op, and the test passed. And where a change legitimately moves two scores, both
must be written down in advance, or the exact-set assertion degrades into "something
changed."

**Are the statements the right statements?** This has no oracle and never will. There is no
experiment that shows whether "does this project have a licence file" belongs in a
readiness score. It is settled by written argument — a numbered decision record with the
reasoning and the rejected alternatives.

**That is weaker than a test and must never be reported as one.** The distinction is written
into the tool's own documentation because it is exactly the thing that erodes: after a few
months, "we reasoned about this carefully" and "we verified this" compress into the same
sentence.

## Two limits worth naming

**A synthetic fixture cannot refute its author.** It contains what its author thought to put
there, so it cannot fail in a way its author did not anticipate. One check — built
specifically to avoid a known trap — passed its fixture throughout while being wrong in
exactly that way, because the fixture placed the evidence where the check looked. It was
caught by running the tool on real material nobody had written to satisfy it.

Two of the tests were therefore pointed at the actual projects rather than a fixture. They
assert consistency rather than correctness, which is much less than validation — but they
can surprise their author, which the fixtures cannot.

**Coverage is not accuracy.** Every statement has an injection test and the exemption list is
empty. That establishes each one detects the thing it was built to detect. It says nothing
about whether the resulting number means anything, and a coverage figure is very easy to
report as though it did.

## If you are in the same position

**Ask what your validation method requires the human to already know.** If the answer is
"the thing the tool produces", the method is circular no matter how carefully the statistics
are done. The circularity in attempt one survived a great deal of thought about sample size,
because sample size was not the problem.

**Separate "does it detect what it claims" from "is it measuring the right thing."** The
first is testable. The second is an argument. Conflating them lets the first one's rigour
launder the second one's absence.

**Manufacture ground truth where you can't obtain it,** and assert the *exact set* of things
that changed rather than the one you expected. The second assertion finds more than the first.

**Say plainly which parts are unvalidated.** A tool that scores things has an authoritative
interface — a number, a level, a colour. That interface makes no distinction between the part
backed by tests and the part backed by an argument someone wrote down one afternoon. Only the
prose can.