When the tool disagrees with you, neither of you gets to win

A scanner scores about forty software projects on readiness. It exists because those projects are built almost entirely by coding agents, and an agent grading its own work grades generously. So the rule attached to it is blunt:

Do not assess readiness yourself and do not hand-write scores. If the tool disagrees with you, say so in the log rather than adjusting the score.

That rule is load-bearing. Without it the scores drift toward whatever the author already believed, and the tool becomes an expensive mirror.

It also means that every mistake the scanner makes arrives as an instruction. Here is what that cost, three times.

The comment that was wrong about the world

A check asked whether a site discloses its use of cookies. It skipped that check for projects with no user-facing surface, and the exclusion carried a one-line justification in the source:

// A published site sets cookies whether or not it has a database.

That sentence is false. A static site with no third-party embeds, or one using a privacy-focused analytics tool that sets none, sets no cookies.

Worth being precise, because this is the one claim here a reader might act on to their cost. The rule in the UK and EU is about storing or accessing information on someone’s device, not about the word cookie. Local storage, session storage and device fingerprinting engage it too. “We set no cookies” and “we store nothing” are different findings, and only the second ends the question. What was established for these projects was the second. If you are checking your own, check for the second.

Twenty-four projects were failing a disclosure requirement they did not have. The scanner’s advice was to add a cookie banner to sites with no cookies — which is not merely noise. It is a worse outcome than silence: an unnecessary consent dialog on every visit, and an inaccurate statement in a legal document.

It was found by the portfolio owner reading the output and saying “the analytics code is for a tool which has no cookies.” Not by any test. The assumption had been written down, in a comment, right next to the code — and a written-down assumption is invisible to every test, because tests check behaviour against intent and this was wrong at the level of intent.

The disagreement where the tool was right and I was wrong

Later, in the same area, I claimed a particular analytics tool was present in one project’s source. The owner said it was not.

He was right. My search had matched the English word inside an unrelated identifier — is_plausible_email — while looking for a product whose name is that same word.

Worth stating plainly because the previous section invites the opposite conclusion. The human is not the tiebreaker. In one case the scanner encoded a false belief about the world; in the other, a person querying it made a sloppy inference and the record was correct. A resolution procedure that says “defer to the human” fails the second case as badly as “defer to the tool” fails the first.

The scanner was right and the output was still harmful

The tool showed a wall of failures across every project. It was accurate: each statement was genuinely unsatisfied.

Ninety-one per cent of it was work not yet due — requirements attaching to a later stage than the project had declared. An early prototype was being shown its production security obligations with the same weight as its missing lockfile.

Nothing was wrong with any individual score. The interface was wrong. A list of true statements, sorted badly, is a tool that trains its users to stop reading it — and once they have stopped, the 9% that was due goes with it.

Why this is worse than an ordinary bug

Most bad tools are ignorable. A linter with a noisy rule gets the rule disabled and life continues.

A tool built specifically so that its judgement outranks yours cannot be ignored — that is the property it was constructed to have. And the interface is a number, a level, a colour. Nothing in the output distinguishes a score resting on a well-tested detector from one resting on an assumption somebody typed into a comment. The presentation is uniformly confident because presentation has no access to the difference.

So a half-built scanner with an authoritative interface is not a neutral work-in-progress. It is an artefact that reliably instructs people to do wrong things, with the full weight of “the tool says so” behind each instruction.

The procedure that came out of it

The mistake in all three cases was treating the disagreement as being about the score. It never is.

A disagreement is a bug report against the statement, not a dispute about the result. Two legitimate outcomes, and adjusting the number is not one of them:

Fix the statement. If the statement is wrong — measures the wrong thing, encodes a false belief, applies where it should not — change the statement and rescan. The score then moves because the rule changed, which is visible and reviewable. The cookie case ended here: the exclusion was rewritten around whether a site actually stores anything on the visitor’s device, and twenty-four false failures disappeared as a consequence.

Record the disagreement and leave the score. If you believe the statement is right in general and wrong about you specifically, write that down where the next person will find it — and leave the failing score in place. It costs something to carry a failure you disagree with, and that cost is the point: it keeps pressure on either fixing the statement or accepting it, rather than letting a local exemption quietly settle the matter.

Never edit the number. A hand-adjusted score is indistinguishable from a correct one afterwards, and it destroys the only property the tool had.

Three things that made the procedure work

Assumptions in comments must be checked like code. The cookie sentence had been read many times and never evaluated, because it sat where explanations sit rather than where claims sit. Any sentence in a comment asserting a fact about the world is untested and probably unexamined.

Sort by what is due, not by what is unsatisfied. A failure list must lead with obligations the thing has actually incurred. Everything else is a roadmap and belongs behind a separate heading. This is not cosmetic — it decides whether the output gets read at all.

Distinguish a state you can’t determine from a state you’ve checked. Adding an outcome meaning “this cannot be established from what is visible here” — as distinct from “no” — removed a whole class of arguments, because a large share of disputes were about claims the scanner was never in a position to make. Naming that gap stops it being scored as a failure.

The part that does not resolve

Deferring to a tool you know to be imperfect is uncomfortable, and it should be. The rule is not “the tool is right.” It is that the alternative — the author of the work also grading the work — is reliably worse, and its errors are invisible where the tool’s are at least written down somewhere a person can argue with.

If that trade stops being worth it, the answer is to fix the statements or stop using the tool. It is never to keep the authority and quietly override the outputs.