Four ratios, four different failures

A tool scoring about forty software projects used four ratios. Each looked reasonable. Each was replaced, and the four failures share no mechanism — which is why the rule at the end is worth more than the four fixes.

The threshold that ranked nothing

The first ratio measured adoption of a structured-logging library: the share of source files importing it. The rubric guessed a threshold of 30%.

Measured across every project that used the language, the highest value anywhere was 22.7%. The threshold was above the observed maximum. Every project failed, and would have failed no matter what any of them did, because nobody in the population was near it.

That was caught immediately by plotting the distribution, and it produced a rule that felt sufficient at the time: derive thresholds from data rather than guessing them. Everything below is a consequence of that rule being wrong.

Failure one: it punished good structure

Recalibrated to 10%, the same ratio then failed a project that logged correctly.

That project built its logger once, in a single file, and called that wrapper from the other 146. Which is the better structure — one place to configure, one place to change. On a share-of-files-importing-the-library measure it scores 1/147.

The criterion would have failed a project for structuring its logging well. Worse, because these scores drive advice: the tool would have told a developer to scatter their imports. A metric that instructs worse code is not a weak metric, it is a harmful one.

The fix was to stop measuring adoption and ask the question directly: is the logging library used at all, and does the project print diagnostics instead? Neither half is a proportion.

Failure two: it punished being a command-line tool

The replacement had a second clause counting print statements — a project that logs properly should not also be printing.

For a command-line tool, print is the output. A tool that prints its results is doing its job. The criterion, as written, told CLI authors to stop printing.

The fix came from noticing that the distinction was already available and being ignored: stdout is what the program was asked to produce; stderr is what it says about itself. Counting hand-written writes to the error stream separates diagnostics from output without any judgement about which module a line sits in.

Paired with a tell-tale rather than a volume: a debug-format specifier inside a print statement is a debugging session nobody cleaned up. One is enough, so there is no count and nothing to calibrate.

Failure three: it measured how much code existed

A third ratio counted unsafe unwrapping operations per source file, against a threshold of 3.0 derived from the distribution.

A 500-file project with 100 of them scores 0.2. A 20-file project with 60 scores 3.0. The second is judged worse, and neither number says whether any of them can actually fire.

Replacing it exposed something the ratio had hidden. The criterion’s own wording said “in non-test code”, and the counter sat one line outside the guard that excludes tests — so it counted the places where unwrapping is correct. A test that unwraps is asserting the value is there.

One project’s count was 84 against an actual figure of 2. Another’s was 888 against 65. Seven of ten projects were failing for the way they wrote their tests, and the ratio made it invisible by turning it into a plausible-looking number.

The replacement asks whether a failure is reachable by ordinary input: unwrapping a mutex lock fails only if another thread has already crashed; unwrapping a file read fails whenever a file is missing. Counting the second kind needs no threshold, because the honest bar is none.

Failure four: it measured file granularity

The last ratio was the share of source files carrying tests.

A well-decomposed project has more, smaller files — so the same test coverage spreads thinner and scores worse. Split one tested file into two and the ratio halves unless a second test file appears. The defect is true before any measurement.

Measured anyway, within a single project type: mean source-file size correlated with the ratio at ρ = +0.612, p = 0.013. Projects with bigger files scored better at “having tests”.

The replacement asks about a window rather than a share: of the twenty most recently changed source files, do more carry tests than not. A window counted in files is the same size in a monolith and in a decomposed project, so it measures testing habit rather than carving.

The confound fell to ρ = +0.424 (p = 0.103). Reduced, not eliminated, and part of what remains has a named mechanism — an inline test block marks its whole file as tested, so larger files still count more easily. Recorded rather than claimed as fixed.

The rule

If you find yourself calibrating a threshold for a scoring metric, the metric is proxying rather than measuring.

The failures share no mechanism. What they share is shape. A ratio needs a threshold; a threshold has to come from somewhere; and the only place available is the observed distribution. Calibrating against your population makes the metric agree with the population — which is the one thing a metric meant to find problems must not do. It also buries the real question, which is whether the numerator and denominator mean what the statement claims.

What replaced them

Each ratio became a statement with no number in it.

• Is the logging library used, and do diagnostics bypass it?
• Can this failure be reached by ordinary input?
• Of the files changed most recently, do more carry tests than not?

None can be gamed by rearranging code without changing behaviour. None needs re-deriving when the population changes. And each is answerable by reading one project, which the ratios were not — a percentile is a fact about a population, not about the thing you are looking at.

Where a number is still allowed

A structural number is not a calibration. “Three files may write to the error stream by hand” comes from a program has one top-level error handler, and a workspace may have one per binary. Not from the spread.

The test is whether you could have written the number down before seeing the data. If not, it is a calibration wearing a principled coat.

One consequence worth expecting

Retiring the ratios retired something else. A whole category of “this score rests on a measurement within one file of its threshold” disappeared — because marginal is only a question a ratio can raise. Machinery built for it had fired exactly twice in its life, both on the last surviving ratio.

If a measurement system has features that only make sense for proportions, expect to lose them too. That is a good sign rather than a cost.