> Test output against the tools people actually have — from The Handover, the-handover.org/docs/test-against-the-tools-people-have
> Authors: Leon Mallett (captivated.online) with Claude Code · Last confirmed working: 2026-08-08
> © Captivated Ltd — free to use in your own work, not to redistribute as a collection. the-handover.org/licence

If you generate a file that something else has to read — an archive, an export, a
feed, a config — the test that matters is whether **the readers your users have**
can read it. A round trip through your own writer and your own reader proves the
two agree with each other, which they will, because you wrote both.

## The case

A small ZIP writer, hand-rolled to avoid shipping a compression library to every
visitor. Store-only, deterministic, about 120 lines. The unit tests passed:
CRC-32 matched the published check value, output was byte-stable across runs,
entry ordering behaved.

Then a test shelled out to the system `unzip`. Two failures, immediately, neither
of which any self-consistent test could have surfaced.

### UTF-8 filenames: valid, flagged correctly, still unreadable

The ZIP format supports UTF-8 entry names. Bit 11 of the general-purpose flag
declares them, and the writer set it correctly.

The archive passed `unzip -t` — integrity fine, no errors. Then extraction failed:

```
error: cannot create out/docs/unicode-café.md
       Illegal byte sequence
```

The version of Info-ZIP shipped with macOS mangles the name in its listing and
refuses to extract it. The archive is not malformed. The reader is old, and it is
the reader millions of people have.

**Being right about the specification is not the same as being readable.**

The fix was not to work around the reader. Entry names in this application come
from slugs, which are ASCII by construction, so a non-ASCII name means something
upstream is wrong. The writer now **refuses** such names with an explanation,
rather than emitting an archive some readers cannot open:

```
createZip: unsafe entry name "docs/café.md". Names must match /^[A-Za-z0-9._\-/]+$/
```

Refusing to produce output you cannot verify is a legitimate answer, and often a
better one than producing it hopefully.

Worth separating: **UTF-8 *content* was never the problem.** Document bodies full
of em dashes and accented text round-trip perfectly. Only the *filename* is
affected, so the test now extracts for real and reads the files from disk, rather
than matching names on a command line.

### Empty archives: structurally valid, practically broken

An archive with no entries is well-formed — a central directory with zero records
and an end-of-directory marker. The writer produced one happily.

`unzip -l` responds:

```
warning [empty.zip]: zipfile is empty
```

and exits **1**. A user handed that file sees an error from their own tools.

An empty pack is also a bug in whatever assembled it. So the writer now throws
rather than emitting one — the failure surfaces where it originated instead of
being posted to someone else.

## Why the unit tests could not have found either

Every passing test shared an author with the code under test. CRC values,
determinism and ordering were all checked against **my** understanding of the
format. Where that understanding was complete, the tests were sound. Where it
was incomplete — how a 2009-era extractor handles a UTF-8 flag, how it reacts to
zero entries — the tests were incomplete in exactly the same places, and silently.

That is the general shape: **a test written from the same model as the
implementation cannot find an error in the model.** It can only find errors in
the implementation of it.

The system `unzip` had no such shared assumptions, which is the entire reason it
was worth invoking.

## What to do

- **Invoke a real consumer in the test suite.** `unzip`, an XML validator, a feed
  reader, `jq`, the actual client library. If a CI runner has it, the test can use
  it.
- **Prefer extraction to inspection.** Extract to a temp directory and read the
  files back, rather than asking the tool about its own listing.
- **Test the boundaries the format allows but consumers dislike** — empty,
  single-entry, non-ASCII, very long names, nested paths.
- **Refuse rather than hope.** If you cannot verify an output shape works, make
  producing it an error with a message saying why.
- **Pin the finding in a test** with a comment naming the reader and the version.
  "Old unzip cannot do this" is a fact with a shelf life, and the next person
  needs to know what to re-check.

## The generalisation

The tools you build with are newer, stricter and better-configured than the ones
your output lands in. Whenever the thing you produce leaves your process, the
question is not whether it is correct. It is whether it is correct **for the
reader on the other side**, and that is a question only that reader can answer.