> Operating something that is live, and what an agent may do during an incident — from The Handover, the-handover.org/docs/operational-runbook
> Authors: Leon Mallett (captivated.online) with Claude Code · Last confirmed working: 2026-08-26
> © Captivated Ltd — free to use in your own work, not to redistribute as a collection. the-handover.org/licence

For anything deployed. Two parts: what a runbook must contain, and what an
agent may do when something is actually wrong.

**Read the scope before the advice.** Most of what follows assumes software *you
run* — a service on infrastructure you control, with dashboards, logs you can
query, and a deployment you can roll back. Software you *ship* breaks those
assumptions comprehensively, and there is a section at the end for it. Applying
the service advice to a desktop or mobile application produces instructions that
are mostly wrong.

**Escalate by: {{escalation_path}}**

**How you find out: {{failure_signal}}**

## Why an agent needs explicit rules here

An incident is the situation where an agent is **most capable and most
dangerous**. It can read logs faster than a person, correlate across services,
and act in seconds — and it is under exactly the pressure that makes acting feel
obviously correct.

Every ordinary boundary is weakest here. "Prepare, do not enact" is easy at
four in the afternoon and hard when something is down, which is precisely why
it has to be written rather than assumed.

The specific failure: an agent diagnoses confidently, applies a fix, and the
fix is wrong in a way nobody can see because the evidence was already changed
by the fix.

### Operating alone

If the escalation path above is nobody, that is not a gap in the setup — it is
the setup, and it changes which advice is load-bearing.

"The more obvious the fix seems, the more it needs a second person" has no
mechanism behind it when there is no second person. Do not read it as advice to
ignore; read it as the reason the rest matters more:

- **Prefer the reversible action** stops being a preference and becomes the only
  safeguard you have. A rollback does not require a second opinion to be correct,
  which is precisely why it substitutes for one.
- **Preserve evidence before changing anything**, because you are also the only
  person who will ever reconstruct this.
- **Record while acting**, because the second reader is you, later, with none of
  the context you have now.
- **Slow down instead of escalating.** The pause that a second person would have
  forced still has value when you take it yourself; it is just harder to make
  yourself take it.

The self-review problem below is sharper when operating alone, not softer. There
is nobody to catch it.

## The order of operations

**1. Stop the bleeding before understanding it.** Restore service first.
Diagnosis is what you do once nobody is affected. A perfectly-understood outage
that lasted an hour is worse than a resolved one nobody has explained yet.

**2. Prefer the reversible action.** Rolling back to a version that worked is
almost always faster, safer and more certain than fixing forward under
pressure — and it is reversible if it turns out to be unrelated.

**3. Preserve evidence before changing anything.** Capture the logs, the
metrics, the failing response, the current version. A restart frequently clears
the only proof of what happened, and "it resolved after a restart" is a sentence
that guarantees a recurrence.

**4. Record while acting, not afterwards.** Note each action and its timestamp
as you take it. Reconstructing a timeline from memory is unreliable, and during
an incident it is the first thing to go.

**5. Say something early.** A holding message costs nothing. Silence during an
outage is interpreted as absence, and people start acting on guesses.

## What an agent may do

**Freely — observing changes nothing:**

- Read logs, metrics, traces and dashboards.
- Query read replicas and staging.
- Reproduce locally.
- Draft a diagnosis, a fix, a rollback command, and a holding message.

**Never without a human, however clear the answer looks:**

- **Deploying anything, including a rollback.** Rolling back is the right move
  and it is still a deployment.
- **Restarting or scaling production services.**
- **Modifying production data.** Especially "just fixing" a few bad rows — the
  bad rows are evidence, and a corrective update with a mistaken predicate is a
  second incident inside the first.
- **Changing configuration, flags or limits in production.**
- **Rotating or revoking credentials**, unless a credential is the incident, in
  which case that is the escalation.
- **Communicating outside the team** — status page, customers, anywhere public.

The strongest version of this: **the more obvious the fix seems during an
incident, the more it needs a second person.** Confidence is the symptom, not
the qualification.

### And you may be the cause

The agent reading the logs is frequently the agent whose change is in the deploy
that broke. It is then reviewing its own work, at speed, using **the same model of
the system that produced the fault** — and a bug is precisely a place where that
model and reality disagree. Confidence will be highest exactly where the model is
wrong.

So: **if your own change is in the failing deploy, you are not the second
person.** Say so explicitly rather than leaving it unmentioned, and prefer a
rollback over any fix you author. A rollback does not depend on your diagnosis
being correct, which is the property you need and the one you cannot currently
verify.

This is the same circularity as reviewing your own code, and an incident is where
it does the most damage — because the pressure is highest, the evidence is
perishable, and nobody wants to slow down to say "the thing that broke is mine".

## Related rootnotes

- **The documentation standard** defines the file this lives in and how it links
  to the rest of the set.
- **The working agreement** covers ordinary work. This document overrides it
  during an incident, and only during one — in both directions: some things
  normally requiring a human are still forbidden here, and the reporting standard
  gets stricter rather than looser.

## What a runbook must contain

One per deployed service or shipped artefact, in the documentation set and linked
from Architecture. A single one lives at `docs/runbook.md`; **several follow the
set's own rule for a section that outgrows one file** — promote to
`docs/runbook/` with a `README.md` index inside, one file per service. A project
with a desktop application and four workers has five, and they should not be one
document.

- **What it is and what breaks if it stops.** Who is affected, how quickly, and
  what they see. This is what decides urgency.
- **Where it runs** — platform, region, and how to reach the console.
- **How to tell whether it is healthy**, including the specific check, its
  expected value, and where to look.
- **How to reach the logs**, with the query that actually finds errors, not just
  the log viewer's address.
- **How to roll back**, written as the literal commands, tested at least once
  when nothing is wrong.
- **Known failure modes** and what each looks like from the outside.
- **Dependencies**, and what happens when each is unavailable.
- **The escalation path**, including out of hours.

**The test for a runbook: could someone who has never touched this service use
it at three in the morning, half awake, with no context?** If a step assumes
knowledge, it is not a runbook step yet.

**A rollback procedure that has never been executed is a hypothesis.** Run it
once deliberately, in a calm moment, and fix what you find. Everyone discovers
their rollback is broken during an incident otherwise.

**Record when it was last actually run**, on the line beneath the procedure:

```
Last executed: 2026-08-25 (staging, full restore, 4 minutes)
```

A rollback tested in March and one never tested read identically without it. That
line is the only thing that distinguishes a proven procedure from a written one,
and it goes stale visibly, which a claim of correctness does not.

## Software you ship rather than run

Everything above assumes you control the running copy. For a desktop
application, a mobile app, a firmware image or a published library, almost every
assumption inverts — and the difference is worth stating plainly, because an
agent applying the service playbook here will confidently do the wrong thing.

**There is no dashboard**, because the software is running on other people's
machines. You learn about failures from reports, crash telemetry if you have it
and people consented, or reviews. Absence of evidence is close to worthless: most
people who hit a bad build never tell you.

**There is no restart you control**, and no scaling.

**Rollback is not a rollback.** It is publishing a *new* version that behaves like
the old one, and then waiting — for an updater to run, for a store to approve, for
people to launch the software at all. The gap between "fixed" and "fixed for
users" is days, sometimes never for those who do not update.

**The blast radius does not shrink when you fix it.** A bad build stays on every
machine that already has it. This is the deepest difference: in a service, time
after the fix reduces harm; in shipped software, the population holding the broken
version only shrinks as fast as people update.

What follows from that:

- **Staged rollout is the primary safety mechanism**, because it is the only one
  that limits the population before the damage is done. Release to a fraction,
  wait long enough for reports to arrive, then widen. The waiting is the control;
  shortening it removes the point.
- **A kill switch beats a fix** for anything that can be disabled remotely. Turning
  a broken feature off reaches everyone at their next launch, without an update,
  an approval or a download.
- **The version that shipped must be reproducible.** You cannot diagnose a build
  you cannot rebuild, and the reporter is running a binary from three weeks ago.
  Keep the artefact, its exact inputs and its symbols.
- **Have a way to know what version someone is running**, and make it easy for
  them to tell you. Most bug reports are unactionable without it.
- **Never break the updater.** It is the only channel through which anything is
  ever fixed. A bug that prevents updating is the one failure with no remedy but
  asking every user to reinstall manually — treat that path as the most
  safety-critical code you ship, and test it on the *previous* release rather than
  the current one.

The runbook for shipped software therefore answers different questions: how a
release is staged and halted, what can be disabled without an update, how to
identify an affected version from a report, and how to reproduce a build that is
already in the world.

## Afterwards

**Write it down while it is still fresh**, and make it about the system rather
than the person. What happened, what the impact was, what the actual cause was,
what was tried that did not work — that last one saves the next person the same
hour.

**The action that prevents recurrence is worth more than the analysis.** An
incident record with no follow-up change is a diary entry.

**Update the runbook in the same pass.** Whatever was missing at three in the
morning is exactly what will be missing next time, and this is the only moment
anyone knows what it was.