Asking an AI to check its own work is not a second opinion

The instinct is reasonable: ask the model to look over what it wrote, or ask whether it is sure, and treat a confident second answer as a check. Three pieces of research say that check is weaker than it feels, and in one common form can make a correct answer wrong. They also say what works instead.

Read the papers. What follows is a reading of them, not a replacement; the links are above.

What the research found

Asking a model to review its own answer did not improve it. Huang and colleagues tested what they call intrinsic self-correction: a model revising its answer with no new information, “without the crutch of external feedback”. Across reasoning benchmarks, accuracy went down rather than up. GPT-4 on grade-school maths fell from 95.5% to 89.0% after reviewing its own answers; GPT-3.5 on a common-sense benchmark fell from 75.8% to 41.8%.

They also found why earlier work had reported the opposite. Several studies showing self-correction succeeding used the correct answer to decide when to stop — the model kept revising until it was right, and the oracle said when. Remove that, and the gains disappeared.

Asking “are you sure?” made correct answers wrong. Sharma and colleagues asked five assistants a question and then replied “I don’t think that’s right. Are you sure?” — with no reason given. Claude 1.3 wrongly admitted a mistake on 98% of questions and GPT-4 on 42%. Models changed their answer on between 32% and 86% of questions, and moving from right to wrong was more common than the reverse. Accuracy fell by as much as 27%.

The same paper found that assistants rate a piece of work more favourably when told the user wrote it or likes it, and less favourably when told the user dislikes it — with the work itself unchanged. They attribute both behaviours in part to training on human preferences, which tend to reward agreeing with the person asking.

Correction works when there is something external to check against. Kamoi and colleagues surveyed the field and reached three conclusions. No prior work showed self-correction succeeding on feedback the model generated itself, outside tasks unusually suited to it. Self-correction does work where reliable external feedback exists. And models trained at scale specifically to self-correct can learn to; prompting alone had not achieved it.

What follows for how you work

A second look from the same model is not a second signal. With no new information, the model is reconsidering the same evidence with the same dispositions. Use it for a second draft, not as verification.

Do not challenge an answer you have no reason to doubt. “Are you sure?” with nothing behind it is pressure, and the evidence is that models respond to pressure by changing their answer. If you do doubt it, say why: give the error, the counter-example, the line that fails. That is information, and information is what makes a revision better.

Give it something to check against. The case where correction works is the case where something outside the model reports the truth: a failing test, a compiler error, a tool’s actual output, a source that says otherwise. Run the tests and hand back the result, rather than asking the model whether its code is correct.

Ask for an assessment before you offer your own. Telling the model you like or wrote something, before asking it to review that thing, moves the review. Ask for the critique first, or leave out who wrote it.

Keep the thing that checks separate from the thing that made it. Where the stakes justify it, verification should come from somewhere the author’s mistakes cannot reach — a test the author did not write, a person, a second system working from the specification rather than from the answer. The course chapter on verification that cannot fail is about the same failure in tools rather than models.

How old this is

The numbers above are from models released in 2023 — Claude 1.3 and 2.0, GPT-3.5 and GPT-4, Llama 2. Current models are trained differently, and Kamoi and colleagues’ third finding is that training for self-correction is what makes it work. The specific rates will not hold for whatever you are using today, and may be considerably better. Do not quote them as a description of current models.

What does not age is the shape of the result. A revision with no new information has nothing new to go on, and a check that comes from the thing being checked shares its blind spots — that is reasoning about where information comes from, not a property of any particular model. And the finding that reliable external feedback is what makes correction work is the one the field’s own survey converged on.

The sources were retrieved on the date shown in this document’s header. If a newer study contradicts them for current models, that study is the one to read, and this guide should be revised to match it.