Telling an AI it is an expert does not make it more accurate

“You are a world-class senior engineer.” “Act as an expert copywriter.” The habit is everywhere, and it rests on an assumption that sounds obvious: call the model an expert and you get an expert’s answer. One study measured it carefully, and found the assumption does not hold for accuracy.

Read the paper. What follows is a reading of it, not a replacement; the link is above.

What the study measured

Zheng, Pei, Logeswaran, Lee and Jurgens assembled 162 roles across six kinds of relationship and eight domains of expertise — doctor, lawyer and ecologist among them — and put each into the system prompt of nine open-weight models from four families, including Llama 3, Mistral and Qwen 2.5. They then asked 2,410 multiple-choice factual questions, and compared every role against a control with no role at all. The paper was published in the Findings of EMNLP 2024.

What it found

No role was reliably better than no role. None was statistically better than the control. Most made no significant difference, and some made answers worse.

Matching the role to the subject helped only slightly. A lawyer did a little better than a doctor on law questions; the authors describe the effect as relatively small.

The best role could not be chosen in advance. Picking the ideal role for each question after the fact would have raised accuracy noticeably. But methods for predicting that role beforehand did barely better than choosing at random, and for one model family did worse. The authors conclude the effect may simply be unpredictable.

What the vendor actually claims

Model providers do recommend giving the model a role. Anthropic’s prompting guide says a role in the system prompt focuses the model’s “behavior and tone”. It does not claim the model becomes more accurate, and nothing on that page ties roles to correctness.

So the study and the vendor do not conflict. What the evidence contradicts is the popular version of the advice — that an expert label makes the answers more correct.

What follows for how you work

Use a role for voice, register and audience — not for correctness. If you want plain English for a non-specialist, or a formal register for a regulator, a role is a reasonable way to say so. Saying it directly works too.

For correctness, hand over what the expert would know. The label carries no knowledge. The definitions, the constraints, the standard the work is held to, the source material, the edge case that bit you last time — those are what change the answer. An expert’s value was never the job title.

Do not hunt for the magic persona. The study’s finding is that the best role for a question could not be predicted ahead of time. Time spent tuning the persona line is time not spent on the context.

Cut the persona line from a context file. A sentence like “You are an expert in this codebase” is read on every session. By the test in What earns its place it is waste: it tells the model nothing it does not know and, on this evidence, does not change how correctly it works. If you want a behaviour, name the behaviour.

How far this reaches

Open-weight models only. The study did not test the large closed models most people use day to day; the authors excluded them on cost. Those may behave differently, in either direction.

Factual multiple-choice questions only. Writing, code, advice and anything open-ended — where framing and tone matter most — were not measured. The finding is about accuracy on questions with a right answer.

The models tested date from 2022 to 2024. Treat the specifics as a description of those models, not of whatever you are using now.

What does not depend on the models is the reasoning underneath. A label is a word, not information: it cannot supply facts the model does not have, and the facts you hand over can. That holds for any model, and it is the same argument this library makes about context files generally.

The sources were retrieved on the date shown in this document’s header.