14 May 2026 ยท 8 minute read
Reading a brief without fooling yourself
A confident model is a convincing way to be wrong, and the defences against that are all in the parts nobody reads.
The most dangerous output I have ever produced was a beautifully formatted analysis that was wrong in the second decimal place of an assumption nobody checked. It was read by eleven people. Every one of them found it persuasive, and the reason they found it persuasive is that it was well made.
Production quality and truth are unrelated, and the human eye cannot distinguish them. A chart with a proper axis and a band around it reads as more reliable than a number scrawled on a whiteboard, regardless of which one is closer to being right.
So the question of how to read a model output is the question of how to stop being impressed by it long enough to find where it breaks.
Read it backwards
Any decent brief ends with what it is standing on: the assumptions, their values, their ranges, and where each came from. That section exists precisely so it can be checked, and it is at the bottom because that is where reference material goes.
Which means everybody reads the headline first, forms a view, and then reads the assumptions with that view already in place. At which point the assumptions are no longer being examined, they are being cleared.
The fix is mechanical. Read the last section first, before you know what the answer is. Count how many of the load bearing numbers came from your own record and how many are defaults. If most of them are defaults, you are about to read a well formatted industry prior with your revenue number attached, and it should be treated accordingly.
The useful part
What a reader thinks, and what it should be
| The instinct | The better move |
|---|---|
| The median is 1.4 million | The median is 1.4 million and one run in ten is below zero. Which of those two facts changes what we do |
| The band is wide, so the model is not much good | The band is wide because the inputs are uncertain. Which input, and can we go and find out |
| It says a competitor reacts in month four | It says that in most of the runs. What would we see in month three if it were about to happen |
| This confirms what I thought | Then it has told me nothing. What would this model have had to produce to change my mind |
| This contradicts what I thought, so the model is wrong | Possibly. Which specific assumption is wrong, and what is the right value |
| The chart looks convincing | The chart is drawn from the same numbers as the table. Read the table |
The fourth row is the one worth writing on a wall. A model that agrees with you has either confirmed something or wasted an afternoon, and you cannot tell which without having said in advance what would have changed your mind.
Three checks that take ten minutes
Check three quotes
Any number read out of a document should carry the line it came from. Read three of them at random. Extraction gets things wrong in specific, boring ways: a total row read as an account, a percentage read as a ratio, a date read as a quantity. Ten minutes of reading quotes catches more errors than an hour of debating the median.
Move the biggest assumption to the ends of its range
Take whichever number the analysis is leaning on hardest, set it to the bottom of its plausible range, re-run, then set it to the top and re-run. If the answer keeps its sign across that whole span, the decision is robust and you can stop worrying about the exact value. If it flips, you have just discovered that the entire analysis is a restatement of one person's opinion about one parameter.
Change the seed
If the output is produced by sampling, it has a seed, and the same seed gives the same answer. Change it and run again. The middle should shift slightly and the story should not. If the narrative changes character between seeds, there are not enough replications behind it to support the confidence of the prose.
Before you start
Write down what would change your mind
This is the only defence against motivated reading that works, and it has to happen before the output exists.
One paragraph, dated, before the analysis is run. What you currently expect, what result would make you abandon the plan, and what result would make you accelerate it. Then run the thing and see which of the three you got.
It feels like bureaucracy. It takes four minutes. And it converts a model from a device for producing supporting material into a device that can tell you something, because you have committed in advance to a version of the world where it is allowed to.
- What I expect, in one sentence, with a number
- The result that would make me stop
- The result that would make me go faster
- The date, so the note cannot be quietly revised afterwards
The brief is an argument with its premises listed
That is the most it can be, and it is a lot. It is not evidence, it is not a forecast, and it is not a decision. It is a structured claim about what follows from a set of stated assumptions, with the assumptions written down where you can attack them.
The value is in that last part. A consultant's recommendation has premises too, and so does the view of the person in the room with the most conviction, but in both of those cases the premises are inside somebody's head and the only way to attack them is to attack the person. Writing them down turns a fight about credibility into an argument about a number, and arguments about numbers can be settled.
Which leads to the only test of a model output that means anything. Not whether it was accurate, because you will not know that for a year and by then the question will have changed. The test is whether it changed what somebody did, and whether the reason it changed their mind was a specific assumption they could name.
Run the question instead of arguing about it
Build a twin from whatever record you have, and see the range rather than the one number somebody liked.