2 June 2026 ยท 6 minute read
Asking a model versus running one
Both are useful. They are useful for different things, and the confusion between them costs more than either of them saves.
I use a language model most days. It reads contracts faster than I do, it drafts better than my first draft, and when I describe a problem badly it often works out what I meant. That is not a small thing and this piece is not an attack on it.
What it will not do is tell me what happens to my cash in month eight, and the reason is not that it is insufficiently clever. It is that answering that question requires carrying state forward through time, and answering a question in prose does not.
Two different operations
Asking is retrieval and composition. You describe a situation, and what comes back is the most coherent account of situations like the one you described. If your situation is typical, that account will be good, and it will contain things you had not considered, which is valuable.
Running is arithmetic with memory. Month one produces a state. Month two starts from that state and produces another. A customer who left in month four is not there in month five. Capacity used in month six is unavailable in month six. Nothing is composed, everything is computed, and the answer at the end is a consequence of every step before it.
The outputs look similar because both arrive as text with numbers in them. The things underneath them are not related.
How far each one gets
The test
Ask the same question twice
This is the cheapest diagnostic available and almost nobody runs it. Take a decision you are working on, ask a model, write the answer down. Open a new conversation, ask again, write that down. Then phrase it as though you have already decided and ask a third time.
The three answers will differ, and the third will be noticeably more supportive than the first. That is not a defect being exposed. It is the tool doing what it does, which is producing the most plausible continuation of the conversation you started, including the parts of the conversation where you revealed what you wanted.
A simulation fails this test in the opposite direction. Ask it the same question twice with the same seed and you get the same answer to the last decimal place, and phrasing the question with confidence does not move it at all, because your phrasing never reaches the arithmetic.
- Same question, new conversation: a different answer
- Same question, leading phrasing: a more agreeable answer
- Same question, same seed, a simulation: identical output, every time
Where asking wins outright
Reading messy prose. A supply agreement written by a lawyer says the term shall continue for a period of twenty four months from the commencement date subject to clause 9.2, and any rule you write to extract that will find the wrong number often enough to be dangerous. A model reads it correctly, and it reads the exception in clause 9.2 as well.
Finding what you missed. Describe your decision and ask what a thoughtful sceptic would raise. The answer is a checklist, generated from every similar situation that has ever been written about, and it will contain two things you had not thought of. Neither of them will be quantified, and that is fine, because quantifying is not what you asked for.
Writing the thing up. A model turns a set of findings into prose faster than any person, and the prose is fine.
All three of those are jobs where the absence of state is not a problem, because the answer does not depend on what happened in month seven.
The combination that works
Ask first, to find the questions. Run second, to answer the ones that are quantitative. Ask again at the end, to write it up and to check what the model missed.
What goes wrong is using one where the other belongs. A simulation asked to tell you what you have overlooked will tell you nothing, because it only knows what is in your files. A language model asked for a number will give you one, fluently, and the number will have come from nowhere in particular.
The failure mode that costs real money is the second one, and it is hard to spot precisely because the output is good. A well written paragraph containing a figure looks exactly like analysis. The only way to tell the difference is to ask where the figure came from, and to keep asking until you either reach a line in a file or reach the end of the answer.
Run the question instead of arguing about it
Build a twin from whatever record you have, and see the range rather than the one number somebody liked.