Listen to this article
AI may have been used to generate this audio.
Press Space or K to play or pause, and J or L to go back or forward 15 seconds.Sources checked 29 September 2026.
A model gives a correct answer to a difficult question. You become more confident in the model. You may also become more confident that its next answer will be correct. Those two updates need not be the same size.
The distinction is easy to lose when a product launch, a funding announcement or a research demonstration is the only new information in front of you. One event can tell you something about the organisation that produced it without making its next result equally predictable.
Tony Fan, Yucheng Liang and Cameron Peng examined this distinction in research published in Econometrica in July 2026. In their experiments, participants tended to underreact when inferring an underlying state and overreact when forecasting an outcome from the same information. The authors connect the difference to the simplifying rules people use for the two questions. [1]
Two bags make the distinction visible
Consider an illustrative probability problem. Bag A contains eight red balls and two blue balls. Bag B contains two red and eight blue. Someone chooses a bag at random, with equal chances, and draws a red ball. They put it back before the next draw.
Before seeing the ball, the probability that they chose A was 50%. After a red draw, it is 80%: red was four times as likely under A as under B, and the bags started equally likely.
The probability that the next draw is red is 68%. There is an 80% chance we are using A, where red has an 80% chance, and a 20% chance we are using B, where red has a 20% chance. Multiplying within each possibility and adding them gives 68%.
The questions concern different unknowns. The first asks which process we are observing. The second includes both uncertainty about that process and variation in what it produces. Copying the first probability into the second question would discard part of the problem.
A second red ball still leaves variation
Keep the same bag, replace the ball, and observe red again. The odds of A started at one to one. Each red observation multiplies them by four, so two red draws give odds of sixteen to one. The probability of A is now 16/17, about 94.1%. The next red draw has probability (16/17 × 0.8) + (1/17 × 0.2), about 76.5%.
Even if we eventually became certain that the bag was A, a blue ball would remain possible. Knowing the process perfectly would give a next-red forecast of 80%. More evidence about which bag we selected cannot remove the variation inside the bag. That is the part the first probability does not include.
There is a small psychological temptation here: after a run of red, describe red as what this bag does. That description drops the blue balls from the account. The arithmetic keeps them in. It also explains why one subsequent blue draw would not by itself make the earlier forecast irrational.
Order deserves care too. In this particular setup, one red followed by one blue gives the same evidence about the bag as blue followed by red. Their likelihood under either bag is 0.8 × 0.2. Starting from equal priors, the posterior returns to 50%. That property comes from the fixed composition and replacement assumptions; it cannot be carried into an evolving business without checking what changed.
A real company is considerably harder to model than two bags. We do not know the full set of possible businesses, their chances of producing an announcement or the effects of everything that happens between announcements. The example gives a distinction to keep track of, not a way to calculate a share price from a press release.
In a product review, separate the claim about the team from the prediction about the next release. A successful demonstration may increase confidence that the team can solve one class of problem. Predicting a reliable customer deployment also requires assumptions about integration, unusual inputs and the conditions of use.
This is one reason to write predictions before seeing the outcome. “The team is good” is difficult to compare with a later event. “The feature will handle these cases at this level of reliability by this date” gives the review something specific to return to.
Keep a record that can be wrong
A forecast also needs a scoring rule that does not punish every uncertain prediction as a failure. Gneiting and Raftery describe how proper scoring rules reward reporting the probability a forecaster genuinely believes. [3] One simple binary example is the Brier score: square the difference between the forecast probability and the outcome, coded as 1 or 0. Lower is better.
If you forecast an 80% chance of red, the score is 0.04 when red appears and 0.64 when blue appears. The blue outcome is possible under your own forecast; its occurrence does not settle whether you were well calibrated. Over many comparable forecasts, you can examine both the scores and how often events assigned an 80% chance occurred.
That is more demanding than remembering the calls you got right. The event, deadline, probability and resolution rule need to be written before the result. Otherwise “the release will work” can become “the demo worked”, and a failed deployment quietly disappears from the record.
For a team, the most useful record may be a small set of recurring predictions with the same definitions. It can reveal that estimates made after an impressive demo are consistently too confident, or that a particular kind of test deserves more weight. The aim is to learn which evidence helps this decision, rather than collect a flattering history of good judgement.
When someone changes a forecast after new evidence arrives, ask them to identify which assumption changed. Perhaps the model is more capable than expected. Perhaps the test was easier to pass than they thought. Those explanations can support different predictions even when everyone agrees on what happened.
The next time an announcement changes your view, write both sentences: what it suggests about the system, and what you now expect it to do. Keep the reasons for the second sentence attached.
Sources
One email when we publish. Research, product decisions, and what teams report back.
