Suppose you find out that someone named Rory L. Newbie predicted the financial crisis. Should you conclude that he has some unique expertise in predicting financial crises? Seems obvious right: someone who has no expertise would need tremendous luck to make a correct prediction, so Rory must be an expert.
But you know that millions of people are making predictions all the time, and even if not a single one of them has any expertise, the numbers guarantee that at least one of them is going to get it right, just by sheer luck. So for sure someone like Rory is going to get it right, that doesn’t make it any more likely that he is a true expert.
But this sounds unfair to Rory. Rory made his prediction all on his own and he got it right. All those other people had nothing to do with it. If Rory were the only person on the planet then when he gets it right he is an expert. It seems that just because there are lots of other people on the planet making predictions, Rory is no longer an expert. How could it be that his being an expert is dependent on how many other people there are in the world?
The way to resolve this is to remember that we only came to know about Rory because he made a correct prediction. If Rory hadn’t made a correct prediction but instead Rube did, then we would have been talking about Rube instead of Rory. No matter who it was that made the correct prediction, and for sure there’s somebody out there who did, we would be talking about that person. The name Rory is a trick because in this scenario it is really naming “the person who made a correct prediction.”
But when there’s only Rory, the name refers to that fixed individual. He was very unlikely to make a correct prediction by dumb luck and so we are correct to conclude his prediction was born of expertise.
Fine, but I leave you with one more paradox for you to resolve on your own. Suppose Rory told his prediction to his wife in advance. For Rory’s wife Rory is a fixed person. While there are still many other predictors on the planet, none of them are Rory. They are irrelevant for Rory’s wife deciding whether Rory is an expert. Now Rory’s prediction comes true. Impossible by dumb luck alone so Rory’s wife concludes that he is an expert. But, following our logic from above, nobody else does.
Normally a difference of opinion between two people is logically consistent provided they were led to their opinions by different information. But Rory’s wife and the rest of the world have exactly the same information. This particular guy Rory made a prediction and got it right. There is nothing that Rory’s wife knows that the rest of the world doesn’t know. And Rory’s wife is just as aware as the rest of the world that there is a world full of people making predictions. For Rory’s wife that doesn’t matter. Why should it for the rest of the world?
Cheap Talk 
12 comments
Comments feed for this article
November 17, 2010 at 2:00 am
Anonymous
Perhaps the question can resolved if we make the distinction between “expert” and “non-expert” as relative versus absolute terms? For Rory’s wife, whether Rory is an expert is independent of anyone else in the world. Whether he is an expert is an indicator variable of whether his prediction is correct. In some sense this is an absolute judgment.
For the rest of the world, Rory’s credentials are judged relative to the rest of the world. A correct prediction only seems unimpressive conditional on observing many predictions.
Just my two cents
November 17, 2010 at 3:21 am
Daniel
I guess the problem is that Rory’s wife and the rest of the world do not have the same information. They have the same data but their data generating process is different. Rory’s wife knows that she would have learned about Rory’s prediction independently of its success, so she does not have to take that into account when deciding whether Rory is an expert or a lucky guy.
November 17, 2010 at 10:32 pm
jeff
This sounds intuitive, but i can’t believe it’s the answer. because even though they have different data-generating processes, its common knowledge what the data-generating processes are. so if they started with the same prior beliefs, it would be common knowledge that they disagree. that’s not supposed to happen.
November 17, 2010 at 9:33 am
Jonathan Weinstein
If Rory’s wife married him for reasons unrelated to his skill, I believe she is simply wrong. I take a Bayesian view: there is some likelihood of spurious success, and some likelihood of a true expert. Our posterior conditional on success depends on the ratio between these, and not on whether there is one hypothesis or many.
Further comment: the likelihood of spurious success can usually be measured quite objectively; likelihood of true expertise is more difficult to know. The number of successes worldwide can be informative about this frequency. So another part of my answer is that the wife *should* pay attention to the rest of the world.
Example: Suppose there are 10^7 prognosticators, with 10^-6 chance of spurious success. If about 10 succeed, we estimate that true experts are very rare. If 1000 succeed, we strongly believe the vast majority are authentic. And, I claim, their wives should make the same inferences.
Trade-off for wives: The more unique your husband appears to be, the more likely he is just full of it.
November 17, 2010 at 10:59 pm
jeff
Jonathan is champion Cheap Talk commenter.
I think my first comparison turns on some implicit correlation whose role is exposed in the second comparison and in your examples. Indeed, if each prognosticator is an expert with independent probability there is no difference between the world-full-of-prognosticators and the solitary-Rory scenarios. Because your posterior is in both cases conditional on the same information: Rory made a correct prediction. (And any information about the outcome of other predictions is irrelevant because their expertise is independent of Rory’s)
The difference appears if you think there is some correlation or aggregate uncertainty. As an extreme example, suppose that with probability p all prognosticators are experts and with probability (1-p) all prognosticators make random predictions uncorrelated with the truth. Then in the solo-Rory case, when he makes a correct prediction you revise upward the probability that he is an expert. But in the world-full-of-Rory-copies case, with probability 1 (or at least very close to it with a large population) you will be presented with a Rory who made a correct prediction, so that conditioning event does not move your posterior (much.) [The posterior probability that the celelbretized Rory is an expert is equal to the posterior probability that all prognosticators are experts, and we don’t update that probability when we just learn whether there was at least one correct prediction. Here the information about the total number of correct predictions would be very useful if we had it.]
Then when we turn to the wife-of-Rory story, she either thinks that there is independence in which case she must agree with the rest of the world, or she thinks there is correlation in which case, as you say, she is wrong not to be paying attention to how many other people got it right.
As for your advice to my wife, she is fully aware of that already thankyouverymuch.
November 17, 2010 at 11:01 pm
jeff
Does it say that there is a market failure in compensating experts? Because the fewer other experts there are in the world the more valuable you are as an expert, but the less likely the world is going to believe that you are an expert and compensate you accordingly.
November 18, 2010 at 8:26 am
Jonathan Weinstein
Yes, I suppose in a very rational world with few experts, there would be a market failure where you have trouble convincing people you are an expert, like trying to sell a good car in a market full of lemons. In practice I suspect this problem is lessened because people underrate the danger of spurious correlation (of course exacerbating the converse problem.)
November 17, 2010 at 10:18 am
Sachin Tyagi
The difference of opinion is due to the difference in assumption made by the wife – she assumes that correct prediction is “impossible by dumb luck” while the rest of the world (ROTW) clearly thinks it to be possible.
Basically, it boils down to the question – What is the probabilty that a person making a correct prediction is an expert? If both wife and ROTW give same answer to this question then there should be no contradiction in their estimation of Rory’s expertise. However the answer to the question must come from outside of this experiment.
November 17, 2010 at 11:57 am
Kevin Dick
The answer to this is straightforward. Everyone’s posteriors that Rory possesses some forecasting skill should be higher than their priors. However, the increase should be directly proportional to their prior that Rory’s prediction would have reached them in any case. You can see this in a probability tree with an extra variable: the prior that Rory’s forecast comes to your attention in each arm of his prediction success.
Then Rory should attempt another prediction. He’s now a priori equally salient to everyone.
November 18, 2010 at 8:28 pm
mike
Both the world and the wife are wrong.
Let’s define experts as being those people whose predictions are wholly caused by the financial system.
Therefore while it is necessary for an expert’s predictions to correlate with the financial system, a high correlation of a person’s predictions is not enough to determine that the financial system caused those predictions.
Both the wife and the world are making a judgment about causality (Rory’s expertness) based on a correlation. This is a fallacy, and they are both wrong.
November 19, 2010 at 11:24 pm
AS
I must be completely missing something here. Consider this example:
Millions of people go to the beaches in the U.S. . Many of the people who go to the beach, out of nervousness, have a “premonition” that they’re gonna get attacked by sharks and share this information with their spouses. The probability that someone who worried about sharks gets attacked by one is high, compared to a specific person getting attacked by a shark.
If I happened to be the spouse of the nervous nelly who has an unfortunate encounter with a shark, will I start thinking that my significant other is an Expert in shark attacks or can see the future ior an expert economist? No. My posterior beliefs will change but I will not start thinking I’m married to Nostradamus just because s/he has an unfortunate accident. Now if s/he were able to predict this 5-10 times in a row, that will really change my beliefs, much like it will change everyone elses.
December 1, 2010 at 8:27 am
Talleyrand
This discussion is insane. You could find out if he has unique expertise by finding out what his prediction was based on, i.e. his argument for the crisis was and whether it makes sense and whether it fits to the data that you have about the process leading up to the crisis. If this is how economists usually reason, no wonder they generally sucked at predicting the crisis.