Nate Silver’s 538 Election Forecast has consistently given Obama a higher re-election probability than InTrade does. The 538 forecast is based on estimating vote probabilities from State polls and simulating the Electoral College. InTrade is just a betting market where Obama’s re-election probability is equated with the market price of a security that pays off $1 in the event that Obama wins. How can we decide which is the more accurate forecast? When you log on in the morning and see that InTrade has Obama at 70% and Nate Silver has him at 80%, on what basis can we say that one of them is right and the other is wrong?
At a philosophical level we can say they are both wrong. Either Obama is going to win or Romney is going to win so the only correct forecast would give one of them 100% chance of winning. Slightly less philosophically, is there any interpretation of the concept of “probability” relative to which we can judge these two forecasting methods?
One way is to define probability simply as the odds at which you would be indifferent between betting one way or the other. InTrade is meant to be the ideal forecast according to this interpretation because of course you can actually go and bet there. If you are not there betting right now then we can infer you agree with the odds. One reason among many to be unsatisfied with this conclusion is that there are many other betting sites where the odds are dramatically different.
Then there’s the Frequentist interpretation. Based on all the information we have (especially polls) if this situation were repeated in a series of similar elections, what fraction of those elections would eventually come out in Obama’s favor? Nate Silver is trying to do something like this. But there is never going to be anything close to enough data to be able to test whether his model is getting the right frequency.
Nevertheless, there is a way to assess any forecasting method that doesn’t require you to buy into any particular interpretation of probability. Because however you interpret it, mathematically a probability estimate has to satisfy some basic laws. For a process like an election where information arrives over time about an event to be resolved later, one of these laws is called the Martingale property.
The Martingale property says this. Suppose you checked the forecast in the morning and it said Obama 70%. And then you sit down to check the updated forecast in the evening. Before you check you don’t know exactly how its going to be revised. Sometimes it gets revised upward, sometimes downard. Soometimes by a lot, sometimes just a little. But if the forecast is truly a probability then on average it doesn’t change at all. Statistically we should see that the average forecast in the evening equals the actual forecast in the morning.
We can be pretty confident that Nate Silver’s 538 forecast would fail this test. That’s because of how it works. It looks at polls and estimates vote shares based on that information. It is an entirely backward-looking model. If there are any trends in the polls that are discernible from data these trends will systematically reflect themselves in the daily forecast and that would violate the Martingale property. (There is some trendline adjustment but this is used to adjust older polls to estimate current standing. And there is some forward looking adjustment but this focuses on undecided voters and is based on general trends. The full methodology is described here.)
In order to avoid this problem, Nate Silver would have to do the following. Each day prior to the election his model should forecast what the model is going to say tomorrow, based on all of the available information today (think about that for a moment.) He is surely not doing that.
So 70% is not a probability no matter how you prefer to interpret that word. What does it mean then? Mechanically speaking its the number that comes out of a formula that combines a large body of recent polling data in complicated ways. It is probably monotonic in the sense that when the average poll is more favorable for Obama then a higher number comes out. That makes it a useful summary statistic. It means that if today his number is 70% and yesterday it was 69% you can logically conclude that his polls have gotten better in some aggregate sense.
But to really make the point about the difference between a simple barometer like that and a true probability, imagine taking Nate Silver’s forecast, writing it as a decimal (70% = 0.7) and then squaring it. You still get a “percentage,” but its a completely different number. Still its a perfectly valid barometer: its monotonic. By contrast, for a probability the actual number has meaning beyond the fact that it goes up or down.
What about InTrade? Well, if the market it efficient then it must be a Martingale. If not, then it would be possible to predict the day-to-day drift in the share price and earn arbitrage profits. On the other hand the market is clearly not efficient because the profits from arbitraging the different prices at BetFair and InTrade have been sitting there on the table for weeks.