Nate Silver’s 538 Election Forecast has consistently given Obama a higher re-election probability than InTrade does. The 538 forecast is based on estimating vote probabilities from State polls and simulating the Electoral College. InTrade is just a betting market where Obama’s re-election probability is equated with the market price of a security that pays off $1 in the event that Obama wins. How can we decide which is the more accurate forecast? When you log on in the morning and see that InTrade has Obama at 70% and Nate Silver has him at 80%, on what basis can we say that one of them is right and the other is wrong?
At a philosophical level we can say they are both wrong. Either Obama is going to win or Romney is going to win so the only correct forecast would give one of them 100% chance of winning. Slightly less philosophically, is there any interpretation of the concept of “probability” relative to which we can judge these two forecasting methods?
One way is to define probability simply as the odds at which you would be indifferent between betting one way or the other. InTrade is meant to be the ideal forecast according to this interpretation because of course you can actually go and bet there. If you are not there betting right now then we can infer you agree with the odds. One reason among many to be unsatisfied with this conclusion is that there are many other betting sites where the odds are dramatically different.
Then there’s the Frequentist interpretation. Based on all the information we have (especially polls) if this situation were repeated in a series of similar elections, what fraction of those elections would eventually come out in Obama’s favor? Nate Silver is trying to do something like this. But there is never going to be anything close to enough data to be able to test whether his model is getting the right frequency.
Nevertheless, there is a way to assess any forecasting method that doesn’t require you to buy into any particular interpretation of probability. Because however you interpret it, mathematically a probability estimate has to satisfy some basic laws. For a process like an election where information arrives over time about an event to be resolved later, one of these laws is called the Martingale property.
The Martingale property says this. Suppose you checked the forecast in the morning and it said Obama 70%. And then you sit down to check the updated forecast in the evening. Before you check you don’t know exactly how its going to be revised. Sometimes it gets revised upward, sometimes downard. Soometimes by a lot, sometimes just a little. But if the forecast is truly a probability then on average it doesn’t change at all. Statistically we should see that the average forecast in the evening equals the actual forecast in the morning.
We can be pretty confident that Nate Silver’s 538 forecast would fail this test. That’s because of how it works. It looks at polls and estimates vote shares based on that information. It is an entirely backward-looking model. If there are any trends in the polls that are discernible from data these trends will systematically reflect themselves in the daily forecast and that would violate the Martingale property. (There is some trendline adjustment but this is used to adjust older polls to estimate current standing. And there is some forward looking adjustment but this focuses on undecided voters and is based on general trends. The full methodology is described here.)
In order to avoid this problem, Nate Silver would have to do the following. Each day prior to the election his model should forecast what the model is going to say tomorrow, based on all of the available information today (think about that for a moment.) He is surely not doing that.
So 70% is not a probability no matter how you prefer to interpret that word. What does it mean then? Mechanically speaking its the number that comes out of a formula that combines a large body of recent polling data in complicated ways. It is probably monotonic in the sense that when the average poll is more favorable for Obama then a higher number comes out. That makes it a useful summary statistic. It means that if today his number is 70% and yesterday it was 69% you can logically conclude that his polls have gotten better in some aggregate sense.
But to really make the point about the difference between a simple barometer like that and a true probability, imagine taking Nate Silver’s forecast, writing it as a decimal (70% = 0.7) and then squaring it. You still get a “percentage,” but its a completely different number. Still its a perfectly valid barometer: its monotonic. By contrast, for a probability the actual number has meaning beyond the fact that it goes up or down.
What about InTrade? Well, if the market it efficient then it must be a Martingale. If not, then it would be possible to predict the day-to-day drift in the share price and earn arbitrage profits. On the other hand the market is clearly not efficient because the profits from arbitraging the different prices at BetFair and InTrade have been sitting there on the table for weeks.
12 comments
Comments feed for this article
October 2, 2012 at 1:27 am
Jirka Lahvicka
True, there is never going to be enough data to evaluate the estimated probability of Obama being reelected, but we could at least evaluate the accuracy of state-level predictions on which the model is based on – there should be enough data after several election cycles (and this is the second presidential election that Nate Silver tries to predict). It is also interesting that the now-cast seems to exhibit lower serial autocorrelation than the Nov. 6 forecast (did not actually test it, since there is no easy way to get the data in a tabular form), so the longer-term trends could result from including economic data and convention bounce corrections.
October 2, 2012 at 10:49 am
Brittany
technically intrade is a probability under the risk-neutral measure. If you think that the states of the world where romney wins are particularly bad (maybe because he only wins if the economy tanks even more), then those states are worth more and so romney’s share price is bid up above the objective probability (assuming risk-aversion among bidders).
This also implies that prices need not be martingales, but can exhibit drift. People who hold the ‘romney risk’ will earn an expected return to compensate. As the election nears, the risk-neutral probabilities converge to the objective probability.
October 2, 2012 at 4:28 pm
Noto
Just wanted to say that I’m happy to see you back blogging again. You had kind of a lull there for awhile this summer. Welcome back!
October 2, 2012 at 5:02 pm
jeff
Thanks. I always take the summer off. Got lots of material saved up.
October 2, 2012 at 9:49 pm
David Miller
Hi Jeff! I am skeptical of your analysis of Nate’s forecast. The way I think of it, he’s essentially running a regression, adding new data each day but using the same regression model. (The time trends, convention bounce adjustments, and such are also based on historical data, so we can think of them as arising from the regression as well.) Well, we both know that I’m no econometrician, but it seems to me that if his regression model is correct then his series of forecasts should be a martingale. If his model is misspecified (as all models surely are) then there will be some bias, and therefore autocorrelation, but I cannot say a priori how his model is misspecified.
You write “If there are any trends in the polls that are discernible from data these trends will systematically reflect themselves in the daily forecast.” But “trends” are just correlations; they are not predictive out of sample unless you have a model. That is, if you think you can discern trends in the data that systematically affect Nate’s daily forecast, that means you think you have a better model than Nate does. More broadly, I think you should believe that Nate’s forecasts do not follow a martingale only if your prior over models is not centered on Nate’s model.
October 3, 2012 at 2:06 am
En vrac | Rationalité Limitée
[…] Jeff Ely propose un billet très intéressant sur le statut des prédictions concernant le résultat des élections américaines. En […]
October 13, 2012 at 10:16 am
Martingales Don’t Do This « Cheap Talk
[…] Martingales, what? talk cheaply Top PostsKellogg/NU Nobel Economics Predictions 2012 […]
October 13, 2012 at 1:57 pm
alexs25
So, I’m not an econometrician or econophysicist or anything of the like, but does it even make sense to think of a Martingale in a bounded interval. Brownian motion in a reflecting or absorbing box is surely not a Martingale, although free Brownian motion is, and so on. It’s a bit fuzzy how to define “no new information arrives” in this sense. Great take on the problem though!
April 4, 2013 at 4:42 pm
Nichol
Thanks to my father who shared with me on the topic
of this webpage, this web site is actually awesome.
April 7, 2013 at 3:36 pm
WA Niagara DG
I’ve read some just right stuff here. Definitely worth bookmarking for revisiting. I surprise how a lot attempt you set to create any such wonderful informative web site.
April 14, 2013 at 5:24 pm
Luxury Deals
all the time i used to read smaller posts which as well
clear their motive, and that is also happening with this post which I am reading at this place.
May 16, 2013 at 2:33 am
Foods To Avoid with SIBO
Why people still use to read news papers when in this technological world everything is available on
net?