Stan Reiter had a standard gripe about statistics/econometrics. Imagine you there is a cave in front of you and you want to map out its dimensions. There are many ways you could do it. One thing you could do is go inside and look. Another thing you could do is stand outside and throw into the cave a bunch of super bouncy balls and when they bounce out, take careful note of their speed and trajectory in order to infer what walls they must have bounced off of and where. Stan equated econometrics with the latter.
That’s not what I am going to say but it is a funny story and its the first thought that came to my mind as I began to write this post.
But I do have something, probably even more heretical, to say about econometrics. Suppose I have a hypothesis or a model and I collect some data that is relevant. If I am an applied econometrician what I do is run some tests on the data and report the results of the tests. I tell you with my tests how you should interpret the data.
My tests don’t contain any information in them that isn’t in the raw data. My tests are just a super sophisticated way to summarize the data. If I just showed you the tables it would be too much information. So really, my tests do nothing more than save you the work of doing the tests yourself.
But I pick the tests. You might have picked different tests. And even if you like my tests you might disagree with the conclusion I draw from them. I say “because of these tests you should conclude that H is very likely false.” But that’s a conclusion that follows not just from the data, but also from my prior which you may not share.
What if instead of giving you the raw data and instead of giving you my test results I did something like the following. I give you a piece of software which allows you to enter your prior and then it tells you what, based on the data and your prior, your posterior should be? Note that such a function completely summarizes what is in the data. And it avoids the most common knee-jerk criticism of Bayesian statistics, namely that it depends on an arbitrary choice of prior. You tell me what your prior is, I will tell you (what the data says is) your posterior.
Pause and notice that this function is exactly what applied statistics aims to be, and think about why, in practice, it doesn’t seem to be moving in this direction.
First of all, as simple as it sounds, it would be impossible to compute this function in all practical situations. But still, an approach to statistics based on such an objective, and subject to the technical constraints would look very different than what is done in practice.
A big part of the explanation is that statistics is a rhetorical practice. The goal is not just to convey information but rather to change minds. In an imaginary perfect world there is no distinction between these goals. If I have data that proves H is false I can just distribute that data, everyone will analyze it in their own favorite way, everyone will come to the same conclusion, and that will be enough.
But in the real world that is not enough. I want to state in clear, plain language terms “H is false, read all about it” and have that statement be the one that everyone focuses on. I want to shape the debate around that statement. I don’t want nuances to distract attention away from my conclusion. In the real world, with limited attention spans, imperfect reasoning, imperfect common-knowledge, and just plain old laziness, I can’t get that kind of focus unless I push the data into the background and my preferred intepretation into the foreground.
I am not being cynical. All of that is true even if my interpretation is the right one and the most important one. As a practical matter if I want to maximize the impact of the truth I have to filter it.
Still it’s useful to keep this perspective in mind.
Cheap Talk 
9 comments
Comments feed for this article
August 1, 2011 at 1:10 pm
twicker
Interesting – and I agree that this is definitely something to keep
In mind.
I’ll mention that allowing access to raw data is fairly standard practice in the world of Org. Behavior and psych. My understanding is that, in fact, many journals require that authors do this, so that others can, in fact, re-run the analyses (or run new analyses), especially when new and more accurate statistical techniques are developed.
August 1, 2011 at 2:12 pm
Basil C. Puglisi
“Stats are controlled by the statistician. We can make the numbers say anything we want 99% of the time. This happens because it’s all just data till we choose what numbers to interpret and how.” – Professor to remain anonymous
This was the most profound thing I learned in Econometrics while earning my graduate degree at Michigan State. It is a statement that helps us understand how to evaluate what we are told and teaches us how to ask the questions to find the answers we want, which in itself can be biased.
I tried to explain this once to someone who lacked experience in the field. I found this example to be the easiest. Geico tells you that drivers that switched from AllState Insurance saved an average of $500!!! This statistic is true, what they fail to tell you is that it is only talking about those that actually switched. The majority of people who called may in fact have faced a rate increase of up to $500, but because they didn’t “switch” the stat remains true.
Economic indicators show that the recession is coming to an end. Yes, that’s true! Those out of date forms of measure devised from the industrial era are telling us that if this was fifty years ago we would have something to be positive about.
Raw data is just that, raw! The data process is completely biased by the researcher from the start and further biased by those that use it as they interpret along the way. As such, it is just a tool to help understand something, not define it.
August 1, 2011 at 4:27 pm
Peter Klein
The bouncy-ball story reminds me of this passage from Coase (JEMS, 2006):
August 1, 2011 at 5:33 pm
mkt
Though by no means a complete solution, doing sensitivity analysis (and presenting the results) helps. E.g. the author can present the preferred results, but also the results from simply doing OLS, and results when some variables are left out or added to the author’s model. If all these lead to the same conclusions, the author can say “even if you don’t believe my model, my conclusions still hold”. If they don’t lead to the same conclusions, the author can argue why his or her model is superior to the alternatives.
August 3, 2011 at 9:03 pm
KLewis
Thanks for thinking outside the box and presenting standard statistical approaches as perhaps skewing or tweaking desired results due to each one’s “priors’ that they bring, whether it is you-the statician/econometrician,or the person you give a program to-that enters the data. It seems almost impossible to have data presented in a form that will bring about the desired result without having someone ‘touch it up.’ Likewise, it is necessary to make such a ‘preferred interpretation’ to reach a ‘truth’ and ‘change minds’ (as you related in pushing the data in the background, regardless of how) when that is what is expected of you in a position of analysis. A well summarized article.
August 12, 2011 at 8:50 pm
Ulises Gómez
There is an old saw that defines an economist as someone who tries to prove that somethin that works in practice works in theory. http://twitter.com/#!/Ulisesgo1
August 20, 2011 at 9:47 am
Reading in Graduate School | You Study Politics, Right?
[…] Ely put it very well recently: My tests don’t contain any information in them that isn’t in the raw data. My tests […]
May 27, 2013 at 8:32 am
Statistics as Principled Argument | You Study Politics, Right?
[…] Abelson does give some attention to Bayesian methods, but a book developing the idea of statistics as rhetoric from a Bayesian point of view would be more coherent. Perhaps we will see something along these […]
May 28, 2013 at 4:58 pm
Meho B'asic
You might find the Q technology developed by Stephenson adequate enough to resolve the issues created with the application of “R” in social sciences.