From The Chronicle of Higher Education
If you’re a psychologist, the news has to make you a little nervous—particularly if you’re a psychologist who published an article in 2008 in any of these three journals:Psychological Science, the Journal of Personality and Social Psychology,or the Journal of Experimental Psychology: Learning, Memory, and Cognition.
Because, if you did, someone is going to check your work. A group of researchers have already begun what they’ve dubbed the Reproducibility Project, which aims to replicate every study from those three journals for that one year. The project is part of Open Science Framework, a group interested in scientific values, and its stated mission is to “estimate the reproducibility of a sample of studies from the scientific literature.” This is a more polite way of saying “We want to see how much of what gets published turns out to be bunk.”
We should do this in economics. But there is a less confrontational way to do it. Top departments in experimental economics attract PhD students who want hands on experience in the lab. These are departments like NYU and CalTech. They would benefit the profession, their students, and the reputation of their PhD programs, i.e. everybody concerned, if they were to add as a requirement that every student receiving a PhD must pick one recently published experimental article and attempt to replicate it.
Thanks to Josh Gans for the pointer.
4 comments
Comments feed for this article
April 26, 2012 at 11:26 pm
devo
That would be hundreds of experiments to replicate.
They themselves in the first paragraph on their website say: “The investigation is currently sampling from the 2008 issues of three prominent psychology journals “.
Now, I admit this could mean they are currently sampling, but hope to do all eventually. Do we have some less ambiguous statement that they really intend to replicate every single paper?
April 26, 2012 at 11:31 pm
twicker
We do, indeed – in their other documents. They’re starting with (IIRC) the first 30 in each journal, and are then planning to add more until they replicate all (assuming they ever actually get to that point).
So – it’s their intention to replicate all, but it’s not necessarily going to happen.
April 27, 2012 at 9:46 am
Replication rite of passage « Aid Thoughts
[…] at Cheap Talk, Jeff Ely discusses the Reproducibility project, an attempt by a group of psychologists to […]
April 29, 2012 at 1:16 pm
Federico
One problem is that economic journals typically don’t publish replications. Journals in other fields do. Even if it’s only to get “hands on experience,” econ students are better off working on original research than on replicating existing experiments.