Everyone is jumping on the bandwagon, including Tyler Cowen, Greg Mankiw, and even Sandeep. They are all trumpeting this study whose bottom line is that student evaluations of teachers are inversely related to the teacher’s long-run added value. The conclusion is based on two findings. First, if my students do unusually well in my class they are likely to do badly in their followup classes. Second, if my students evaluate me highly it is likely that they did unusually well in my class.
I am not jumping on the bandwagon. I have read through the paper and while I certainly may have overlooked something (and please correct me if I have) I don’t see any way the authors have ruled out the following equally plausible explanation for the statistical findings. First, students are targeting a GPA. If I am an outstanding teacher and they do unusually well in my class they don’t need to spend as much effort in their next class as those who had lousy teachers, did poorly this time around, and have some catching up to do next time. Second, students recognize when they are being taught by an outstanding teacher and they give him good evaluations.
The authors of the cited study are every time quick to jump to the following conclusion: older, experienced teachers, and especially those with PhD’s know how to teach “lasting knowledge” whereas younger teachers “teach to the test.” That’s a hypothesis that sounds just right to all of us older, experienced teachers with PhD’s. But is it any more plausible than older experienced teachers with tenure don’t care about teaching and as a result their students do poorly? Not to me.
Dear 310-2 students who will be filling out evaluations this week: please don’t hold it against me that I am old, experienced, and have a PhD.
Cheap Talk 
11 comments
Comments feed for this article
June 13, 2010 at 9:42 am
Ed Dolan
I, like Jeff, am a supporter of student evaluations of teachers. At one point I spent ten years as a teacher and administrator in a graduate business program that did not have tenure, and relied heavily on student evaluations for faculty retention decisions. I reject the hypothesis that students give high evaluations to instructors who dumb down their courses, teach to the test, grade high, and joke a lot in class. On the contrary, they resent such teachers because they are not getting their money’s worth. I observed a positive correlation between overall evaluation scores and a key evaluation-form item that indicated that the course required more work than average. Informal conversations with students known to be serious tended to confirm the formal evaluation scores.
I have less experience at tenure-based institutions, but my tendency is to think most of the bad teachers are either burnt-out tenured profs who no longer bother either with class prep or research, or second-rate newbies who are incapable of coping simultaneously with class prep and publish-or-perish pressures.
June 13, 2010 at 1:42 pm
Anonymous
Most evaluations require students to write comments about the instructor and also about course, after having completed the initial set of questions which require the student to assign a number on a scale of 1 – 10 (or 1 – 5, 1 being the worst and 5 being excellent etc) or assign a letter grade (A – E). Note we get the number response together with the comments on the same page, so it is the same student we are talking about. This is what I observed when I looked at the number responses of a student and his/her comments:
(1) There were students who would assign “unsatisfactory” for every question and would also not write anything under the comments sections; either about the course or the instructor. Now one would imagine that if a student was extremely disappointed with the prof, he/she would write at least something negative about the teacher. Also of some relevance is the fact that some of these students would also check “unsatisfactory” for the question that they were regular in the class.
(2) There were students who assigned “unsatisfactory” to some questions like: the instructor was a good teacher/this was an interesting course etc etc, but also gave very high points to questions like: the instructor was well prepared/organized etc etc. Almost always these were the students who would write lengthy comments about improving the course as well as comment on the instructor. I have found their criticisms to be spot on and wherever possible I have tried to incorporate those suggestions.
(3) There were some (a minority) who gave very high points to almost all questions and few of them also wrote in the comments sections.
(4) My scores were generally lower for large classes (over 130 students) and where these classes had students from different majors in comparison to say a pure econ slass which generally had juniors and seniors and where the capacity was about 50. Point (1) was typically prevalent for the large classes.
No matter what, students do care about what they are learning in the class, whether the instructor is working hard with them in the course (in addition to their grade). Claiming that students give high points to teachers who dumb down the material/teach to the test is just off mark.
June 13, 2010 at 2:49 pm
Michael Bishop
I would be hesitant to eliminate those end-of-class surveys, but I do think this paper contributes to other evidence that there may be enough problems with them to justify additional/alternative forms of teaching evaluation.
I didn’t get into all my thoughts in my blogging of the paper, but perhaps I emphasized the alternative interpretations a bit more than others have.
http://permut.wordpress.com/2010/06/11/professor-quality-and-professor-evaluation/
June 14, 2010 at 8:30 am
Brian Moore
“Second, students recognize when they are being taught by an outstanding teacher and they give him good evaluations.”
Do we know this for sure? Perhaps they know when they have an outstanding teacher, but by definition, those are relatively few.
“First, students are targeting a GPA. ”
If that’s so, then it seems that in the cases where they don’t have outstanding teachers (most of the time) they will be uninterested in quality, and more interested in what grade they get.
From Anonymous in the comments:
“Claiming that students give high points to teachers who dumb down the material/teach to the test is just off mark.”
If that’s so, you have to assume a world where a least a dominant majority of students who get D’s and F’s stoically accept that the reason for that is they didn’t study hard enough, as opposed to believing it was the fault of the teacher; who was either dumb/confusing/too hard/too mean. Does this sound like an accurate portrayal of the average 18-22 year old college student to you? I wasn’t in college that much long ago, and I can guarantee you that it wasn’t the case then.
June 14, 2010 at 11:50 am
Professor Quality and Professor Evaluation « Permutations
[…] Added: Jeff Ely has an interesting take: In Defense of Teacher Evaluations. […]
June 14, 2010 at 1:40 pm
sandeep
Jeff: You know I love the intra-blog debates.
To you first point: The authors study correlation between current student evaluations and future performance not correlation between current grade and future performance.
If student evaluations reflect the grade for the course, this difference does not matter. But if student evaluations are done before the grade is given, they will reflect happiness with teacher more than happiness with grade. So, your theory would have to be more (too?) complicated to fly.
June 14, 2010 at 4:45 pm
Sam
I’ve taught seniors(majors and non-majors) at a business school for the last five years and I have the following observations.
1. With time, I’ve found an increasing proportion of my class to be very professional, very hard working, very smart and very discerning. It could be that the university is on the up and up and is attracting better students or because of the recession the students have become more focussed about everything or maybe something else.
2. All students care a lot about their GPA.
3. A majority of students (even non-majors) care a lot about the quality and relevance of their coursework and the enthusiasm of the instructor for the course, the effort the instructor put in the course and the sincerity of the Instructor in demanding work from them. They’re spending a lot of their time and money into their education and even if they discover that they don’t like the course topic too much, they do want to be pushed and they do want a good product for their investment.
4. While there will always be a few students who would not take responsibility for their grade, I’ve found that most of my students recognize and acknowledge when the course is fair, demanding, relevant and accept their Bs or Cs when it was the result of a well fought fair competition.
June 14, 2010 at 6:11 pm
Newly Minted MBA
As a recent student I will tell you that I rated my professors very aggressively on a number of factors the most important of which was if they inspired me to like the subject and learn more. I think professors too much teach material and forget to inspire what makes their subject interesting, valuable or great. Make it relevant and students will appreciate it. I personally never cared for grades(my GPA is prime proof of that) because I didn’t think a grade says much other than I can do well on an exam. However the professors that showed commitment to the class through their preparation, energy, excitement and student support got 10’s from me. What is taught in a classroom needs to be a foundation to build on and the building will only occur if the prof leaves a mark.
June 15, 2010 at 7:29 am
twicker
Quick notes:
1) Note that this only examined one course (Calc I). Most of the discussion I’ve seen has expected that this phenomenon will hold for all courses in all disciplines; however, I suspect that deeply learning English Composition I would, in fact, be extremely helpful in English Composition II.
2) Note also that these are freshmen we’re dealing with, and ones who have specifically chosen a regimented, military path. For this group of students, they likely have very few benchmarks for evaluating their professors’ performance; thus, it makes absolute sense that they would evaluate those professors based on how well they were prepared for the tests/quizzes/etc. Thus, the main problem seems to be that the tests/quizzes/etc. aren’t properly constructed at the USAFA, not that there’s anything inherently wrong with evaluating professors.
3) The results might actually indicate that the highest-rated new professors may, in fact, be the best long-term professors, since the students likely rate those professors based on their adherence to the syllabus and the newbie profs that most diligently prepare for that first class may be the ones most likely to diligently prepare for later classes.
Say you’re the new instructor who comes in to teach X 101 (here: Calc I). It’s the first time you’ve taught at this university, it’s your first class, and you have a set syllabus. So, you teach to the syllabus — *really* well. Later, you finish up that dissertation that was hanging out there, get your PhD, hop on the tenure track, and also end up teaching X 102 — meaning you now know how X 101 maps to X 102. Meanwhile, you also talk to the folks teaching Y 241 (here: Aerospace Engineering I), and begin to see how X 101 connects to Y 241. You’re also getting visits from former students and discussing how things are going in their classes – giving you even more insight into how X 101 maps to X 102, Y 241, Z 252 (say, Mechanics of Deformable Bodies), etc.
Now — what characteristics predict effective educator ability? One of the main ones is the ability to listen to your students and discern where they’re having problems and how to connect what they’re learning *now* with what they need to know *later.* I suspect that the instructors who initially do a better job of listening at first and connecting their students with what those students need to know — those same instructors (now professors w/PhDs) will be more likely to keep that skill than any instructor that doesn’t do this initially, and, thus, will be more likely to adapt her/his X 101 class to ensure better learning later in X 102, Y 241, etc.
So – again, it seems that the problem isn’t with teacher evaluations; it’s with the tests and quizzes that an incoming freshman would, quite naturally, use as the method for evaluating whether or not Instructor A actually performed well. Fix the methods for evaluating the students, then the students have better information for evaluating the instructors (and instructors have better information for constructing the classes). But don’t get rid of teacher evaluations.
June 26, 2010 at 1:15 am
ipoeng
tank,for your info
February 7, 2011 at 7:33 pm
Using student evaluations to measure teaching effectiveness « Real Learning Matters
[…] Ely, Northwestern professor of economics, objects to the authors’ interpretation of their results: I don’t see any way the authors have ruled […]