Malcom Gladwell’s recent New Yorker article compares “the quarterback problem” to the challenge of finding a good teacher. It’s an interesting article, but it is, perhaps, too narrow in its focus.
For those who don’t know, the quarterback problem is defined as the extremely difficult task of selecting a quarterback to play in the NFL from the pool of college football quarterbacks. It’s deceptively challenging to do this because there’s so much data available and so many ways to rank college football quarterbacks. However, the college football game is so different from the NFL game that success at the college level seems to have very little correlation, or perhaps no correlation, to success at the professional level.
One of the best examples of this is the comparison of Peyton Manning and Ryan Leaf, who were both extremely successful college quarterbacks drafted first and second in the 1998 NFL draft. Most experts thought it was a toss-up as to which of these two would have a better career. Of course, it’s clear to everyone now that Peyton Manning is a lock for the Hall of Fame while Ryan Leaf is famous for being a complete bust in the NFL.
The key element of the quarterback problem is that past results simply aren’t useful in predicting future success. Gladwell argues that this is also true of selecting good teachers. He claims that the usual metrics used to measure hiring and promotions for teachers, such as master’s degrees, teaching certifications, and other cognitive standards, are just as useless in attempting to determine a good teacher as college football statistics are in trying to determine a successful NFL quarterback.
Another important element of the quarterback problem is that the difference between ‘good’ and ‘bad’ is extremely large, potentially several orders of magnitude. There are very few ‘good’ quarterbacks in the NFL. There aren’t even very many quarterbacks decent enough to serve as an emergency backup. The quarterback position in the NFL may be the single toughest position to play in all of professional sports. Gladwell argues that the same is true of teachers:
Suppose that Mrs. Brown and Mr. Smith both teach a classroom of third graders who score at the fiftieth percentile on math and reading tests on the first day of school, in September. When the students are retested, in June, Mrs. Brown’s class scores at the seventieth percentile, while Mr. Smith’s students have fallen to the fortieth percentile. That change in the students’ rankings, value-added theory says, is a meaningful indicator of how much more effective Mrs. Brown is as a teacher than Mr. Smith.
It’s only a crude measure, of course. A teacher is not solely responsible for how much is learned in a classroom, and not everything of value that a teacher imparts to his or her students can be captured on a standardized test. Nonetheless, if you follow Brown and Smith for three or four years, their effect on their students’ test scores starts to become predictable: with enough data, it is possible to identify who the very good teachers are and who the very poor teachers are. What’s more—and this is the finding that has galvanized the educational world—the difference between good teachers and poor teachers turns out to be vast.
It follows that if you want a school system filled with good teachers, then you have to be willing to identify the poor teachers and get rid of them. This is the only solution to the quarterback problem. It’s a brutal process for both the teachers and the administration. Time Magazine recently had a cover story on Michelle Rhee’s unusual approach to improving schools in the nation’s capital, which is attempting to implement this brutal process.
Rhee wants to solve the quarterback problem the only way possible: by mitigating its effects. Simply put, if you can’t identify good teachers without seeing how they perform in the classroom, then you have to hire a bunch of teachers, watch their classroom performance, identify those that are succeeding, and reward them. Similarly, you have to identify teachers that are failing and eliminate them. Time’s article does a good job explaining why doing these two things is extraordinarily complicated in the teaching industry.
Of course, this is exactly how they solve the quarterback problem in the NFL. On-field performance is everything. Many of the best quarterbacks were identified as such by their play in real NFL games as backups for injured quarterbacks. Matt Cassel is a great example. In college, he never started a game and served as a backup for Carson Palmer and Matt Leinart. He was drafted into the NFL and played as a backup for Tom Brady, who suffered a season-ending injury in the first game of the 2008 season. Brady’s injury made Matt Cassel a starting quarterback for the first time since high school, which would undoubtedly determine his future in the NFL. If he played well, he would likely be rewarded with a starting role for another team during the off season. If he played poorly, he would fall into the nameless abyss of all the other failed NFL quarterbacks.
After reading Gladwell’s article, I had to wonder, how many other professions are like that? Surely the quarterback problem isn’t just limited to teachers and NFL quarterbacks. The first thing that came to my mind was a Paul Graham essay about great programmers, which is really a must-read for anyone in the software industry. In it, Graham talks about the nature of great programmers, and summarizes the problem of identifying them by saying, “The problem is, if you’re not a hacker, you can’t tell who the good hackers are.” Fred Brooks also talks about the vast difference between a great programmer and an average programmer in The Mytical Man-Month. Here’s Fred Brooks on great software designers:
The differences are not minor – it is rather like Salieri and Mozart. Study after study shows that the very best designers produce structures that are faster, smaller, simpler, cleaner, and produced with less effort. The differences between the great and the average approach an order of magnitude.
Clearly, selecting a software engineers fits the definition of the quarterback problem. It would be very interesting to study how the extreme challenge of creating a start-up company performs as a system for identifying great programmers.
Apparently, some people believe there’s a quarterback problem in selecting good lawyers. Although, I have no particular experience with this, I think the environment in which lawyers at big law firms operate is strikingly similar to the ideal solution to the quarterback problem. There’s an incredibly small percentages of lawyers who end up making partner at a big law firm, which indicates to me that there’s a quarterback problem in trying to hire a big law firm partner.
I’ve also seen the suggestion that selecting a mate is a version of the quarterback problem, but I personally think that’s taking things too far. I don’t think that people really have an objective idea of what a good mate is, let alone what metrics to use in measuring potential mates. Furthermore, the role of being a “mate” really isn’t the same thing as having a job.
I believe the quarterback problem is potentially much more prevalent than people currently recognize. I also think that the solution to the quarterback problem is clearly defined. The two important lessons to learn and apply from the quarterback problem:
- Don’t be afraid to give people a chance. They might surprise you.
- Don’t be afraid to make a change when things aren’t working out.
These two steps are the best known solution to the quarterback problem. What other fields could benefit from implementing them? If you have any suggestions for other areas where this problem seems to occur, please mention them in the comments.