My latest column for the Huffington Post discusses the research on quarterbacks and the NFL draft I recently published with Rob Simmons. This research has generated quite a bit of interest. Malcolm Gladwell discussed this in a New Yorker article in December of 2008. This article then became part of Gladwell’s What the Dog Saw, and Steven Pinker – in his review of this book – singled out the story of quarterbacks and the NFL draft as one particular tale he didn’t like.
Pinker’s reaction led to quite a bit of discussion at Gladwell’s blog. And it led to two posts in this forum.
Brian Burke – of the Advanced NFL Stats – also participated in this discussion last year. And last month he offered two more posts on the topic.
Brian asked if I could comment on these latest stories. And after a delay of a few weeks, I finally found the time to write down some thoughts.
Where We Agree
Let me begin by noting two points where I think Brian and I agree. First, Brian and I both believe one needs to look at per-play performance. In other words, we know that quarterbacks taken earlier in the draft will see more time on the field. So if you look at aggregate performance metrics, quarterbacks taken earlier will tend to look better. But if these quarterbacks are truly offering more – and not just playing more – then we should see evidence in the per-play numbers.
We also appear to agree that the correlation between draft position and performance shouldn’t be very large. A quarterback’s performance – as Brian notes – is dependent upon more than just the quarterback’s ability. The performance of teammates and coaches also plays a significant role. All of these interaction effects would be difficult for scouts to forecast. So even if the scouts were focusing on the correct factors in evaluating a quarterback’s ability, there would still be a problem.
Given this point, one wonders what this discussion is about. Brian and I agree on the fundamental issue: Draft position doesn’t do a great job of predicting future performance. Although it is true we are in more agreement than some might suspect, there are some issues with Brian’s latest approach to this issue that I think I should note.
The Story of the Research
Before I get to my reaction to Brian’s latest, though, let me briefly tell the story of the research Rob and I have published. The impression one gets from much of the discussion people have offered is that Rob and I simply looked at the correlation between draft position and performance. Once we saw low correlation coefficients, we wrote our paper.
If one reads the paper, though, one sees there is much more to our tale. The question that motivated our story was “what determines where a quarterback is selected in the NFL draft?” The list of factors we considered included a quarterback’s height (in inches), his body mass index (and BMI squared), his Wonderlic score, time in the 40 yard dash, whether or not he played in the Division I-AA or Football Championship Subdivision (as opposed to the Football Bowl Subdivision or what was known as Division I-A), and various performance metrics. Our list of performance metrics included career plays, completion percentage, interceptions per attempt, yards per attempt, the NFL’s QB rating, and various Wages of Wins metrics (i.e. Wins Produced per play, Net Points per play, and QB Score per play).
What we found is that draft position is a function of height (taller quarterbacks are drafted first), BMI (and BMI squared), the Wonderlic score (quarterbacks with higher scores get drafted first), 40 yard dash times (faster quarterbacks get drafted first), playing FCS football (playing for an FCS team means you get drafted later), career plays (more plays gets you drafted earlier so it is better to stay in school), and interceptions per attempt (more interceptions and you get drafted later). We also found – in different estimations – that the NFL’s QB rating and the Wages of Wins metrics mattered. But completion percentage and yards per attempt did not statistically impact draft position.
Okay, all of this looks good. Quarterbacks are being evaluated with respect to performance on the field as well as factors measured at the NFL combine.
We then, though, looked at how these factors predicted future performance. We considered a variety of performance measures (i.e. Wins Produced per 100 plays, the NFL’s QB rating, completion percentage, interceptions per attempt, and passing yards per attempt). We also looked at performance at different points in a quarterback’s career. In sum, we ran many, many regressions. And here is our basic result (directly quoted from the paper): In all of our formulations, we never found that the combine factors, or the college performance with respect to Wins Produced per 100 plays or QB rating, had a significant impact—of the expected sign—on NFL Wins Produced per play or NFL QB Rating at any level of experience in the NFL.
Now we did find that completion percentage in the NFL was statistically related to college completion percentage (although the explanatory power was somewhat low). Completion percentage in college, though, did not predict where a quarterback was taken. And the factors that did predict where a quarterback was taken did not predict future performance.
Quarterback and Draft Position
In presenting our research we began with the story of draft position and quarterback performance. But the actual time-line of the research is reversed. Once we had evidence that the factors that drive draft position were not related to future performance we wondered “does draft position predict future performance?” It is this research that Brian discusses, but I am not sure I agree with his approach. What follows are some reactions to his two posts.
Sample and Draft Order
Beyond the issues discussed above, our study also looked at the link between quarterback performance and where a quarterback was selected in the draft from 1970 to 2007. Brian has argued that the 1970s should not be included in the data set. He has also argued that where a quarterback is selected isn’t as important as the order in which the quarterbacks were chosen because the former is impacted by team need. The order – Brian argues — should tell us more about how decision-makers ranked the quarterbacks.
Darren Rovell recently asked me to update our analysis for an article he posted at his CNBC blog. Darren’s article – Is it Time to Abolish the NFL Combine? — offers analysis that incorporates both of Brian’s points. Specifically, Darren reports the following two tables:
These two figures get at both of Brian’s suggestions. And as one can see, our story remains the same. When we look at performance from 1980 to 2009 we still see that quarterbacks selected from 11-50 outperform those taken from 1-10. Furthermore, when we look at draft order and performance – as opposed to draft position and performance – we still see very low correlations.
Evaluating Players Who Didn’t Play (or Play Much)
So why are Brian finding somewhat different results? The key is what to do with quarterbacks who didn’t play or didn’t play much. The correlations reported in our article (and in the analysis posted by Darren) restricted the sample of quarterbacks to those who attempted at least 100 passes (what we did in the book) or participated in at least 100 plays (what I did for Darren). In other words, the analysis required that a quarterback play in about four games per season.
The basic argument for this restriction is that the numbers from a quarterback who didn’t play much in his career are probably not reflective of a quarterback’s value. To illustrate how the first games of a quarterback’s are misleading, guess the identity of the quarterback who posted the following numbers in his first 73 pass attempts: 384 passing yards, 31 rushing yards (on five attempts), seven interceptions, one touchdown, and a QB Score of -29.
These numbers indicate this quarterback is very bad. Luckily for this quarterback, he got another chance to play. And eventually, John Elway – yes, this is what Elway did in his first 73 attempts – got much better. For quarterbacks taken later in the draft, though, their career probably ends after these first 73 attempts.
Now is it likely that all quarterbacks who play badly will become Elway? No, and that isn’t the point we are making by restricting our study to quarterbacks who actually spent time on the field. What we are arguing is that a quarterback who really hasn’t played much is not generating numbers that tell us very much. And since we want to know how draft position relates to performance, we need to include performance measures in our analysis that are indicative of a quarterback’s actual ability.
In Brian’s view, though, these players who didn’t play much or never played at all were simply not good enough. Consequently, Brian either a) includes players in his analysis who had less than 100 plays (or pass attempts) and/or b) assigns a relatively low value to players who never played.
The approach taken by Brian increases the correlation between draft position and performance. By assuming that quarterbacks taken later would be poor quarterbacks, or by including the numbers generated by lower drafted quarterbacks who didn’t play much, Brian biased his evaluation of quarterbacks taken later downwards. And consequently, the correlation between draft position and performance is increased (although not by much).
In the end, when you assume those drafted in later rounds are poor players, you suddenly discover a somewhat stronger relationship between draft position and performance. But it’s important to emphasize how this analysis is constructed. Brian has essentially assumed his answer.
And there is reason to think quarterbacks who never played – or played very little – are not evaluated perfectly. Quarterbacks are not like other players on a football team. A back-up at almost every other position gets to play some in each game. Teams, though, want to play their starting quarterback. Consequently, during the week a back-up quarterback doesn’t get many reps in practices. And even if the back-up was getting reps, practice – where defensive players don’t get to keep hitting the quarterback – is not the same as a real game. Therefore, we should not be surprised by stories where a quarterback is traded or cut by one team only to later excel for another franchise (i.e. Kurt Warner). It simply is hard to evaluate how a quarterback who only gets a few practice reps is going to actually perform week after week as a starting quarterback.
The focus on quarterbacks taken later in the draft, though, can obscure the basic story Rob and I are telling. Let me offer another look at this issue that highlights the basic problem decision-makers have in evaluating future signal callers. ESPN.com reports that the NFL drafted 14 quarterbacks in 2010. ESPN.com also lists between 40 to 50 quarterbacks who could have been selected. And there are even more quarterbacks who played football for an FBS or FCS school who are not listed. One suspects that there are more than 14 quarterbacks who played college football last year who had the physical skills to play in the NFL. But because the NFL only needs less than 100 quarterbacks, many of these players do not get the opportunity.
Let’s, though, focus on those that do get an opportunity. In 2009 the NFL selected nine quarterbacks (of the more than 40 ESPN listed as potential draft picks). The first of these nine was Matt Stafford. He received a contract worth $41.7 million in guaranteed money from the Detroit Lions. Four picks later the Jets selected Mark Sanchez. His contract had $28 million in guaranteed money. And then with the 17th pick in the draft, Josh Freeman went to the Tampa Bay Buccaneers. Freeman signed a contract with a guarantee of $10.2 million.
Such a pattern is hardly unique to the 2009 draft. A few positions in the NFL draft are literally worth millions of dollars. But does the NFL know enough to award contracts in this fashion?
Again, we have already seen that many factors that get a quarterback drafted are not related to future performance. But we can also look at the link between draft position and performance. This time, though, let’s completely ignore the late round picks and only focus on quarterbacks taken in the first sixty picks. And let’s consider quarterbacks who averaged at least 50 plays per season (as opposed to 100). Finally, we will also consider order of selection as well as a quarterback’s draft position. Looking at data from 1980 to the present, here are the correlations between draft position and performance:
As one can see, our story is still the same. Where a quarterback is taken in these first 60 picks tells us quite a bit about the quarterback’s pay. But it doesn’t tell us much about his performance in the NFL.
And again, this shouldn’t be that surprising. There appear to be significant issues with respect to how quarterbacks are evaluated on draft day. Many of the factors that drive draft position are not related to future performance. Furthermore – and this is a point often made in this forum (as well as in The Wages of Wins and Stumbling on Wins) – performance of quarterbacks in the NFL is hard to predict even when we are using past NFL performance to make the prediction. Given all these issues, we should not be surprised by the stories of Tim Couch, Akili Smith, David Carr, Joey Harrington, Alex Smith, JaMarcus Russell, etc. Each of these quarterbacks was supposed to be a future stars on draft day. But as fans of the teams that selected these players, the star performances envisioned on draft day never actually appeared.