Last year I started my stint at the NY Times by calling attention to just how bad NFL preseason predictions are. I compared the “advanced” projections for team win totals compiled by a fellow stats website called Football Outsiders to two benchmarks. They had predicted doom and gloom for the Jets last year, and my article was intended to relieve Jets fans of needless worry. As it happened, the Jets made the playoffs and went all the way to the AFC Championship game.
The first benchmark was a mindless 8-win prediction for every team. Let's call this the Constant Median Approximation system, or CoMA for short. This benchmark represents zero knowledge. It’s what you would guess if you had no information at all about any of the NFL teams except that they each play 16 games. Certainly anyone can out-predict a coma patient, right?
The second benchmark is a very simple formula based on a regression from the previous season win totals for each team. The formula is 6 plus a quarter of last year’s wins. For example, a 12-win team from 2009 would be predicted to have 6 + 12/4 = 9 wins in 2010. I call simple methods like this ‘Koko the Monkey,’ named after George Costanza’s unflattering nickname at Kruger Industrial Smoothing.
There are several ways of comparing predictions to actual results. One of the simplest and most practical is called Mean Absolute Error (MAE), which is simply the average of the difference between the predicted and actual numbers. A more common method among statisticians is to use something called Root Mean Squared Error (RMSE). RMSE is the square root of the average of the squares of the differences between predicted and actual values. This method tends to penalize predictions that are very far off and rewards predictions for high accuracy.
So how did the coma patient and Koko the monkey compare to FO's predictions for 2009? Keep in mind, the lower the number, the better the accuracy.
I’m reminded of the classic line from the movie Billy Madison. After the main character gives a nonsensical answer to a quiz-show question, the moderator replies, “What you've just said... is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.”
I realize we're having a little fun at FO's expense, and I know it’s not completely fair to pick on only them. Most predictions from self-declared experts are probably just as inaccurate. But FO sells their predictions, and that makes their products fair game. After all, customers should know what they’re getting for their money. People rarely go back to verify predictions (except Gregg Easterbrook), and those that make the predictions usually only mention the cherry-picked few that were somewhat accurate.