Paul from San Francisco was among those raising a natural question after reading about Football Outsiders' pessimistic projection for the 49ers in 2012.
"I'm not trying to criticize, but I do think that you, as someone reporting on Football Outsiders' stats/projections, have a responsibility to look into how accurate their projections have been over time," Paul wrote.
2011 Football Outsiders Win Projections
Criticism isn't a bad thing, Paul, so no worries there. Football Outsiders' win projection model came close to correctly forecasting the order of finish in the NFC West last season. The projected win totals for each team failed to match up.
Paul passed along a link to an Advanced NFL Stats thread featuring criticisms from Brian Burke regarding Football Outsiders' prediction results from the 2009 season. Football Outsiders' Aaron Schatz also offered thoughts on that 2009 season.
I'm generally sympathetic to those making season-long predictions before training camps begin. They cannot account for key variables such as injuries. Some would say that makes predictions useless. I think there's value in searching for ways to figure out what might happen, or at least what should happen. It's a given that actual results will vary. How much those results vary matters, of course.
2011 Football Outsiders Win Projections
The first chart shows what Football Outsiders called its "mean win projection" for each NFC West team heading into the 2011 season.
A disclaimer says these are published "in response to reader requests" even though "we do not expect any teams to win the exact number of games in their mean projection -- particularly since no team can win 0.8 of a game."
Additional breakdowns offer the percent chance each team might win three or fewer games, four to six games, seven to eight games, nine to 10 games and 11-plus games.
The second chart compares projections against results for 2011, ranked by how much each team exceeded Football Outsiders' projections. Nine teams finished within one victory of the projections. That included three teams from the NFC East. Twenty-one teams finished within 2.6 victories of the projections.
Thirteen teams finished within one game of 8-8 last season. Predicting an 8-8 record for every team would have produced more outcomes within one game of projections than Football Outsiders managed. The fun, of course, is in figuring out which teams appear most likely to diverge from conventional wisdom.
The team with the largest disparity, Indianapolis, played the season without Peyton Manning. Football Outsiders predicted a decline for the Colts even with Manning still in the picture, saying "the run is over" and Indianapolis was now only an "average team, fighting decline and hoping for new stars to support the still-great Manning."
These comments on the Colts illustrate the futility of grading predictions in some cases. For while the chart says Football Outsiders was least accurate when it came to Indianapolis, the analysis holds up well in retrospect. The assumption before the season was that Manning would probably play, and play well.
Meanwhile, Football Outsiders appears to have been correct on the Chicago Bears, nearly nailing their actual win total. But the projection could not assume Chicago would lose Jay Cutler and Matt Forte to late-season injuries. Football Outsiders was arguably more "accurate" on the Colts than on the Bears.
We're free to run through the chart to see where injuries or other unpredictable factors came into play. I haven't gone through the predictions in enough detail to say with any certainty whether Football Outsiders consistently fares well or poorly with its projections. I do value some of the stats and analysis produced in their annual almanac, some of which I hope to pass along in the future.
Schatz, the Football Outsiders' founder, has said he doesn't always agree with his projection model's results. Like most of us, he takes into account subjective factors when making up his mind on a team. For example, he thinks the 49ers will exceed the model's 7.2-win projection for the 2012 season. He thinks San Francisco is more likely win the division with a record around 9-7.
Thanks to Paul for asking a fair question.