With the 2016 college football season in the books, it’s a great time to look back and reflect on the past year. Just as coaches break down the prior season, ESPN Analytics does the same to determine the strength of its metrics.
In the interest of transparency, we are going to share how ESPN’s Football Power Index performed this season. From the preseason to the bowl season, below you will find a full breakdown of FPI’s game predictions and season projections.
As a quick reminder, FPI is a forward-looking system designed to measure team strength. Although much of the attention is placed on FPI’s team rankings (1 through 128), its ultimate goal is to accurately predict game and season outcomes.
Preseason projections
FPI begins with a preseason rating that is based on four factors that hold different weights: prior performance, returning starters, recruiting data and coaching tenure. As in every year, there were a number of hits and misses in preseason FPI, but overall it correlated pretty closely with the final AP poll rankings.
USC, Oklahoma State and Penn State were all teams that FPI rated quite a bit higher (at least 10 spots) than the preseason polls did, while Notre Dame, Michigan State and TCU were all teams it rated quite a bit lower (at least eight spots).
Of course, there were a number of misses, including FPI’s top two teams in the preseason: Florida State and LSU. Each team failed to meet expectations. FPI also overrated the SEC as a whole, with Tennessee, Ole Miss and Georgia among the teams FPI overvalued in the preseason.
Though rankings are fun to debate, preseason FPI should ultimately be judged on each team’s projected win total, which accounts for team strength and schedule, and conference projections.
FPI’s preseason projected win totals were within one win of the actual win totals for more than a third of FBS teams, and within two wins for nearly two-thirds of teams. There were a number of teams, however, that FPI over- or underestimated.
UCLA (projected to win 9.0 games), Marshall (projected to win 7.8 games) and Michigan State (projected to win 7.5 games) were all teams that FPI overestimated entering the season. Colorado (projected to win 5.0 games), Alabama (projected to win 9.0 games) and Western Kentucky (projected to win 9.1 games) were teams it underestimated. It’s worth noting that very few systems would project a team to win 12 or 13 games in a season, and almost all of those teams’ projected win totals were in line with the expectations set by Vegas.
Still, for a second straight year FPI struggled with Alabama, which was again tasked with replacing its quarterback and 10 other starters. We will be looking into Alabama and how it continues to baffle the system in the offseason.
Overall, FPI did pretty well with its season win total projections, but it has room for improvement in identifying the favorites in each conference. In six of the 10 FBS conferences, the FPI preseason favorite did not win it all. The MAC, Big 12, C-USA and Sun Belt were the exceptions as FPI favored Western Michigan, Oklahoma, Western Kentucky and Appalachian State in those races. Eventual champions Washington (Pac-12), Clemson (ACC) and San Diego State (Mountain West) also entered the year with the second-best chance to win their conferences.
In-season game projections
A system’s season projections are only as good as its single-game predictions, and FPI had a fairly good year predicting games in a rather unpredictable year. The team FPI favored won 72.1 percent of games this season, according to predictiontracker.com, which was a higher percentage than the Vegas opening or midweek lines favorites. Going back further, the FPI favorite has won 75.1 percent of FBS games since 2005, which is a better percentage than the Vegas favorites (74.5 percent) during that time.
FPI also was one of the top systems on that website against the spread. It “won” 53.4 percent of its picks against the spread and finished sixth out of 64 systems in that category.
As with any projection system, if the FPI favorite loses, it doesn’t mean the system was totally wrong. If a team has a 60 percent chance to win, it is expected to lose 40 percent of the time. Taking it a step further, if a team is favored to win by one point and instead loses by two points, it’s a more accurate prediction than if that team won by 15, despite the different outcome (win versus loss).
A better way to judge a system’s performance is a more complex concept called mean absolute error, which is what FPI was designed to minimize. As the name would suggest, mean absolute error measures the average error -- or the difference between a system’s projection and the actual final score -- in its predictions. FPI finished 10th out of the 64 systems in that category.
Overall, we were happy with how FPI’s game predictions performed this season. As with every year, we will examine how we can make them better in the offseason.
Strength of record
While this article was mainly designed to critique our Football Power Index, we would be remiss if we didn’t mention our other main metric: strength of record (SOR). This metric was released in 2014 as a backward-looking résumé rating designed to quantify the impressiveness of each team’s W-L record, given its schedule.
Unlike FPI, which has a set goal of maximizing its game predictions, it is harder to measure SOR’s successes or failures. Though SOR was not designed to project the CFP Rankings, we have found it has correlated very closely with them the past few years.
For the second straight season, the four teams that made the playoff ranked in the top four in SOR at the time of playoff selection. Dating back to the first College Football Playoff in 2014, 11 of the 12 playoff teams ranked in the top four in SOR at the time of selection. Ohio State was the lone exception in 2014.
Across the six CFP ranking releases this season, only three times was there a team in the committee's top four that wasn't in SOR's top four. Again, SOR was not designed to match the committee’s rankings, but it appears to be a good tool to understand their thought process.
As the 2016 season comes to an end, we cannot help but look ahead to next season, which should feature some strong teams and individual performances. Keep an eye out for 2017’s initial preseason FPI, which should be released shortly after signing day.