Nate Silver: Don't bench your starters yet

Stat geeks are unanimous in very little. But one thing that they almost all agree on is that most of the time NBA coaches are being excessively timid when benching players with foul trouble. If you want to win, put your best players out there as much as possible, right?

Yesterday, however, I blogged about some new research finding that in fact, coaches were smart, but not for the reasons we thought. It's not that protecting stars from fouls is so important. It's that when they're playing with a lot of fouls, they don't play as effectively.

So now even the data-centric were divided on the topic of benching players.

Nate Silver is a big name in modern analysis. A star of baseball sabermetrics, he went on to start one of the most respected sites anywhere -- FiveThirtyEight -- for assimilating political polling data. It's now part of The New York Times.

Silver's also something of a basketball fan, and he examined this new research praising coaches for their conservatism, and he found it entirely unconvincing. He e-mails:

I'm pretty skeptical about that study you linked to about it being better to bench starters with foul trouble.

They are pretty abbreviated in describing their methodology so I can't be certain about any of this, but it seems like there might be a number of things which would complicate the analysis but which they aren't really accounting for:

The big problem is that a team having a starter in foul trouble is an indicator that the team is playing poorly that evening. Players pick up fouls for a reason, such as because they're having trouble defending their opponent.

While they seem to have some basic measure of opponent quality in their equations, it may not suffice to explain things like matchup difficulties caused by size or tempo discrepancies, that the team is at the back end of a long road trip, is indifferent to the outcome of the game (because it's tanking, etc), that the referee is calling the game tighter than would be optimal given its style of play, and so forth. Also, team personnel can change significantly over the course of the year, so having just one variable to reflect opponent quality over the course of the entire season may be too blunt.

Basically, they seem to find that when a team has a starter in foul trouble, it's less likely than to win than you might expect based on the score differential and a few other simple variables -- in other words, it will play poorly. But since having a starter in foul trouble is an indicator of poor play to begin with, all it may really reveal is that poor play is predictive of further poor play, to an extent that their simple model isn't really able to capture. I may be missing something, but I don't think you can make the inference from their model that subbing out the player is optimal.

Also, they find that the effect is very substantial in the third quarter, but fairly minimal otherwise. I don't see any reason that should be the case, and it seems suggestive to me their result is some kind of statistical artifact stemming from their model design rather than anything "real."