The devil is in the randomness

If the Sloan Sports Analytics Conference is a convocation for the high priests of statistical rigor, then a discussion of randomness is like a theological seminar on doubt. Call it whatever you want -- “noise,” “variance,” what have you -- but the presence of what can’t be predicted is serious business indeed for the world’s highest-profile predictors.

And the “True Performance & the Science of Randomness” panel was not short on high-profile predictors. Rockets GM Daryl Morey moderated a panel that included Nate Silver, Jeff Ma (of blackjack nerd caper "Bringing Down the House" fame), newly installed Cleveland Browns president Alec Scheiner, sports consultant and professor Benjamin Alamar and sabermetric blogger Phil Birnbaum. The conversation was uncut philosophy, focusing less on practical applications than the intellectual processes underlying the analytics movement.

TrueHoop at MIT Sloan Sports Analytics Conference

We joke about the Sloan Conference being a cult of nerds, a pious gathering of the risk-averse and socially maladroit. Here the faithful gather to suck the magic out of sports, to turn the mysteries of the human body into spreadsheet data fields, and reduce the expertise of lifelong scouts into mundane facts. But what emerged over the course of the “Science of Randomness” discussion sealed the coffin on that myth as the panel explored the ragged edge of reason.

The panel started innocuously enough, but took a heady turn quickly. A softball question from Morey led the panel: How do you gather data in such a way that insights stand out from misleading information? In short — you can’t. There are best practices, sure. You can explore the root causes of your metrics, as when you examine which pitches a pitcher throws in which situations by way of explaining his ERA. You can evaluate your staffs on their process rather than their outcomes. These things the panelists agreed on. What they also agreed on, however, is that certainty is an impossible goal, in sports or any enterprise.

It’s difficult to talk about advanced metrics and their rise in a broad way without addressing the conflict between new analytics and “old school” observers. It is a halfway exaggerated disagreement; we bloggers have not charged the convention center with our TI-83s, and very few aging scouts have ever moved a plug of tobacco over in their cheek to yell at us to get real jobs. But it is also very true that the analytics movement sees itself as a corrective.

As Nate Silver said, “Humans are very good at finding patterns, and the problem is that sometimes we have overactive imaginations.” Sloan is where those people gather, those who believe they provide the necessary checks on the human imagination as it pertains to sports. It’s easy to see their proselytizing as strident and self-important needling, like a freshman economics major who won’t stop quoting Marx to his suburban parents.

It is true that some analytics devotees are as inflexible and ham-handed as the stereotype. That risk runs with any new movement or methodology. But “The Science of Randomness” was a needed reminder that at its core, this movement is an open-minded one, and its best practitioners are almost shockingly agnostic. Good science is constant scrutiny -- are my methods right? Do I need to tweak my models to account for new information? But randomness is the inviolable truth. It is telling that of the five panelists, two are famously gamblers; Silver is a poker player and Ma made a fortune on blackjack and his story of that fortune. For these two, analytics is about the measurement of doubt, about pitting your resources against the certainty that you will often be wrong, and hoping you can beat it more often than it beats you.

In a way, hearing these panelists talk made analytics seem a sport of its own. The opponent is error, and through mastering the human tendency to act on our biases, we can increase the number of upsets we pull off. Taking a long view, and minimizing the weakness of the human mind, was a dominant theme.

This was Scheiner’s main point. Taking over a woeful Browns franchise, he knows that success is more about the marginal reduction of error than the short-term victories, but that his approach is asking a lot of fans. “How much short-term randomness do you tolerate?” he asked. “How much time do you have to prove a theory?”

In a sense, every panelist from “The Science of Randomness” is asking the same thing -- they are proving their theories in wins and losses. Sport provides the structure for their contest with the uncontrollable. Before they were in sports, they worked as tobacco control economists, or attorneys specializing in international diplomacy. Silver has moved on, famously, to political analysis. The versatility of the panelists underscore that, at its core, the analytics movement is about much more than the games themselves -- it’s about engaging the limits of what we can honestly say we know. Perhaps Birnbaum, the least famous panelist by a wide margin, summed up the appeal of analytics best when he said: “Even if God came down and said this number is absolutely correct, there’s still randomness.”