Nate Silver Is not a Witch

Early this year, Nate Silver wore a halo. Having called 50 of 50 states correctly in November’s election, Silver was declared the true winner of the presidential race and even the staunchest of realists were starting to suspect there might be some supernatural forces on his side. His 1.000 batting average helped maintain his book’s best-seller status, drove unprecedented traffic to the New York Times website’s politics section, and won him the uncontested title of America’s favorite statistician.

So why did he leave for ESPN?

The unexpected move was retroactively attributed to a number of potential causes. Maybe his dry, statistical, Billy Beane approach to politics didn’t mesh with the Times’s journalistic standards. Maybe he wanted the Grantland-esque freedom to realize his own creative vision. Or maybe he really just wanted to reconnect with his first love, sports.

While any of these reasons may be valid, I suspect there’s another reason Silver ditched his established spot in the limelight. And I suspect there’s only one publication that covered the move to ESPN from the right angle.

That publication is The Onion.

The article was titled “Nate Silver Warns Against Overestimating His Value To ESPN” and, as is standard for The Onion, its content can be entirely inferred from the headline. Though the authors had satirical intentions, they hit the nail on the head: Realizing he couldn’t replicate his 2012 feat, Silver quit the world of punditry while he was ahead.

Yes, Silver will continue to be a better predictor than a TV pundit. Are you surprised? Statistics will consistently score higher than gut feelings. But it’s not magic.

There are two reasons Silver can’t throw a perfect game again in 2016. The first is regression to the mean: Whenever someone does very well in any context, part of his success is due to luck and he probably won’t do as well next time. Regression to the mean is the reason the Colts are favored to win 8.5 games this season after their 11-5 record last year. It’s the reason an artist with a fantastic debut album never meets expectations with her second release. It’s the reason educators often believe the proverbial carrot is less effective than the proverbial stick, since students do worse after doing well (whether or not they were lauded) and do better after doing poorly (whether or not they were punished). Regression to the mean is the reason for the sophomore slump, the Sports Illustrated cover jinx, the Madden curse—all names given to imaginary causal explanations for a counterintuitive universal truth. And it’s one reason Silver won’t predict all 50 states next time.

The second reason is that his model secretly isn’t that great. It’s miles above anything out of a pundit’s mouth, but as far as statistical models go it’s needlessly complex. This allows him to overfit to past results and dilute the actually important variables.

Silver has a history of developing exceedingly complicated statistical models: He first became famous for the baseball prediction model PECOTA, described by its inheritors as “large, complex, and full of creaky interactions and pinch points.” While it’s still considered state-of-the-art, it only performs marginally better than Marcel, a “Monkey Forecasting System” you can carry out with pen and paper, designed to represent an absolute baseline level of competence.

So how does Silver’s FiveThirtyEight model compare to the simple models of the presidential forecasting world? Not favorably. Political scientist Drew Linzer built a statistical model based only on approval rating adjusted by state polls, and it blew Silver out of the water. While Silver’s November 6 forecast showed a 70 percent chance of Obama victory and an expected electoral vote count of 313, Linzer predicted Obama’s win with 99 percent certainty and correctly called the 332-206 result—wait for it—in June!

But Silver is far smarter than I’ve given him credit for. He knows what regression to the mean is, and he has acknowledged that his model is center-skewed compared to other statisticians’. And that’s why he’s off to ESPN.

He did what no other poll-watcher could do: He made statistics cool. He explained himself. He made nice charts and compelling visuals. He was hailed as a nerd demigod and a rock star. His legacy is not an impressive election prediction. His legacy is a new wave of election-followers dedicated to realism over sensationalism, and he doesn’t want to tarnish it by showing them the other end of the bell curve.

Nate Silver knows that 2016 can only hurt his pristine reputation. That’s why he’s done the mainstream media one last favor, by leaving it behind.

Milo B. Beckman ’15 is a government concentrator in Eliot House.