The Science of Prediction: Losing the Super Bowl

I am a little late to the party, but I just finished reading Nate Silver’s The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t (Penguin: 2012). On the whole, it is an excellent book that makes the fields of statistics and probability accessible to lay readers. Silver builds on his own work in baseball and in predicting election results to explore why it is so challenging to make predictions about the future across many fields, from meteorology to earthquakes to national security.[1] One of the chapters I found most interesting concerned the behavior of “pundits,” or the people you might see on television on Sunday morning making predictions about political outcomes. Silver analyzed a particular group of pundits and found that their predictions were no more accurate than a coin toss—and yet they are still asked to return to the show every week.

Silver draws on fascinating work by Phil Tetlock to explore a number of reasons why political outcomes are difficult to predict and why television pundits in particular are so bad at making predictions. Silver highlights some major pitfalls that pundits (and we) fall into when making predictions about the future: failing to think probabilistically and failing to update predictions when we receive new information about the world. No one can be 100% certain about what the future holds. Making good and useful predictions requires us to be explicit about the amount of uncertainty attached to a particular prediction. For example, as Silver points out in a later chapter, we are much more accustomed to seeing uncertainty expressed in weather forecasts (40% chance of rain) than in forecasts of political phenomena (20% chance this intervention will succeed). Making good predictions also requires that we be willing to change our predictions in the face of new information. Political candidates, for example, are often criticized for changing their policy positions (“flip flopping”), but an unwillingness to update our assessments and predictions in the face of new information inhibits the development of sound predictions and sound policy. So, for example, if we predict that a certain football team is likely to win the Super Bowl (60% chance) and then the team’s star quarterback sustains a career-ending injury, we would be foolish not to update our assessment of the likelihood that the team will win. We should be applying the same logic to our public policy making, particularly in the realm of national security.

The authors of an open letter to President Obama published in the National Interest on June 3 commit both of these cardinal sins of prediction. They urge the president to freeze American troop levels in Afghanistan at 10,000, barring “emergency conditions” that might favor a modest increase. The authors identify themselves as the Ambassadors to Afghanistan and invoke the ghosts of 9/11 and the threat of IS to justify their prediction that keeping troop levels this high will make the United States safer. They do not include any estimate of how likely their recommendation is to achieve the desired outcome (which they do not specify with any clarity). Nor do they seem to have updated their prescriptions after fifteen years of failure for our current policies. Daniel Davis published a great rebuttal two weeks later. As he argues, “This open letter to the president on Afghanistan is like a group of NFL coaches who have led teams to last place finishes ten straight years while trying to convince an owner to take their guidance on how to run his team. Their advice would be immediately rejected.”

In my last post on accountability, I expressed anger about the fact that a generation of pundits and politicians and advisors who have been wrong repeatedly about the wars in Iraq and Afghanistan are still invited to give advice about the future direction of American policy. There is one very good reason why these individuals are allowed to keep making public (failed) predictions: certainty sells. That is, it is much easier to sell books and make appearances on Meet the Press by overstating the confidence of one’s predictions than it is to be explicitly honest about the limits of one’s knowledge of a given issue and about the uncertainty attached to a policy prescription. I am tired of listening to people who brought us the failed wars in Iraq and Afghanistan tell us that we need to keep troops there, that we need to send just a few more Special Operations Forces, and that we need to stay just a few more years. The Super Bowls of Afghanistan and Iraq are over, and we lost. It’s not clear that anyone else won, but we definitely lost. It’s time to fire the coaches.

[1] The last chapter, in which Silver attempts to model the frequency of terrorist attacks like earthquakes, suffers from being totally atheoretical. I don’t think that this approach to predicting the distribution of a human behavior like terrorist attacks works as well as it does for understanding the distribution of natural phenomena like earthquakes.