What is Nassim Taleb's criticism of 538's election model?

Nassim Taleb has tweeted criticism about 538's political predictions that I don't follow. He seems to be making a theoretical argument. I am very weak on stochastic modeling (though I know basic PDEs). Can someone help me understand at least the gist of what he is saying?

Solution 1:

Taleb's example chart is not particularly helpful to his presentation, as it suggests that FiveThirtyEight were always closer to the final result than his preferred methodology. In fact the two were estimating different things, and the chart demonstrates this with 538's much greater volatility.

Taleb seems to be saying that the assessed probability X wins the actual election (as seen $100$ or even $5$ days before the election) should be much closer to $50\%$ than FiveThirtyEight suggest, as lots of different things could happen to shift opinion in either direction between now and election day. He uses the mathematics of options for this, including the "no arbitrage" condition, but that detail is not particularly relevant.

FiveThirtyEight's methodology essentially takes current information available, especially but not exclusively from recent opinion polls, and projects uncertainty around this information taking into account other factors such as demographics, endorsements and polling trends, and then runs this uncertainty though an electoral college system to translate into uncertainty over the final result.

Taleb's analysis includes the possibility that things will change and that this could change people's opinions in a way not covered by FiveThirtyEight's look at current opinions. So while 538 concentrates on current uncertainty, Taleb's analysis is dominated by future uncertainty. If 538 says Clinton currently has a $74.1\%$ chance of beating Trump, then Taleb would reply that this is excessive (he would say the number should now be closer to $50\%$ as opinions could change between now and November), to which 538 would respond that their number was an assessment of the current position given their best assessment of voters' current opinions and the uncertainty about reports of the current position.

Taleb has bolded the word future to highlight this difference.

Solution 2:

In short Taleb argues that Silver's predictions are too volatile to be reasonable. An extreme example of this principle is that if a year from the election an outcome has a 95% probability then you should be unlikely to have an event that would drop the probability to 50% the next day; if such an event were likely you would be assigning a probability closer to 50%. While such swings could happen occasionally they should not happen repeatedly.

It is likely that Nate Silver's models are too sensitive to information well in advance of the election; perhaps having a stronger prior close to 50% more than a couple of weeks from the election is more reasonable.

That said Taleb demonstrates this for one highly idealized model--a Brownian motion. The real process has discrete shocks at predetermined times, the debates. It also has randomly timed shocks such as Comey's statements. I suspect even a model that accounted for those would have been suggesting Nate's forecast should have been less volatile . Also the only prediction I heard from him was a statement that a Trump Presidency is very possible---qualitatively similar to Nate Silver's result .