Nate Silver

November 15th, 2012 at 7:00 am by David Farrar

The Sydney Morning Herald reports:

The political emperors have no clothes, stripped bare by a big-data wizard named who showed dispassionate maths was more reliable than pundit intuition and cherry-picked .

Silver, 34, a statistician who previously predicted the career trajectories of baseball players, accurately tipped 49 out of 50 US states (with the 50th, Florida, highly likely to be accurate as well as Obama is ahead with 97 per cent of the votes counted) and most Senate contests.

As right-wing pundits attacked him and his “voodoo statistics” for failing to see that the election was on a knife edge – and in the case of some conservative wingnuts, for being openly gay and “effeminate” – Silver held his nerve and for the entire election cycle maintained that the data always pointed to an easy Obama victory. …

Even after Obama’s dismal first debate performance, Silver’s probability of Obama winning never dipped below 61.1 per cent, rising to more than 90 per cent on election day.

I am a big fan of both Silver’s analytic skills, and his demeanour while under fire. He deserves a lot of credit.

It is worth pointing out though that all the major polling aggregation sites did very well as reported by Cnet:

But Silver wasn’t the only one to do exceptionally well in the prediction department. In fact, each of the five aggregators that CNET surveyed yesterday — FiveThirtyEight, TPM PollTracker, HuffPost Pollster, the RealClearPolitics Average, and the Princeton Election Consortium — successfully called the election for Obama, and save for TPM PollTracker and RealClearPolitics handing Florida to Romney, the aggregators were spot on across the board when it came to picking swing state victors.

So if you listened to the polls rather than the pundits, you were likely to be correct. Why then is Silver the new political celebrity rather than say Mark Blumenthal who does HuffPost Pollster?

I think it is partly because Silver was attacked by several prominent pundits before the election. Those attacks backfired by giving him not just accuracy but vindication.

The other reason is that Silver does a bit more than just aggregate and weight the polls. His extra tweeks may not make a huge difference but they are seen as useful by many.

In addition to picking the winner in all 50 states — besting his 49 out of 50 slate in 2008 — Silver was also the closest among the aggregators to picking the two candidates’ popular vote percentages. All told, he missed Obama’s total of 50.8 percent by just four-tenths of a percentage point (50.4) and Romney’s 48 percent by just three-tenths of a point (48.3) for an average miss of just 0.35 percentage points. HuffPo Pollster and RealClearPolitics tied for second with an average miss of 0.85 points.

This may change a bit as the final votes come in. It is worth noting also that Silver didn’t have a 100% accuracy rate with calling Senate races. Again this takes nothing away from his highly deserved reputation – just that even his model is not infallible  The strength of his model, as I see it, is that it learns from the past.

So what does Silver do to predict who wins. His exact methodology is secret (he has said he may reveal more over time) but he has detailed what he does for Senate races. My summary of it is:

  1. Average the polls for that state
  2. Give more recent polls a higher weight using an exponential decay formula
  3. Weight by sample size so larger sample polls have more weight
  4. Assign an accuracy rating to each pollster and weight those historically more accurate, higher. Exclude polls from very dodgy pollsters or polls released by parties. Note that many other polling aggregators also do steps 1 to 4. What is unique to Silver tends to be the later steps.
  5.  Adjust the result based on the national trend, so if nationwide one party has dropped say 5% in one week, assume it applies to that state also.
  6. Adjust the result based on observed “house effects” for pollsters. So if one pollster consistently has Democrats 2% higher than they get, then take 2% off their poll.
  7. Adjust polls of registered voters as if they were of likely voters, based on the normal difference between such polls (Republicans do better with likely voters).
  8. Do a regression analysis of the state based on their partisan voting index, their party identification, donations to candidates, incumbency status, approval ratings for incumbents, and previous offices a candidate has been elected to
  9. Add the results of the regression analysis to the weighted average of polls, as if it is a poll.
  10. Do an error calculation
  11. Stimulate the election and report how often one candidate beats the other over multiple simulations

So Silver has a very sophisticated model. I think for presidential elections he also uses economic data such as GDP growth and unemployment rates. Over time as more and more data is gathered, his model should remain accurate or become even more accurate.

There will be times when it will be wrong, just as the polls sometimes get it wrong. No model can compensate if the election is very volatile and large numbers of voters change their mind or are undecided in the final few days. Events will always matter.

Tags: , ,

28 Responses to “Nate Silver”

  1. Sonny Blount (1,845 comments) says:

    Nate Silver has a model that worked well for the 2008 and 2012 electorate. If the underlying characteristics change then his model will need to change. This affects all pollsters and pundits whose accuracy comes and goes, for example Rasmussen was extremely accurate for several election cycles in the 90′s and 00′s but the picture they used of the electorate was off this time.

    Silver also bombed very badly in the 2010 elections when his assumptions were inaccurate, unless he has taken on board that epic failure then he will do badly in 2014 as well.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  2. Nigel Kearney (864 comments) says:

    Two things:

    It’s easier to be an acclaimed pollster when you have never predicted a Republican win. The first time he does that, the knives will be out, whether he is correct or not.

    An accurate prediction issued the day before the election is not that useful. What is more useful is getting it correct one or two months out. Success at one does not imply success at the other. Many extra factors come into play.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  3. Sonny Blount (1,845 comments) says:

    Nigel,

    Silver bombed badly in the 2010 Republican win. He called 75% against the GOP gaining 67 house seats.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  4. iMP (2,244 comments) says:

    Some of the polls were right this time; but they are raw tools and do not always pick the public undercurrents. The 2012 election was not won on policy issues (deficit, economy, taxes) but much broader considerations which averaged out across broad demographics.

    All we have are guesses, intuition, poll surveys (what people say they would do, which we extrapolate 1000 fold) and pundit gut instinct.

    I love that politics is unpredictable.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  5. peteremcc (341 comments) says:

    Technically a prediction model like Nate’s can never be “wrong”.

    When it predicts 80% Obama and 20% Romney, it’s not saying that Obama will win, it’s saying that 80% of the time Obama will win and 20% of the time Romney will win.

    If Romney wins, it doesn’t show the model is wrong, just that the 1/5 chance, rather than the 4/5 chance happened this time.

    It’s a minor, but important difference.

    Vote: Thumb up 1 Thumb down 0 You need to be logged in to vote
  6. nasska (10,659 comments) says:

    How do pollsters adjust for people such as myself who loathe having their dinner interrupted by phone calls & take revenge by politely giving totally bogus information?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  7. Graeme Edgeler (3,267 comments) says:

    This may change a bit as the final votes come in. It is worth noting also that Silver didn’t have a 100% accuracy rate with calling Senate races.

    While Silver’s model had the 51/51 he got in electoral college as the most likely of any single outcome outcome, his model predicted he’d probably be wrong. Even he only thought he had a 12.5% chance of being *that* correct.

    Same with senate races: if he thinks there are four seats with a 75% chance of going to one party, the most likely results is that they split 3-1 (42% chance), with 4-0 being second (32% chance). When someone is putting reasonable sized error margins in, we should *expect* them to be wrong. If they’re not wrong sometimes, then their model is less certain than it should be.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  8. somewhatthoughtful (452 comments) says:

    iMP, 2012 was very much won on policy. People voted that they didn’t want the Republican’s fascist, batshit crazy religious bullshit ruining their lives.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  9. Sonny Blount (1,845 comments) says:

    Um Somewhatthoughtful,

    The GOP won the house. The electorate has endorsed them, do you understand American politics?

    Obama was nearly unseated and was returned with the most heavily reduced majority since Andrew Jackson in 1832.

    Vote: Thumb up 0 Thumb down 1 You need to be logged in to vote
  10. eszett (2,334 comments) says:

    Sonny Blount (1,629) Says:
    November 15th, 2012 at 7:17 am

    Silver also bombed very badly in the 2010 elections when his assumptions were inaccurate, unless he has taken on board that epic failure then he will do badly in 2014 as well.

    Given the epic failure of your past predictions, sonny, it seems that you haven’t taken much on board of your past failings yourself.

    http://www.kiwiblog.co.nz/2012/10/the_al_smith_dinner.html#comment-1036963

    Sonny Blount (1,629) Says:
    October 20th, 2012 at 8:50 am

    The excitement of the race is past. It’s over, Romney has won, Barry is gone.

    Now its just whether we get a landslide and if the GOP can also take the Senate to go with Congress and the Presidency.

    Vote: Thumb up 1 Thumb down 0 You need to be logged in to vote
  11. eszett (2,334 comments) says:

    Sonny Blount (1,629) Says:
    November 15th, 2012 at 8:34 am

    Um Somewhatthoughtful,

    The GOP won the house. The electorate has endorsed them, do you understand American politics?

    Obama was nearly unseated and was returned with the most heavily reduced majority since Andrew Jackson in 1832.

    And Romney/Ryan got 1 million less votes than McCain/Palin. Now that’s gotta tell you something as well

    Vote: Thumb up 1 Thumb down 0 You need to be logged in to vote
  12. Graeme Edgeler (3,267 comments) says:

    Technically a prediction model like Nate’s can never be “wrong”.

    Yes it can. Some states he predicted as 100% likely to be won by one or other candidate.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  13. Graeme Edgeler (3,267 comments) says:

    The GOP won the house. The electorate has endorsed them, do you understand American politics?

    The GOP certainly won the House, but the claim that this is an endorsement of their policies is suspect. Democratic House candidates received more votes between them than Republican House candidates.

    Vote: Thumb up 1 Thumb down 0 You need to be logged in to vote
  14. Manolo (13,363 comments) says:

    A new gay and left-wing icon has been discovered.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  15. Ed Snack (1,737 comments) says:

    Eszett, it tells me that large scale, deceitful, negative campaigning works well when not properly answered both in response and in kind. But of course to criticize the “One” would be racist….Which explains the heavy emphasis on calling anything and everything critical of Obama racist for so long, setting the stage to make a Republican negative campaign that much harder to implement.

    It also tells me that when you have almost the entire MSM in the tank for you that depressing the opponents turn-out is the immoral but effective way to go.

    However we do have to wait for the final vote totals to be announced once counting has finished; the final tallies may not be less than the 2008 results.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  16. somewhatthoughtful (452 comments) says:

    Ha, Sonny, you’re a moron and I don’t think you actually understand how the US system works.

    You can dispute facts all you like but here it is:

    Obama won the presidency; by a landslide,

    Congress (well the portion that was up for election this time) involved the repubs losing 8 seats and the dems picking up 5 (http://en.wikipedia.org/wiki/United_States_House_of_Representatives_elections,_2012)

    In the senate, the dems increased their majority by 2 seats http://en.wikipedia.org/wiki/United_States_Senate_elections,_2012

    Yet somehow the platform wasn’t the problem? Message from the voters of the USA: go fuck yourself and your shitty platform.

    The hilarity is if those moronic christians would just fuck off then the repubs would probably have a stranglehold on US politics. I do love seeing the GOP being killed as it squirms for the “values voters” (i.e. morons who believe in fairies and that a book written by a bunch of corrupt scholars in the middle ages has more weight than empirical evidence).

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  17. Luc Hansen (4,573 comments) says:

    So what does Silver do to predict who wins.

    Sort of. He assigns probabilities. A firm prediction would be when his model assigns zero chance for a candidate to triumph. For a gambler, if the price is right, a few bucks on, for example, a 25% chance can be a worthwhile punt. That’s still a one chance in four runs. His model doesn’t say the 25% chance will lose.

    I see Sonny Blount is taking up the bullshit cudgels for the “resting’ (Iicking his wounds) Tom Hunter with his assertions re the 2010 mid-terms. Here is the relevant information from Wikipedia:

    In final vote tallys as of December 10, 2010, the Republicans had a net gain of 63 seats in the House, 8 more than the total predicted on election eve though still within the reported confidence interval.[103]

    That election saw an unexpectedly large swing on the day of the election, but Silver’s model still foretold of a handsome GOP win.

    The same source for the 2010 Senate races:

    The 538 model had forecast a net pickup of 7 seats by the Republicans in the Senate, but the outcome was a pickup of 6 seats.

    Not bad for a pro-liberal bias!

    But Sonny, shoot the messenger, by all means.

    Vote: Thumb up 1 Thumb down 0 You need to be logged in to vote
  18. Sonny Blount (1,845 comments) says:

    eszett (1,631) Says:
    November 15th, 2012 at 8:41 am
    Sonny Blount (1,629) Says:
    November 15th, 2012 at 7:17 am

    Silver also bombed very badly in the 2010 elections when his assumptions were inaccurate, unless he has taken on board that epic failure then he will do badly in 2014 as well.

    Given the epic failure of your past predictions, sonny, it seems that you haven’t taken much on board of your past failings yourself.

    http://www.kiwiblog.co.nz/2012/10/the_al_smith_dinner.html#comment-1036963

    Sonny Blount (1,629) Says:
    October 20th, 2012 at 8:50 am

    The excitement of the race is past. It’s over, Romney has won, Barry is gone.

    Now its just whether we get a landslide and if the GOP can also take the Senate to go with Congress and the Presidency.

    My mistake is a perfect example of what needs to be remembered in this thread.

    I put too much stock in pollsters that got the 2004 and 2008 elections almost exactly right.

    I will be careful not to go all in in 2016 with the guys who got 2008 and and 2012 right.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  19. eszett (2,334 comments) says:

    Ed Snack (728) Says:
    November 15th, 2012 at 8:56 am

    Eszett, it tells me that large scale, deceitful, negative campaigning works well when not properly answered both in response and in kind.

    Well, the campaign was very negative from both sides. But that seems to be the norm in America.

    When it comes to deceitful however, actually it doesn’t, otherwise Romney would have won. Even the Republicans are embarrassed about his flip flops and number of downright lies (not even distortions) that he produced.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  20. Sam (497 comments) says:

    How much effect will the publishing of Silver’s predictions, not that he has earned some fame, have on the result on the day? Regardless of whether he gets it ‘right’ or ‘wrong’, if he is seen as some kind of Delphic Oracle, isn’t that actually going to affect the outcome to some degree??

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  21. eszett (2,334 comments) says:

    My mistake is a perfect example of what needs to be remembered in this thread.

    Your mistake Sonny was that you only heard what you wished to be true and ignored everything (e.g. reality) that contradicted it.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  22. Sonny Blount (1,845 comments) says:

    somewhatthoughtful (371) Says:
    November 15th, 2012 at 8:58 am
    Ha, Sonny, you’re a moron and I don’t think you actually understand how the US system works.

    You can dispute facts all you like but here it is:

    Obama won the presidency; by a landslide,

    Congress (well the portion that was up for election this time)

    Every congress seat is up every election moron. And the GOP won it.

    Obama lost 9% of his majority from 2008, the GOP house lost 3% of their majority from 2012.

    You can’t call one a repudiation of policy and not the other. Obama won on character over Romney, the exit polls all had Romney ahead on economic management for one.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  23. Sonny Blount (1,845 comments) says:

    eszett (1,633) Says:
    November 15th, 2012 at 9:42 am
    My mistake is a perfect example of what needs to be remembered in this thread.

    Your mistake Sonny was that you only heard what you wished to be true and ignored everything (e.g. reality) that contradicted it.

    You are correct eszett. Quite ironic when discussing the current idolatry of Silver.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  24. BlairM (2,286 comments) says:

    I have no problem with what Nate Silver does per se. I do have a problem with people ascribing god-like genius to him, when the result of this election came about by factors nobody could have predicted.

    Sandy, ORCA, Obama’s ability to turn out blacks in even greater numbers than last time (100,000 more in Ohio!), Romney not being able to turn out rural whites – none of these things can be anticipated by a statistical model.

    I think he got it right by accident rather than by design. He certainly fluffed the 2010 midterms. And if your retort is that he would get it right “most of the time” then it becomes farcical to talk about, since we can only know that once he has been doing this for another 20 years or so.

    I think the bulk of those kissing his arse are simply happy their guy won.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  25. scrubone (3,048 comments) says:

    The problem is that you only have elections ever 2-4 years. So you might consider one pollster to have a 2% bias towards democrats but they adjust that themselves in the meantime. So it’s to a certain extent you’re trying to hit a moving target on that score.

    But it does make sense to make those adjustments in the meantime.

    Part of me is a bit sad that this one was so closely and accuratly predicted. It would have been nice to have some real shocks early in the night.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  26. Ed Snack (1,737 comments) says:

    Ezsett, what crap, the Obama campaign was way more negative than the Romney campaign all the way down to the election. Remember the ads on Bain Capital, accussing Romney of all sort of unspecified “crimes” committed by Bain (on actions that happened years after he left Bain), the accusation that someones spouse died of cancer because Bain shipped their job off overseas so they lost medical coverage (it was 5 years between the job change and the cancer), the Harry Reid tax lies (and Reid, a man who’s made himself relatively wealthy by kickbacks on sweetheart real estate deals !), and so on and so forth.

    The worst I can see on the Romney side is their ad on Jeep moving production to China, whereas Jeep were talking about new production (of course they were, they’d never announce job losses just before an election, would they ?). And that claim was also widely rebutted in the MSM so probably had a negative impact, whereas all the Obama lies were basically run by the MSM.

    Hell, the Dems are even boasting about how they pigeon-holed Romney from the start of the campaign, it’s not just sour grapes about a perceived bias, its precisely the bias that the Democrats set out to win the election with; because the simply couldn’t run on Obama’s record which is consistent failure in almost everything but an A+ in two, campaigning and blaming Bush for the economy.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  27. eszett (2,334 comments) says:

    The worst I can see on the Romney side is their ad on Jeep moving production to China,

    Yeah right Ed. That’s the worst you can see? That’s a joke, right?

    Remember on of the first Romney ads where apparently Obama was saying “if we keep talking about the economy, we are going to loose”

    That was the main catch phrase of the ad
    Only problem is Obama was actually quoting McCains campaign.

    That very first ad was never corrected or taken down and is still on Romneys youtube page.

    Romney and the GOP have only got themselves to blame for their lose.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  28. weizguy (120 comments) says:

    Sonny, The GOP held congress because of a Gerrymander, not because of policies or popularity:
    http://www.washingtonpost.com/opinions/harold-meyerson-gops-gerrymandered-advantages/2012/11/13/4785e4d6-2d2f-11e2-a99d-5c4203af7b7a_story.html
    http://www.slate.com/articles/news_and_politics/jurisprudence/2012/11/the_supreme_court_may_gut_the_voting_rights_act_and_make_gerrymandering.html

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote

Leave a Reply

You must be logged in to post a comment.