Nate Silver
November 15th, 2012 at 7:00 am by David FarrarThe Sydney Morning Herald reports:
The political emperors have no clothes, stripped bare by a big-data wizard named Nate Silver who showed dispassionate maths was more reliable than pundit intuition and cherry-picked polls.
Silver, 34, a statistician who previously predicted the career trajectories of baseball players, accurately tipped 49 out of 50 US states (with the 50th, Florida, highly likely to be accurate as well as Obama is ahead with 97 per cent of the votes counted) and most Senate contests.
As right-wing pundits attacked him and his “voodoo statistics” for failing to see that the election was on a knife edge – and in the case of some conservative wingnuts, for being openly gay and “effeminate” – Silver held his nerve and for the entire election cycle maintained that the data always pointed to an easy Obama victory. …
Even after Obama’s dismal first debate performance, Silver’s probability of Obama winning never dipped below 61.1 per cent, rising to more than 90 per cent on election day.
I am a big fan of both Silver’s analytic skills, and his demeanour while under fire. He deserves a lot of credit.
It is worth pointing out though that all the major polling aggregation sites did very well as reported by Cnet:
But Silver wasn’t the only one to do exceptionally well in the prediction department. In fact, each of the five aggregators that CNET surveyed yesterday — FiveThirtyEight, TPM PollTracker, HuffPost Pollster, the RealClearPolitics Average, and the Princeton Election Consortium — successfully called the election for Obama, and save for TPM PollTracker and RealClearPolitics handing Florida to Romney, the aggregators were spot on across the board when it came to picking swing state victors.
So if you listened to the polls rather than the pundits, you were likely to be correct. Why then is Silver the new political celebrity rather than say Mark Blumenthal who does HuffPost Pollster?
I think it is partly because Silver was attacked by several prominent pundits before the election. Those attacks backfired by giving him not just accuracy but vindication.
The other reason is that Silver does a bit more than just aggregate and weight the polls. His extra tweeks may not make a huge difference but they are seen as useful by many.
In addition to picking the winner in all 50 states — besting his 49 out of 50 slate in 2008 — Silver was also the closest among the aggregators to picking the two candidates’ popular vote percentages. All told, he missed Obama’s total of 50.8 percent by just four-tenths of a percentage point (50.4) and Romney’s 48 percent by just three-tenths of a point (48.3) for an average miss of just 0.35 percentage points. HuffPo Pollster and RealClearPolitics tied for second with an average miss of 0.85 points.
This may change a bit as the final votes come in. It is worth noting also that Silver didn’t have a 100% accuracy rate with calling Senate races. Again this takes nothing away from his highly deserved reputation – just that even his model is not infallible The strength of his model, as I see it, is that it learns from the past.
So what does Silver do to predict who wins. His exact methodology is secret (he has said he may reveal more over time) but he has detailed what he does for Senate races. My summary of it is:
- Average the polls for that state
- Give more recent polls a higher weight using an exponential decay formula
- Weight by sample size so larger sample polls have more weight
- Assign an accuracy rating to each pollster and weight those historically more accurate, higher. Exclude polls from very dodgy pollsters or polls released by parties. Note that many other polling aggregators also do steps 1 to 4. What is unique to Silver tends to be the later steps.
- Adjust the result based on the national trend, so if nationwide one party has dropped say 5% in one week, assume it applies to that state also.
- Adjust the result based on observed “house effects” for pollsters. So if one pollster consistently has Democrats 2% higher than they get, then take 2% off their poll.
- Adjust polls of registered voters as if they were of likely voters, based on the normal difference between such polls (Republicans do better with likely voters).
- Do a regression analysis of the state based on their partisan voting index, their party identification, donations to candidates, incumbency status, approval ratings for incumbents, and previous offices a candidate has been elected to
- Add the results of the regression analysis to the weighted average of polls, as if it is a poll.
- Do an error calculation
- Stimulate the election and report how often one candidate beats the other over multiple simulations
So Silver has a very sophisticated model. I think for presidential elections he also uses economic data such as GDP growth and unemployment rates. Over time as more and more data is gathered, his model should remain accurate or become even more accurate.
There will be times when it will be wrong, just as the polls sometimes get it wrong. No model can compensate if the election is very volatile and large numbers of voters change their mind or are undecided in the final few days. Events will always matter.
Tags: 2012 US Presidential election, Nate Silver, Polls
November 15th, 2012 at 7:17 am
Nate Silver has a model that worked well for the 2008 and 2012 electorate. If the underlying characteristics change then his model will need to change. This affects all pollsters and pundits whose accuracy comes and goes, for example Rasmussen was extremely accurate for several election cycles in the 90′s and 00′s but the picture they used of the electorate was off this time.
Silver also bombed very badly in the 2010 elections when his assumptions were inaccurate, unless he has taken on board that epic failure then he will do badly in 2014 as well.
Vote:November 15th, 2012 at 7:17 am
Two things:
It’s easier to be an acclaimed pollster when you have never predicted a Republican win. The first time he does that, the knives will be out, whether he is correct or not.
An accurate prediction issued the day before the election is not that useful. What is more useful is getting it correct one or two months out. Success at one does not imply success at the other. Many extra factors come into play.
Vote:November 15th, 2012 at 7:20 am
Nigel,
Silver bombed badly in the 2010 Republican win. He called 75% against the GOP gaining 67 house seats.
Vote:November 15th, 2012 at 7:36 am
Some of the polls were right this time; but they are raw tools and do not always pick the public undercurrents. The 2012 election was not won on policy issues (deficit, economy, taxes) but much broader considerations which averaged out across broad demographics.
All we have are guesses, intuition, poll surveys (what people say they would do, which we extrapolate 1000 fold) and pundit gut instinct.
I love that politics is unpredictable.
Vote:November 15th, 2012 at 7:38 am
Technically a prediction model like Nate’s can never be “wrong”.
When it predicts 80% Obama and 20% Romney, it’s not saying that Obama will win, it’s saying that 80% of the time Obama will win and 20% of the time Romney will win.
If Romney wins, it doesn’t show the model is wrong, just that the 1/5 chance, rather than the 4/5 chance happened this time.
It’s a minor, but important difference.
Vote:November 15th, 2012 at 8:03 am
How do pollsters adjust for people such as myself who loathe having their dinner interrupted by phone calls & take revenge by politely giving totally bogus information?
Vote:November 15th, 2012 at 8:04 am
While Silver’s model had the 51/51 he got in electoral college as the most likely of any single outcome outcome, his model predicted he’d probably be wrong. Even he only thought he had a 12.5% chance of being *that* correct.
Same with senate races: if he thinks there are four seats with a 75% chance of going to one party, the most likely results is that they split 3-1 (42% chance), with 4-0 being second (32% chance). When someone is putting reasonable sized error margins in, we should *expect* them to be wrong. If they’re not wrong sometimes, then their model is less certain than it should be.
Vote:November 15th, 2012 at 8:23 am
iMP, 2012 was very much won on policy. People voted that they didn’t want the Republican’s fascist, batshit crazy religious bullshit ruining their lives.
Vote:November 15th, 2012 at 8:34 am
Um Somewhatthoughtful,
The GOP won the house. The electorate has endorsed them, do you understand American politics?
Obama was nearly unseated and was returned with the most heavily reduced majority since Andrew Jackson in 1832.
Vote:November 15th, 2012 at 8:41 am
Given the epic failure of your past predictions, sonny, it seems that you haven’t taken much on board of your past failings yourself.
http://www.kiwiblog.co.nz/2012/10/the_al_smith_dinner.html#comment-1036963
Sonny Blount (1,629) Says:
October 20th, 2012 at 8:50 am
The excitement of the race is past. It’s over, Romney has won, Barry is gone.
Now its just whether we get a landslide and if the GOP can also take the Senate to go with Congress and the Presidency.
Vote:November 15th, 2012 at 8:43 am
And Romney/Ryan got 1 million less votes than McCain/Palin. Now that’s gotta tell you something as well
Vote:November 15th, 2012 at 8:50 am
Yes it can. Some states he predicted as 100% likely to be won by one or other candidate.
Vote:November 15th, 2012 at 8:53 am
The GOP certainly won the House, but the claim that this is an endorsement of their policies is suspect. Democratic House candidates received more votes between them than Republican House candidates.
Vote:November 15th, 2012 at 8:56 am
A new gay and left-wing icon has been discovered.
Vote:November 15th, 2012 at 8:56 am
Eszett, it tells me that large scale, deceitful, negative campaigning works well when not properly answered both in response and in kind. But of course to criticize the “One” would be racist….Which explains the heavy emphasis on calling anything and everything critical of Obama racist for so long, setting the stage to make a Republican negative campaign that much harder to implement.
It also tells me that when you have almost the entire MSM in the tank for you that depressing the opponents turn-out is the immoral but effective way to go.
However we do have to wait for the final vote totals to be announced once counting has finished; the final tallies may not be less than the 2008 results.
Vote:November 15th, 2012 at 8:58 am
Ha, Sonny, you’re a moron and I don’t think you actually understand how the US system works.
You can dispute facts all you like but here it is:
Obama won the presidency; by a landslide,
Congress (well the portion that was up for election this time) involved the repubs losing 8 seats and the dems picking up 5 (http://en.wikipedia.org/wiki/United_States_House_of_Representatives_elections,_2012)
In the senate, the dems increased their majority by 2 seats http://en.wikipedia.org/wiki/United_States_Senate_elections,_2012
Yet somehow the platform wasn’t the problem? Message from the voters of the USA: go fuck yourself and your shitty platform.
The hilarity is if those moronic christians would just fuck off then the repubs would probably have a stranglehold on US politics. I do love seeing the GOP being killed as it squirms for the “values voters” (i.e. morons who believe in fairies and that a book written by a bunch of corrupt scholars in the middle ages has more weight than empirical evidence).
Vote:November 15th, 2012 at 9:02 am
Sort of. He assigns probabilities. A firm prediction would be when his model assigns zero chance for a candidate to triumph. For a gambler, if the price is right, a few bucks on, for example, a 25% chance can be a worthwhile punt. That’s still a one chance in four runs. His model doesn’t say the 25% chance will lose.
I see Sonny Blount is taking up the bullshit cudgels for the “resting’ (Iicking his wounds) Tom Hunter with his assertions re the 2010 mid-terms. Here is the relevant information from Wikipedia:
That election saw an unexpectedly large swing on the day of the election, but Silver’s model still foretold of a handsome GOP win.
The same source for the 2010 Senate races:
Not bad for a pro-liberal bias!
But Sonny, shoot the messenger, by all means.
Vote:November 15th, 2012 at 9:35 am
My mistake is a perfect example of what needs to be remembered in this thread.
I put too much stock in pollsters that got the 2004 and 2008 elections almost exactly right.
I will be careful not to go all in in 2016 with the guys who got 2008 and and 2012 right.
Vote:November 15th, 2012 at 9:39 am
Well, the campaign was very negative from both sides. But that seems to be the norm in America.
When it comes to deceitful however, actually it doesn’t, otherwise Romney would have won. Even the Republicans are embarrassed about his flip flops and number of downright lies (not even distortions) that he produced.
Vote:November 15th, 2012 at 9:41 am
How much effect will the publishing of Silver’s predictions, not that he has earned some fame, have on the result on the day? Regardless of whether he gets it ‘right’ or ‘wrong’, if he is seen as some kind of Delphic Oracle, isn’t that actually going to affect the outcome to some degree??
Vote:November 15th, 2012 at 9:42 am
Your mistake Sonny was that you only heard what you wished to be true and ignored everything (e.g. reality) that contradicted it.
Vote:November 15th, 2012 at 9:48 am
Every congress seat is up every election moron. And the GOP won it.
Obama lost 9% of his majority from 2008, the GOP house lost 3% of their majority from 2012.
You can’t call one a repudiation of policy and not the other. Obama won on character over Romney, the exit polls all had Romney ahead on economic management for one.
Vote:November 15th, 2012 at 10:02 am
You are correct eszett. Quite ironic when discussing the current idolatry of Silver.
Vote:November 15th, 2012 at 10:34 am
I have no problem with what Nate Silver does per se. I do have a problem with people ascribing god-like genius to him, when the result of this election came about by factors nobody could have predicted.
Sandy, ORCA, Obama’s ability to turn out blacks in even greater numbers than last time (100,000 more in Ohio!), Romney not being able to turn out rural whites – none of these things can be anticipated by a statistical model.
I think he got it right by accident rather than by design. He certainly fluffed the 2010 midterms. And if your retort is that he would get it right “most of the time” then it becomes farcical to talk about, since we can only know that once he has been doing this for another 20 years or so.
I think the bulk of those kissing his arse are simply happy their guy won.
Vote:November 15th, 2012 at 10:59 am
The problem is that you only have elections ever 2-4 years. So you might consider one pollster to have a 2% bias towards democrats but they adjust that themselves in the meantime. So it’s to a certain extent you’re trying to hit a moving target on that score.
But it does make sense to make those adjustments in the meantime.
Part of me is a bit sad that this one was so closely and accuratly predicted. It would have been nice to have some real shocks early in the night.
Vote:November 15th, 2012 at 12:24 pm
Ezsett, what crap, the Obama campaign was way more negative than the Romney campaign all the way down to the election. Remember the ads on Bain Capital, accussing Romney of all sort of unspecified “crimes” committed by Bain (on actions that happened years after he left Bain), the accusation that someones spouse died of cancer because Bain shipped their job off overseas so they lost medical coverage (it was 5 years between the job change and the cancer), the Harry Reid tax lies (and Reid, a man who’s made himself relatively wealthy by kickbacks on sweetheart real estate deals !), and so on and so forth.
The worst I can see on the Romney side is their ad on Jeep moving production to China, whereas Jeep were talking about new production (of course they were, they’d never announce job losses just before an election, would they ?). And that claim was also widely rebutted in the MSM so probably had a negative impact, whereas all the Obama lies were basically run by the MSM.
Hell, the Dems are even boasting about how they pigeon-holed Romney from the start of the campaign, it’s not just sour grapes about a perceived bias, its precisely the bias that the Democrats set out to win the election with; because the simply couldn’t run on Obama’s record which is consistent failure in almost everything but an A+ in two, campaigning and blaming Bush for the economy.
Vote:November 15th, 2012 at 1:33 pm Vote:
November 16th, 2012 at 1:33 am
Sonny, The GOP held congress because of a Gerrymander, not because of policies or popularity:
Vote:http://www.washingtonpost.com/opinions/harold-meyerson-gops-gerrymandered-advantages/2012/11/13/4785e4d6-2d2f-11e2-a99d-5c4203af7b7a_story.html
http://www.slate.com/articles/news_and_politics/jurisprudence/2012/11/the_supreme_court_may_gut_the_voting_rights_act_and_make_gerrymandering.html