How the pollsters did

November 29th, 2011 at 9:00 am by David Farrar

I blogged on Friday the final by the five public telephone pollsters and the final poll by Horizon. A fuller analysis will be done once we get final results, but for now I’ll do a quick analysis of how each pollster did for each party.

Note that this is not comparing apples and apples entirely. Those pollsters whose final poll was earlier in the election period naturally do not pick up what happens in the final few days. And there are other factors at work such as sample sizes. So this is not about saying who is “best” and “worst” but just a quick look at were they broadly in the right ballpark for the various parties.

This shows the actual result, and the (absolute) difference between the final poll for that pollster and the final result. Where the difference was greater than 1.5%, I have highlighted them in red.

This is just one of several ways to analyse it. One can also total up the differences for each pollster. Also you can count how many had a result within the margin of error for that poll. I’ll comment on each poll result.

Roy Morgan

They were the pollster that got NZ First closest. They had National and Greens too high and Labour too low. They did not record results for the Conservative Party at all, but otherwise were pretty good.

Fairfax Research International

National significantly too high, but Labour pretty accurate. Undershot NZ First and did not report on Conservatives. Other Minors within range.

3 News Reid Research

Like everyone had National too high (but within margin of error) and like most had the Greens too high. All other minor parties within 1.5% except NZ First whom they had at half what they got.

One News Colmar Brunton

Overall seemed to get things closest. National 2% too high and NZ First 2.6% too low, all others less than a 1% variance.

NZ Herald Digipoll

Also did well. National too high and NZ First too low, but did have them over 5%. Slightly more variance with the minor parties but none greater than 1.5%.

Horizon Poll

Of the nine significant parties, Horizon only got two of them within 1.5% – the Maori Party and United Future parties. They were the least accurate with National (14.2% out), NZ First (4.1% out), ACT (1.7% out) and Mana (1.8% out). They also had Conservatives at close to double what they actually got.

Very amusingly, Horizon are boasting how they consider their poll to have been highly accurate. It staggers me how anyone can put out a poll which had National only 5% ahead of Labour and then could claim it was “close to forecast” when the actual result was a gap of 21%.

20 Responses to “How the pollsters did”

  1. Auberon (810 comments) says:

    What an utter disgrace Horizon is, from their shonky original database made up of the contact list of the Business Council for Sustainable Development, to their common practice of only including the political element of their poll at the tale end of polls primarily about fringe issues, therefore unlikely to attract large numbers of participants. And then there’s their online format, which makes it easy for organised groups to mass enroll their members, therefore skewing the results. Bollocks, bollocks, bollocks.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  2. iMP (2,665 comments) says:

    Speaking of polls, it beggars belief that Denis O’Rourke and Richard Prosser are elected as ‘NZF MPs’ with 636 and 538 actual votes, respectively in Christchurch, while Nicky Wagner or Brendan Burns will lose Chch Central by as much as 11,000 actuals. This is MMP, and shows the appalling contempt the system has for voters. Yes, O’Rourke/Prosser are elected on a Party Vote wave, but who ever votes for a party based on the people on its List? This is not democracy.

    I believe the review should rank Party List candidates after the election, based on their electorate votes (ie they must stand in an electorate to be on a List, which National now does). This forces them to fight a local campaign, and work hard.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  3. Richard29 (377 comments) says:

    @ iMP
    “with 636 and 538 actual votes”
    By which I presume you mean ‘electorate votes’. The electorate vote is the less important vote under MMP. Electorates are effectively FPP two horse races, so it doesn’t make sense for small party supporters to give their votes to a candidate and split the vote. Just look at Wellington Central the Greens got 8000 party votes but their candidate got just 3000 electorate votes – 5000 or so Greens voted strategically for grant Robertson to give him the win.

    “but who ever votes for a party based on the people on its List?”
    The whole idea of the party vote is people are not voting for the people they are voting for the policy platform of the party. Most NZ First voters don’t give a damn who the MP’s are – they are voting for 8 seats worth of influence for Winston and trusting him to pick the team that best allows him to deliver on his promises.

    (heheheheee – I managed to include ‘Winston’, ‘trusting him’ and ‘deliver on his promises’ all in one sentence!)

    Given you seem to strongly preference the electorate vote for the ‘person’ a better question might be how many people are voting for the person and not the party? I guess the answer to that would be to remove the party affiliations of the candidates from the ballot and see who people vote for – most people wouldn;t have a clue who they are voting for – I rather suspect you would get local government levels of voter turnout (20-40 something percent) and a landslide victory for Adam Aardvark and the ‘AAA’ party.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  4. iMP (2,665 comments) says:

    The system is skewed. Voters who UTTERLY reject a personality at local body level or for the Electorate Vote, are appointed to parliament on a Party List simply as in-fill. Tom Scott’s cartoon of the NZF Hillbillies arriving in Wellywood, got it right. You could
    have Border Collies on your list, and they would still be ‘elected’ under this anomaly.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  5. nasska (16,781 comments) says:


    As a border collie owner I resent the association of an intelligent hard working dog of sober habits with a NZ First list MP.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  6. leftyliberal (655 comments) says:

    Interesting that Horizon is claiming they got it right due to their percentage for National being of eligible voters not actual voters. They don’t appear to have predicted the high “I’m not going to bother voting” vote, however, making this point completely bunk.

    IMO NZ is over-polled. It would be better to combine some of the polling (eg TV one work with one of the newspapers) in order to generate larger sample sizes and to ensure non-land-line folk get covered. In addition, weighted sampling within age + sex based on turnout would be useful (if not already done).

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  7. tvb (5,518 comments) says:

    Nothing about ipredict which is the best of all. Different methodology but a player nevertheless.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  8. s.russell (2,089 comments) says:

    It will be interesting to see the next batch of polls. My theory is that National’s underperformance compared to those polls was not becasue the polls were wrong, but because a lot of National supporters did not bother to vote – ironically because the polls told them they did not need to, National had it in the bag.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  9. zinger (3 comments) says:

    Well you might expect Horizon Research to have its head right up its own arse re their rubbish numbers, but the real question is why the ostensibly credible Sunday broadsheet Sunday Star Times continued to publish them – on the front page what’s more.
    It is obvious to anyone who is not an ostrich, that when all the mainstream polls (with reasonable accuracy in the last few elections) are saying about 50% for National and one weirdo outfit is saying 33%, which side is more credible.
    The paper should be embarassed and apologetic but I rather suspect it will be water off a duck’s back – a pretty sad reflection on the quality of some mainstream journalism in NZ!

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  10. Nick R (949 comments) says:

    @ DPF – Do you have any idea why Horizon got the National vote so badly wrong? They did enough to get most of the other parties within 2%, which would probably be close enough for them to be taken seriously, if they hadn’t missed the National vote by such a long way.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  11. insider (939 comments) says:

    I know they are within the margins of error, but being out by 50%-100% on a small party like Act or NZ First seems just as bad as 14% on a large party. We saw some major variations on the bottom end of the party vote across all the reputable pollsters, which means you have to be a bit cautious in overinterpreting the micro vote parties – Mana, Act, UF, Cons, Maori. I suspect just one person’s opinion could significantly affect that data.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  12. Auberon (810 comments) says:

    Not just the Sunday Star Times zinger, but also Radio Live which peppered the campaign with its “exclusive Radio Live Horizon Research polls,” each of them more b/s than the previous.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  13. Someone Else (140 comments) says:

    For the minor parties, polling and the 5 percent threshold creates all sorts of problems. People are influenced by polls and may change their vote if they fear it being wasted. Either remove the polls or the 5 percent threshold and you won’t have these problems.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  14. fishe (179 comments) says:

    Are there good reasons to allow poll results to be published publicly?

    Sure, political parties etc. obviously need polling to establish and confirm strategy decisions, however there seems to be enough evidence collected over the years that poll results have a powerful effect on voter decisions.

    At a more general level, the Wisdom of Crowds effect (where the combined opinion of a mass of people can be remarkably ‘correct’) is significantly skewed when individuals are exposed to the choices of others. E.g. check this study:

    I recognise people are informed by all sorts of opinion and inputs around election time, but this is something with a strong and likely negative effect that doesn’t seem to be too difficult to control for.

    DPF: what are your thoughts on this? Would it be wise/possible to make illegal the publication of political polls?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  15. Jimbob (641 comments) says:

    How about no polls in the last week of the election campaign. Then there is no influence.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  16. fishe (179 comments) says:

    Jimbob: I reckon you’d have to make that no polls in the last 2-3 months to remove most of the influence…and then you might as well not publicise them at all.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  17. JamesS (352 comments) says:

    The opinion polls were remarkably accurate and I am impressed.

    It is difficult to precisely forecast percentages in a General Election because of all the variables – it rains, something happening at the last minute, one party’s excellent grassroots organisation in certain electorates, one party’s lack of grassroots organisation (etc) – which can put the percentages out a little bit.

    There is also the slight matter that probably 40,000 decided to vote Winston at the last minute, not so much an inaccuracy in polling but more a case of various people ummmm-ing and ahhhh-ing for a couple of weeks and then deciding to tick the NZ First box.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  18. Morgy (173 comments) says:

    Will this mean Martyn Bradbury will STFU about landlines vs cellphone polls?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  19. aitkenmike (99 comments) says:

    iMP: Equally, you could have a border collie stand for the electoral seat of Taranaki-King Country, and as long as he had the word ‘national’ beside his name, he would be the local MP, ESPECIALLY in a FPP format. The solution to your problem (if one sees a problem) is open lists.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  20. Richard Prosser (3 comments) says:

    iMP, the system of voting we have is called MMP. It works like this. You have two votes, one for a local electorate MP who is selected by simple plurality under the First-Past-The-Post system, and the other for the Party Vote which is counted proportionally by way of percentages.

    Getting all bitter and twisted because you didn’t get the election result you might have preferred is no real reason for bagging the system, unless of course you don’t really like the idea of democracy.

    NZ First candidates all ran Party Vote campaigns, telling voters that we weren’t looking for the electorate vote. The Party Vote was what we were after, it’s what we got, and we’re in Parliament because of that.

    If you want an example of a system that has appalling contempt for voters, look no further than FPP. In 1978 and again in 1981 Labour received more actual votes than National, yet National still achieved an absolute majority of seats in the House. Democracy? I don’t think so.

    Appalling contempt for voters is the 1981 result where Social Credit got 21% of the vote but just 2% of the seats. Democracy? Not in my book.

    Appalling contempt for voters is the FPP race which electorate candidates still run, allowing a Member to be returned as the “winner” despite being ‘utterly rejected’ by anything up to 65% of voters in that electorate.

    Appalling contempt for voters is the blatant stitch-ups rorted on the people in electorates such as Epsom and Ohariu.

    Appalling contempt for voters is John Key claiming he has a mandate for asset sales, when an overall numerical majority of voters utterly rejected both him and his Party.

    We’re there now whether you like it or not, and we will work with all Parties to promote good policy and oppose bad policy, in the best interests of the nation and it’s people. Be happy. Negativity is not the way forward.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote