NZ public poll methodologies

June 10th, 2014 at 10:00 am by David Farrar

nz-polling-methods

Andrew at Grumpollie has put together this very useful table showing the different methodologies of the five public pollsters (I don’t count Horizon) in NZ.

Russell Brown at Public Address noted:

One big advantage for the political Left of John Banks’ sorry experience with the courts last week is that it meant people weren’t talking about the Left’s really awful result in the latest Roy Morgan poll.

Morgan has National up seven points to 52.5% support, and Labour and the Greens both down to a combined 38%. The Greens shed 4.5 points to slump to 9% support, their lowest level since 2011. This will hurt at The Standard and the Daily Blog, where and their inclusion of mobile phones are something of an article of faith.

It’s possible that this is an outlier poll — it does, after all, show Act doubling its support — but while Gary Morgan’s commentary on the results is typically bonkers, there’s nothing in particular wrong with the company’s methodology. And, significantly, the swing is reflected in the regular Government Confidence Rating (whether New Zealand is “heading in the right direction” or not.) It simply looks like a very healthy post-Budget poll for National.

But a friend put another interpretation to me on Friday: that the public has had a look at Internet-Mana and decided a potential centre-left coalition is really not to its taste. Perhaps Labour has internal polling to similar effect, explaining the spluttering reaction of of a number of Labour MPs to the prospect of cooperating with the party of Kim Dotcom and Laila Harre.

The commentary on the Roy Morgan polls is generally hilarious, and somewhat removed from reality. This doesn’t mean their polls are inaccurate.

However what Andrew’s table shows is that we know very little about how they conduct their polls – which would help people make a judgement on reliability.

The other four pollsters have signed up to the NZ Political Polling Code. This requires signatories to publicly release significant aspects of their methodologies. This is an important step for transparency. Roy Morgan has not signed up to the code, and we don’t know a lot about how their polls are done. We don’t even know if they weight the polls to the NZ adult population.

This doesn’t mean their polls are wrong, just as it doesn’t mean pollsters who have signed up will always get it right. For example a poll I did on attitudes to smoking and lung cancer found a lower prevalence rate for smoking than the census. Now the census figure is almost certainly the more accurate, so the difference may be down to how people respond to a phone poll vs a census, or it may be that even with weighting we under-surveyed current smokers. Good pollsters will always be critiquing their own methodology and considering how to enhance or review it.

It would be a very good thing if Roy Morgan did release more information on their methodology, so people can understand their results better in the right context.

Tags: ,

20 Responses to “NZ public poll methodologies”

  1. wreck1080 (3,787 comments) says:

    I wonder what would happen if labour said they refuse to deal with the loopy greens /mana/internet.

    Risky but do labour really want to form an unstable govt?

    Vote: Thumb up 5 Thumb down 0 You need to be logged in to vote
  2. Pete George (23,165 comments) says:

    But a friend put another interpretation to me on Friday: that the public has had a look at Internet-Mana and decided a potential centre-left coalition is really not to its taste. Perhaps Labour has internal polling to similar effect, explaining the spluttering reaction of of a number of Labour MPs to the prospect of cooperating with the party of Kim Dotcom and Laila Harre.

    Last week’s Roy Morgan poll would barely have covered the news of Laile Harre becoming leader and the IP-MANA deal so I think it’s too soon to determine this. The next poll may give us more of an inkling.

    Vote: Thumb up 1 Thumb down 0 You need to be logged in to vote
  3. alloytoo (456 comments) says:

    “One big advantage for the political Left of John Banks’ sorry experience with the courts last week is that it meant people weren’t talking about the Left’s really awful result in the latest Roy Morgan poll.”

    Actually people were, and it was usually part of the same conversation.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  4. flipper (3,766 comments) says:

    So how does all that apply to Curia’s rolling average…or to Pundit’s Silver based “Poll of Polls” ????

    What say U … DPF????

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  5. ShawnLH (4,330 comments) says:

    The Morgan poll confirms the trend over the last few months. Support for the Left overall is falling, and Labour is stuck in the 29-30% level where it will be unable to win the election.

    Labour’s real problem now is that a belief amongst it’s supporters that it has no chance of winning will translate on election day to a collapse in Labour voter turnout.

    Here’s hoping! :)

    Vote: Thumb up 6 Thumb down 0 You need to be logged in to vote
  6. davidp (3,551 comments) says:

    >Morgan has National up seven points to 52.5% support, and Labour and the Greens both down to a combined 38%. The Greens shed 4.5 points to slump to 9% support, their lowest level since 2011. This will hurt at The Standard and the Daily Blog, where Roy Morgan polls and their inclusion of mobile phones are something of an article of faith.

    The polls might be awful for them, but Labour and the Greens are partying this week. After six years of effort they’ve finally had a victory and driven John Banks from parliament. Okay, he might have been retiring anyway. And it is hard for them to present the court verdict as being about anonymous donations when David Cunliffe has refused to disclose the names of his anonymous donors. But when you’re a left wing politician or supporter you need to rejoice in every triumph, no matter how minor and how infrequent.

    Vote: Thumb up 5 Thumb down 0 You need to be logged in to vote
  7. OneTrack (2,754 comments) says:

    ” it meant people weren’t talking about the Left’s really awful result in the latest Roy Morgan poll.”

    No, they weren’t talking about the poll because it was being downplayed? by the LSM. For example, was it covered at all in the Herald?

    Vote: Thumb up 3 Thumb down 0 You need to be logged in to vote
  8. OneTrack (2,754 comments) says:

    “But when you’re a left wing politician or supporter you need to rejoice ”

    When they “get” one of the enemy.

    Vote: Thumb up 1 Thumb down 0 You need to be logged in to vote
  9. OneTrack (2,754 comments) says:

    And when you think the NZ left wing cant get any lower, they dig down further.

    In a nutshell, they don’t like what the polls are saying, so now their strategy is attack the polls and polling companies?

    Are they are so convinced of their godliness and infallibility, that everybody else, including the polls, must be simply “wrong”, hence need to be “fixed”. What a joke.

    There can’t be many sharks left to jump. Surely.

    Vote: Thumb up 3 Thumb down 0 You need to be logged in to vote
  10. johnwellingtonwells (127 comments) says:

    So the reported sample size of the poll is 1000. But how many interviews were attempted? Is the small sample achieved representative? How is the sampling error calculated?
    National Council on Public Polls

    All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with nonresponse, error associated with question wording and response options, and post-survey weighting and adjustments. Therefore, Harris Interactive avoids the words “margin of error” as they are misleading. All that can be calculated are different possible sampling errors with different probabilities for pure, unweighted, random samples with 100% response rates. These are only theoretical because no published polls come close to this ideal.These statements conform to the principles of disclosure of the National Council on Public Polls (USA)

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  11. Alan (1,073 comments) says:

    What I’d like to see are subsamples. Breakdown by sex and age. I’d also like to see results based on previous affiliation.

    The headline VI numbers are of pretty limited use

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  12. Andrew (81 comments) says:

    @johnwellingtonwells at 11.52am

    Absolutely – I wrote a post or two on sources of error. Here’s some.

    Error in polls and surveys*
    https://grumpollie.wordpress.com/2013/03/24/error-in-polls-and-surveys-2/

    What IS a pollster’s job anyway?
    https://grumpollie.wordpress.com/2014/04/03/what-is-a-pollsters-job-anyway/

    The number of NZ households with landlines, and the problems with phone surveys
    https://grumpollie.wordpress.com/2013/12/07/the-number-of-nz-households-with-landlines-and-the-problems-with-phone-surveys/

    Vote: Thumb up 1 Thumb down 0 You need to be logged in to vote
  13. johnwellingtonwells (127 comments) says:

    Alan
    The sample sizes are too small and the results unreliable to breakdown the results further. As mentioned in my previous posting, in some cases the results are weighted in order to correct differences in composition, but this leads to further problems. Also bear in mind that a larger household probably has a greater chance of there being someone at home to answer the phone, than a smaller household. Polling companies should be required to state the number of interviews attempted as well as the number of interviews achieved.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  14. Jimmy Russell (2 comments) says:

    It’s odd how people cling on to one poll and the next. All the polls combined show the everything to be stable. Unfortunately we don’t have a Nate Silver to create an algorithm to rate the reliability of each polls and provide better interpretation.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  15. Andrew (81 comments) says:

    Actually sampling approaches that select only one person per household have a bias toward people who live in smaller households.

    I complete agree that all surveys should report response statistics. The thing is, I’ve seen some very dodgy response rate calculations used for surveys in NZ, which inflate the true response rate. My view is, before any progress can be made in this regard, the industry needs to agree a standard response rate formula and set of call outcomes. Also they need to provide resources to help people know what things like response rates and refusal rates mean.

    Something to keep in mind though – response rates aren’t really relevant to quota surveys, that try to interview hard-to-reach people through targeting and replacement (not by targeting a high response rate).

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  16. publicwatchdog (2,225 comments) says:

    Polls … polls … whatever……show me ONE poll or political commentator who predicted that I’d poll 4th in the 2013 Auckland mayoralty with early 12000 votes? How
    many votes do Kiwi bloggers predict Ill get when I stand against John Key in Helensville? Penny Bright http://www.dodgyjohnhasgone.com

    Vote: Thumb up 0 Thumb down 4 You need to be logged in to vote
  17. big bruv (13,454 comments) says:

    ” How many votes do Kiwi bloggers predict Ill get when I stand against John Key in Helensville?”

    Fuck all is the number of votes you will get Penny. Not enough to unseat the PM and not enough to worry any of the other parties.

    Mind you, I think I might come out and have a gander at any public meeting that you and Key are speaking at, I would love to watch him wipe the floor with you.

    BTW, have you had the exterminators in yet?, cant have your house going up for no reserve Auction if it is infested with fleas can we.

    Vote: Thumb up 3 Thumb down 0 You need to be logged in to vote
  18. Nostradamus (3,079 comments) says:

    Reposted in the public interest:

    Nostradamus (2,847 comments) says:
    May 17th, 2014 at 3:38 pm

    Penny Not-So:

    In 2013 – as an Auckland Mayoral candidate, I campaigned against CORRUPT corporate control of the Auckland region, and got nearly 12,000 votes.

    Yes, you’ve made this claim a number of times now, but let’s put those “nearly 12,000 votes” in context.

    The election results are here.

    So what did I find when I put those results into Microsoft Excel?

    - You received 11,723 votes out of a total 352,656 votes – or 3.32% of the total vote.
    – Even if BLANK (7,147) and INFORMAL (1,584) votes are excluded from the analysis, you still only got 3.41% of the total vote.

    One could reasonably infer that the 96.68% of voters who didn’t vote for you want you to pay your rates! Now that’s something to think about next time you make this dodgy claim.

    And then we have your Return of Electoral Donations and Expenses form. It appears that you spent $42.78 on “4 cans spray ink”. Were those used to deface other candidates’ signs?

    My prediction: if you go up against John Key (an honourable man who pays his rates), the good people of Helensville will vote for him, not for you!

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  19. stuarts-burgers (101 comments) says:

    Just be contacted by Roy Morgan’s for a “short survey” first question was my age and when I told interviewer my age was thanked and told they have enough in that group but was asked if there was any body else in the house they could speak to.
    When told no the interviewer thanked me and asked if they could ring me back next month. I said yes so will await there call.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  20. Andrew (81 comments) says:

    Oh wow – really interesting!

    So definitely a quota survey, and they’ve flagged you by quota for their next wave. That’ll really help to keep costs down. I’ll need to think through the implications of each wave being non-independent.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote

Leave a Reply

You must be logged in to post a comment.