New Zealand Business Council for Sustainable Development

The Herald on Sunday has an article by Peter Neilson.  Peter, a former Labour MP, is the Chief Executive of the New Zealand Business Council for Sustainable Development.  He refers to polling in the article:

New polling shows some fascinating coalition and policy combinations are possible this year….

And among all voters, Green is the most-favoured coalition partner for National and Labour, so a betting person would say John Key will need to stitch up a deal with the Greens.

Now which polling is this? One presumes it is their inhouse polling done under the brand name ShapeNZ.  ShapeNZ is not an independent company, but is run directly by the Council.

Now this is in itself nothing wrong.  It can be very logical and cost efficient for an organisation to do its own market research.  But one does have to be very careful about referring to such research.  Neilson should have referred to the fact the research was (presumably) done by his own organisation, rather than just generically refer to “polling”.

ShapeNZ use online surveys to gather information.  Now again, this is not necessairly a bad thing.  Many market research companies use online surveys for certain clients.  However if the research you are doing needs to be a random representative sample of all adult NZers (such as for political polling) then online surveys can be problematic.

In the UK, You Gov, has managed to produce consistently reliable poll results using online samples. To be successful you need to build up a very large base of people who are willing to take part in your surveys.  For example if you need 1,000 responses for a survey, then you need to get tens of thousands in your database, so you can get enough responses each time, and so they are not always the same 1,000 people and also so you can check it is balanced by gender, age, geography, income etc etc.

You also need to provide prizes or points which can be redeemed for prizes as an incentive for people to take part.  Otherwise you only get responses from those motivated by the issue.  And already you have problems in getting a representative sample as very poor families may not have Internet access and very wealthy people are probably not very motivated to take part in a survey so they can gain a $50 prize or equivalent.

Now these are not impossible to overcome.  You Gov does it well.  But as far as I know, none of the big companies in NZ use online polling for published political polling – they all use phone polling which is still the industry standard. Some of them are doing some political polling over the Internet but are not publishing the results.  My assumption is they are either trialling the technology to see how it compares to their phone polling, or it is research for a private client.

So ShapeNZ’s use of online polling on political issues is not unheard of, even though it is rare.  But there is one major difference which I think undermines the reliability of their results.

Other polling companies recruit people to their online panels trough their phone and postal surveys, plus their websites.  They are already dealing with a fairly mixed and balanced group of NZers through their work for other clients.

However ShapeNZ works only the New Zealand Business Council for Sustainable Development, as they are part of it.  I suspect many of their online survey panel members have come from those who have been visiting their website.

And here is the problem – you probably have a self-selecting sample. Those who are interested in green, environmental and sustainable issues are not the same as the total population.  Hence it is no surprise that a majority of respondents would pick the Greens as the most favoured coalition partner for National.

Now ShapeNZ probably does its best to mitigate by, for example, balancing the sample based on party vote.  So they may have a sample which is 40% Labour, 38% National etc so they can say it is the same proportion as the last election.

The problem with this though, is that not all National voters are the same.  There are “green” National voters and (for example) National voters who think climate change is  vastly exaggerated.  And the ShapeNZ survey probably only captured “green” National voters as the other sort would probably never have been visiting their website.

So overall I would be very suspicious of the poll results Peter Neilsen refers to.  Of course this is based on assumptions because he hasn’t said where these results come from.  Ideally he should make the full results public so they can be judged in context, rather than just refer to “polling”.