Herald repeats misinformation
James Dann is a Christchurch local body politician and blogs on the Christchurch rebuild etc. His latest blog post is on the Nielsen survey for CERA which has upset a few people because in fact it found not everyone was unhappy.
James has said the survey is unrepresentative:
The key to this survey is in the methodology. It was undertaken between August 29 and October 15 last year. 2381 people responded to the survey. 1156 of them were from Christchurch City, 618 from Selwyn District and 607 from Waimakariri District. So that means that of the people who did respond, more than half (1225) AREN’T ACTUALLY FROM CHRISTCHURCH. I mean no disrespect to the people of Rolleston, Lincoln, Rangiora, Kaiapoi etc. I am sure they have had a hard time. But they don’t have the same issues that people who actually live in the city do. They haven’t had to deal with a lack of services, portaloos, red zoning, TC3 land, roadworks to the same extent that people who live in the city do. …
Yet the survey then merges these three population groups, and continues to extrapolate from the combined population for another 100 pages. If you get to the end of the report, you’ll find the populations of Christchurch, Selwyn and Waimakariri. They are 289,000, 30,000 and 36,000 respectively. More than 80% of the adult population lives in Christchurch, and yet more than 50% of respondents come from less than 20% of the population, a section of the population that the survey shows have a more positive response in this survey.
Lies, damn lies, and statistics.
Now the Herald has gone and reprinted his blog post in the Herald.
James has made a fatal mistake. He has not read the survey results.
He is correct that the raw sample is not reflective of the overall population. That 80% live in Christchurch but only half the responses were from Christchurch people.
But what he has either deliberately or accidentally omitted, is that the results are weighted so that the results do precisely match the area, gender and age of the local population. Nielsen say in their report:
The sample design over-sampled residents of the two districts with smaller populations to ensure that the sample size within each district was sufficient to allow reliable and robust analysis.
At the analysis stage, the data was adjusted by a process called weighting. This process adjusts for discrepancies between the profile of people who completed the survey and the known profile of residents of greater Christchurch.
Population statistics are obtained from Statistics New Zealand data and is based on the latest population projections.
Weighting increases the influence of some observations and reduces the influence of others. So, for example, while 618 or 26% of completed interviews came from Selwyn District, the population of Selwyn actually represents about 8% of greater Christchurch. Thus, the data was adjusted so that 8% of any ‘greater Christchurch’ result reported is based on the responses of Selwyn residents.
So the criticism of Dann are entirely invalid, and worse the Herald has repeated them on their website. This is not a matter of opinion. His failure to mention that the results are weighted by area totally invalidate his criticism. The Herald was aware of this also, as one of their editorial staff tweeted his blog post and I pointed out this failing in a response (which they retweeted).
Appendix 5 of the poll results on page 103 details their weighting matrix with precise quotas for area, gender and age. It is a model of transparency.
Incidentally how significant were the differences in responses from those in Christchurch itself, and the two neighbouring districts? Here’s the figures for overall quality of life – good vs poor.
- Christchurch City 72% good vs 7% poor
- Selwyn 85% good vs 4% poor
- Waimakariri 82% good vs 4% poor
- Greater Christchurch 74% good vs 7% poor
So the results for Greater Christchurch are almost identical to Christchurch City.
The only question I have is whether Mr Dann knew the survey was weighted and decided not to mention that fact, or whether he just failed to read the results in full.
UPDATE: He has amended his original blog post to now mention the weighting, but note that the version picked up by the Herald doesn’t include this.
UPDATE 2: Dann was a Labour Party candidate at the last local body elections. Yet the Herald didn’t think to mention this. Shocking.