Did the Judge understand margins of error?
August 11th, 2005 at 4:19 pm by David FarrarI was concerned when I heard the arguments in court yesterday that the TV3 poll had a margin of error of 3.2%, and that this is greater than the difference between the result of 1.6%, 1.4% and 0.4% (being ACT, United and Progressive respectively).
I was even more concerned that the Judge said “It [TV3] had based its decision on a single opinion poll — which expert testimony had said had a margin of error greater than the margins between the smaller parties.”
You see the fundamental mistake is that the 3.2% margin of error is a maximum one which applies for a sample of 1,000 for a result of 50%. It means with 95% confidence the true result is between 46.8% and 53.2%. But if the result is less than 50%, then the margin of error shrinks. This is crucial.
The actual margin of error, and 95% confidence interval for each party is:
ACT 1.6% = 0.8% moe = range of 0.8% to 2.4%
United 1.4% = 0.7% moe = range of 0.7% to 2.1%
Progressive 0.4% = 0.4% moe = range of 0.0% to 0.8%
Now one can also calculate the probability that one party is actually ahead of another party. There is a calculator (excel) linked here.
The probability ACT is ahead of United Future is 64.3%. The probability ACT is ahead of Progressive is 99.7% and probability United is ahead of Progressive is 99.1%.
Therefore any decision to allow Progressive into the debate, on the basis of the margin of error, is highly flawed. There is only a 0.3% chance (at 95% confidence) that they have more support than ACT.
I look forward to seeing the court record (if there is a way to see it) to find out whether it was made clear to the Judge that the 3.2% margin of error quoted had not relevance to the issue. The relevant margins of error range from 0.4% to 0.8%.
August 11th, 2005 at 4:34 pm
Thanks for that clarification. I have to admit I hadn’t even thought of this, despite once battling through Bursary Stats.
Vote:August 11th, 2005 at 5:11 pm
interesting, you expect this level of mathematical illiteracy from the District Court, not the High Court. Another judge to put on the must go to list if ever involved in defending litigation on financial derivatives?
just need to put on a smart suit and talk lots about “fairness” bound to get off.
Vote:August 11th, 2005 at 5:52 pm
You are right of course David. The winner on the day was the UF/Progressive legal team’s litigation tactics. The chance for reasoned analysis and good evidence is limited with proceedings commenced at such short notice, and the result shows it. I guess I can understand the predicament Young J was in. I hope people judge Dunne poorly for this. I didn’t expect much better from Anderton, but I would have thought Dunne to be above this sort of lark.
Vote:August 11th, 2005 at 5:57 pm
I was getting silently frustrated sitting in the back of the court when the discussion of margins of error was occuring; the discussion centred around, as you note, expert testimony that said the parties were within the margin of error.
Now whilst the Judge, and the lawyers, were calling the difference statistically insignificant in oral discussion (and that was wrong) in the decision itself the reference was to the expert testimony.
And I do note that, whilst may it not mean a great deal, the ranges at the 95% confidence level of 0.7-2.1 and 0.8-2.4 have substantial overlap.
Vote:August 11th, 2005 at 6:03 pm
Who was the “”expert”" on the margin of error?
Vote:Plus remember its an injuction so the exhaustive analysis doesnt apply.
No doubt TV3 wont use an opinion poll to sort the wheat from the chaff again
August 11th, 2005 at 6:25 pm
Thank you David! Great explanation. Yay, someone else who understands what “margin of error” actually means! It’s one of my pet peeves that most journos don’t understand/misuse the phrase “margin of error” every time they talk about the smaller parties.
Vote:August 11th, 2005 at 9:36 pm
pertinax/DPF,
the expert evidence advanced on behalf of the plaintiffs was given (by affidavit) by Professor Stephen Levine of Victoria University’s politics programme. He is a highly regarded and frequently published authority on New Zealand elections and polling.
Vote:August 12th, 2005 at 8:03 am
You know more about this than me, but I’m assuming that is just the error that may occur because the people sampled aren’t exactly representative of those who were available to be sampled.
The fact that those available to be sampled may not represent all voters introduces a separate and addditional error. And it’s an error that could heavily influence the accuracy of the smaller party numbers.
Vote:August 12th, 2005 at 8:33 am
Oh there are many errors that can affect a poll, outside the statistical sampling error. And frankly deciding on the basis of a single poll is a silly thing to do.
But what concerns me is that the decision references the margin of error issue, and it is not clear that it was made known to the Judge that this is smaller for results less than 50%.
Vote:August 12th, 2005 at 10:43 am
The bottom line as many pollies have said is the only poll that matters is on election day Everything up to then is speculation Look how the Socialists and the Nats and NZF have bounced around over the past few weeks If any one of those polls was election day then the government would be different from the result of another of the polls.Dont get too excited theres a lot more water to flow under the bridge until the 17th Sept
Vote:August 12th, 2005 at 12:59 pm
Ahh. Blessed P values
Vote:August 12th, 2005 at 1:32 pm
Statistical Significance:
Thanks for the informative piece on ‘margins of error’. Didn’t realise that they weren’t margins, nor describe errors.
Am intrigued to know what the appropriate statistical approach is to exclude fringe-dwellers. Something along the lines of a multi-value exclusive hypothesis test with the associated p-values? Well out of my depth here.
If I run for parliament under the banner of the ‘Politicians who understand Statistics Party’, and one of the 1000 people who gets polled is my Mum (who says she’ll vote for me), what statistical measure could be called upon to stop me from featuring on the next TV3 debate?
-Paul
Vote:August 12th, 2005 at 2:45 pm
Margins of error are not the only things in the technical and scientific world that are being misunderstood in media and more widely, and even by judges as in this case.
But then I would say that, wouldn’t I: I run Science on the Rocks, a small website interested in science and reason, and science and culture.
Vote: