Polls and Prediction Markets

February 26th, 2010 at 2:00 pm by David Farrar

I attended on Wednesday night the launch of “Key to Victory” which is the normal post election campaign review book edited by Stephen Levine and Nigel Roberts.

I find these books so fascinating, I was even reading it during the speeches!

Bryce Edwards has reviewed (h/t iPredict Blog) a chapter by and on what sources of information best predicted the election outcome. Was it an individual poll, the markets or the of .

The amount invested in iPredict was considerable:

  • $64,500 was traded over the likely nature of ‘the Maori Party’s post-election relationship with National’
  • $25,800 was traded over the Wellington Central battle between Grant Robertson and Stephen Franks
  • $132,100 was traded over whether ‘there will be a National prime minister after the 2008 election’
  • $413,000 in total was invested in election-related predictions

And how did iPredict do”

So, how accurate was iPredict in 2008? McGirr and Salmond conclude that although iPredict overestimated the eventual support for both Labour and National, it was more accurate any individual polling company.

And the individual polls:

In reality in 2008, McGirr and Salmond found this to be the case – with Colmar Brunton and DigiPoll exaggerating public support for National, and Roy Morgan exaggerating support for Labour (p.264).

So which polling companies were most accurate and inaccurate? McGirr and Salmond say that TV3’s TNS poll was the best (as it was in 2005 as well), and Fairfax’s Neilson pool was the poorest.

The TV3 poll is the one that currently shows a 27% gap! Mind you they are now with Reid Research, so there may be a different methodology used now.

Then they look at the polls of polls published by three outlets – NZPA, Rob (at 08 Wire) and myself (at curiablog).

In addition to the five opinion polls, some observers attempted to average out the idiosyncratic errors of the individual polls by aggregating them into a “poll-of-polls” using different methods. The New Zealand Press Association simply took the average of the estimates of the six most recent polls, while The New Zealand Herald took the average of the last four polls. Two blog-based polls-of-polls – one run by David Farrar of New Zealand’s premier political blog Kiwiblog, and one hosted at a smaller blog [run by author Rob Salmond] called 08wire – weighted more recent polls with larger sample sizes more heavily (p.257).

And how did the poll of polls do?

McGirr and Salmond say that ‘Poll-of-polls consistently performed well during the 2008 campaign, outperforming most of the opinion polls and the prediction markets’ (p.270). They therefore advocate that both the media and public should pay much more attention to this highly accurate source of political information.

Tis has prompted me to update the poll of polls widget, which is below.

Salmond ranks the different outlets for their accuracy to the final result. In order they were:

  1. NZ Herald poll of polls 6.1 (error from result)
  2. NZPA poll of polls 6.8
  3. Curiablog poll of polls 8.1
  4. TV3/TNS poll 9.6
  5. 08 wire poll of polls 13.6
  6. iPredict 15.7
  7. TVNZ/Colmar Brunton poll 16.8
  8. NZ Herald/Digipoll poll 19.8
  9. Roy Morgan poll 20.8
  10. Fairfax/Neilsen poll 29.6
  11. NZ Political Stockmarket 109.5

The NZ Political Stockmarket used virtual money, so it shows what a difference real money can make.

The authors conclude that media outlets should not just report the individual poll results when they commission a poll, but also publish regular info on a poll of polls and on iPredict.

Incidentally I will probably review and tweak the curiablog methodology a bit when I have some spare time.

Tags: , , , , ,

13 Responses to “Polls and Prediction Markets”

  1. Mr Nobody NZ (391 comments) says:

    Does anybody else find the Yellow label for Act difficult to read?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  2. Repton (769 comments) says:

    It would be interesting to look at the demographic of iPredict users and compare that with the error in the results.

    Incidentally DPF, your weighted poll JPG looks pretty bad — lots of JPG fuzzing. Why don’t you use PNG instead?

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  3. stephen (4,063 comments) says:

    “Does anybody else find the Yellow label for Act difficult to read?”

    Painfully so.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  4. kevin_mcm (152 comments) says:

    given their performance does it really matter?? :)

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  5. PaulL (5,987 comments) says:

    How did 08 wire get so different a result from yourself and the Herald? Wishful thinking, deliberate statistical manipulation, or just a methodology error? Seems that there are only so many ways to calculate an average.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  6. Repton (769 comments) says:

    How did 08 wire get so different a result from yourself and the Herald? Wishful thinking, deliberate statistical manipulation, or just a methodology error? Seems that there are only so many ways to calculate an average.

    Let’s say I survey 10 people and discover that National is on 40% support.

    And you survey 1000 people and discover that National is on 55% support.

    If I sum the two scores and divide by two (i.e. average the results), I get 47.5%. Would you feel that that is more accurate than your result alone?

    Now throw in another poll of 800 people taken two weeks earlier… Combining these results isn’t as straightforward as you might think.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  7. Dirty Rat (504 comments) says:

    stephen @ 2.25pm

    your post is soooooo ironic

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  8. Dirty Rat (504 comments) says:

    554 out of 1010 is more than 47.5% (providing they hadnt polled twice)

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  9. RKBee (1,344 comments) says:

    Which ever way you look at the National states… they could poke their finger at us take the money and still win.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  10. RKBee (1,344 comments) says:

    Stats sorry .. Labour on the other hand could offer us all the money in world and a big hug and still wouldn’t win on their stats. Really whats the point of the other smaller parties other than obtaining money under false pretences.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  11. vibenna (305 comments) says:

    Lots of interesting stuff on election forecasting methods at this link: http://pollyvote.forecastingprinciples.com/

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  12. Anthony (796 comments) says:

    Stephen I agree – the yellow for Act is painful as is their poll rating. Act always rates low between elections though.

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote
  13. expat (4,050 comments) says:

    you need to put your rates up David ;)

    Vote: Thumb up 0 Thumb down 0 You need to be logged in to vote