Trusting polls

Here’s a little guide to trusting polls, which media and readers may find useful, when they see other media trumpet a poll they have commissioned. This is not a comprehensive guide – that could fill up many scores of pages.

1. Is the poll a random selection poll (normally done via phone or in person) or is it an Internet panel poll (less reliable) or a website poll (of almost no reliability)
2. If the poll is a random selection poll, then how big is the sample size. Note this is less important with other polls.
3. Has the company got an accurate track record of polling? How many years have they been polling for? What did they predict for the last election? Have they not only predicted results (who wins) which can be fairly easy but also got the actual percentage vote gained correct?
4. Is the result they are proclaiming in line with any other public polling company? If during the last couple of months four other companies have all independently said Party X is at around Y%, then how likely is it every other company is wrong, and this poll which says they are much higher (or lower) is correct?
5. Consider also if the other polling companies have a long history of public polling, and were very accurate at the last election, but this new poll with incredibly different results, is done by a company that did not exist at the last election? Does that make it even easier?
6. Finally consider then whether giving undue prominence to such a poll, is reporting the news, or inventing the news?

Some media take this stuff very seriously. I know in the past that the TV networks have had polls re-done, and methodologies audited, when they have had a poll result totally at odds with all the other public polls. In fact there have even been times when a poll may not even be published, because the broadcaster sees it as so far out of line with other polls, and indeed upon redoing the poll, a more credible result has occured (remember 1 out of 20 polls will be outside the normal sampling margin of error).

Other media seem content to trumpet highly suspect results – results unmatched by an other company or outlet, as exclusive front page leads. I suppose they have to, as you are loath not to use something you have paid for as a story. But other media, should be very cautious, especially in election year, as to how they report such polls, especially when they are Internet panel polls, not random selection phone or in person polls.