More on the Herald so called poll of polls
On the 2nd of October I wrote:
So the last six polls, all from different firms, have had ACT ahead of Opportunity by margins ranging from 1% to 6%, and the Herald-MOTU poll of polls says that the poll of polls means Opportunity is ahead of ACT.
On the 8th of October the Herald did an explainer, and have made some changes. Much of their explanation has nothing to do with why they had Opportunity above ACT, when every single pollster had them 1% to 6% behind. But there were some nuggets of useful info:
Well, most importantly, it means the Poll of polls is not a polling average.
Then don’t call it a poll of polls!!!
Crucially, the Poll of polls doesn’t just take polled values as is. Instead, it considers how polling values have changed in the recent past, which it uses to predict how voter sentiment may have evolved since the polls were actually conducted. The model’s predictions are then updated as new polls are included.
This seems to be saying that if a party has been going up in the polls, or down in the polls, then they predict that trend will continue. Now I’ve been following polling aggregators and forecasts for over 15 years, since Nate Silver made them famous (and prior to that Real Clear Politics). This is a very very rare practice, and would normally only occur in an environment where you have hundreds or thousands of polls, as inputs to a model.
Certainly some forecasters such as Nate Silver do include non polling information such as economic data, and even expert predictions. But they make it crystal clear that they are doing so, and in Silver’s case they provide different outputs (lite, classic, deluxe) so readers can pick between the simple polling average and that which incorporates other data.
Could you imagine what would happen if an actual polling company said “Hey National got 32% in this poll, but as they were at 30% in the last poll, they have momentum, so we are going to report they are at 34%!”
It is worth reflecting on what the actual polls since the Herald poll of polls had Opportunity ahead of ACT have said. They are:
- Roy Morgan: ACT 4.5% ahead
- Reid Research: ACT 3.5% ahead
- Verian: Act 2.0% ahead
- Curia: ACT 4.4% ahead
So if the momentum formula was a useful thing to do, then you would have expected the next set of polls to indeed show Opportunity ahead. But none of them do – the last 10 polls all show ACT ahead, but the herald poll of polls still shows Opportunity ahead of ACT.
This is not a trivial issue. Millions of people may have seen this forecast and thought it accurately reflected the state of the race.
The Herald will continue to use predictions of the outcome of a “hypothetical election held today” as the primary output of the Poll of polls. This is, after all, what we are most interested in.
However, to make it easier to understand what the model is doing, we will also publish two additional estimates. First, we will publish the model’s estimate of the polling average today to help readers understand the magnitude of pollster effects.
Second, we will publish the model’s estimate of the polling average one month ago to help readers understand the model’s sense of underlying momentum. It will also enable readers to check the model’s estimates against the most recent five or six polls.
Extra data is good, as is the explanation. But what the Herald should have done is made clear from the beginning that they were not just taking poll results and weighting them for size, recency or house effects, but also adding in an assumption about momentum (that is very dubious).
What has also been unacceptable is the refusal to actually release the formula being used, so people can judge for themselves. I do a simple polling average at Curia, and the methodology is published here.
If it is necessary to modify the underlying model to more accurately reflect voter sentiment, then we will make changes and explain the reasoning that prompted them to readers. However, at present we do not believe changes are necessary.
You’ve still got a “poll of polls” claiming Opportunity is ahead of ACT, despite the last 10 polls saying the opposite. I think changes were necessary some time ago.
Finally a note of caution. As the election gets closer, the Herald’s model will no doubt reduce the adjustments it makes for stuff like momentum, so that the final pre-election forecast is basically just a polling average (with house effects adjusted for). It is quite possible, even probably, that will be accurate. Polling averages done well, are often superior to any individual poll.
But that doesn’t mean their “poll of polls” six weeks out is at all accurate, or useful. In fact it is highly misleading. Readers should not have to read a 10 page article to understand what it is and isn’t. Do not call it a poll of polls when by your own words “it is not a polling average”.