Thoughts on the Scottish Independence Referendum II: The Polls

As I indicated yesterday, polling on the Scottish independence referendum has tightened significantly in the past couple of weeks, perhaps best illustrated here. Not surprisingly, I’ve been asked to do a bit of local media on this issue over the past few weeks, and yesterday on air I was asked the inevitable “who will win?” question. I suggested that the current polling is overstating the estimate of the yes support, and that in the end I’m expecting at least a four point (i.e. 52% No, 48% Yes) victory for the unionists. While interpreting the accuracy of polling data in this context is atypically difficult (which could be interpreted as a shifty, thinly veiled move to hedge my bets), we do have some theoretical and empirical guidance.

There are at least three variables to consider when evaluating the accuracy of these polling data. First is the effect of turnout on the accuracy of the likely voter models employed by the various polling houses. As discussed here at UK Polling Report, estimates of likely voters based on previous elections are of less help in this referendum than for a typical election., and additionally, turnout is expected to be impressive:

A stunning 97% of the electorate has registered to vote in the referendum, meaning turnout is expected to be very high, which could also delay the vote. There will be a lot of ballots to count. Turnout in Scotland at the 2010 general election was 64%.

To add some empirical lustre to this anecdotal extrapolation (again, from UK Polling Report):

Polls aren’t very good at predicting an actual percentage for turnout – people overestimate their likelihood to vote, and the actual turnout figures they are compared to are a bit ropey because of inaccuracy and incompleteness of electoral registers – that aside, they are pretty good at predicting relative turnout, and the referendum looks set to have a much higher turnout than any recent election.

If turnout is impressively high, the negative effect of variations in the reliability of likely voter models is somewhat attenuated. Likewise, expectations that we ordinarily would have based on variance in turnout, such as a lower turnout would mean a disproportionately older and wealthier electorate, are of lesser significance. Furthermore, as I discussed yesterday, the eligible voter pool for the referendum is, well, strange:

First, the eligible electorate is an interesting question, with the primary criterion being residence in Scotland.  Any British citizen resident in Scotland can vote, as well as residents of Commonwealth countries (with “indefinite leave to remain” in the UK, which is the British version of the green card), EU citizens resident in Scotland, and a few others. As a student of turnout, the voting age has been lowered to 16 for the referendum, which is intriguing. However, while a French (or German, or Polish, or Lithuanian, or Jamaican, or Canadian) citizen resident in Scotland is eligible to vote in the referendum and help determine whether or not Scotland becomes an independent country, Scots living in England, Wales, Northern Ireland, or New Zealand can not.

This is not an ordinary Westminster or Holyrood electorate, so extant likely voter models have to take this into account, yet there’s limited empirical experience to draw upon. This is not to say that we’re flying blind, because we’re not, but I expect polling data to be less reliable than it is for a Westminster, US Presidential, or US Congressional election.  Yet, as support for independence is stronger among the young, any decline in turnout from these super high predictions will asymmetrically reduce the Yes vote, as the young will stay at home at a higher rate than older cohorts.

Second is the social desirability factor, or to put it in British parlance, a variation on the “shy Tory” voter, what they also call “differential response”. Briefly, on average people prefer to offer what they perceive to be a socially desirable response to a polling question. The guess, given the enthusiasm and emotional appeal to nationalism and patriotism displayed by the Yes campaign, is that should social desirability have an effect in these polls, it would be a “shy No” supporter. In other words, the perception by any given respondent would be that the desirable response is pro independence, so the suspicion is that there’s a marginal, but significant, percentage of the pro Independence estimate that in reality will vote no on the day. I think this might be overstated a bit, however. The effect of social desirability is directly related to the form of the polling methodology. Not surprisingly, face-to-face interviews have the strongest impact on social desirability, as when one is confronted with a real live human being right in front of you, the desire to give the acceptable response is considerably stronger than in other polling settings. As the majority of the polls in the field are internet based, where the respondent is speaking neither to a physical presence nor a human voice at the end of a telephone line, this effect should be mitigated to a certain extent. Intriguingly, yesterday’s piece in UK Polling Report suggests a slightly different take on this phenomenon:

I think there’s more risk from the other side of the same coin – “enthusiastic yesses”. It is very clear from activity online and reported campaigning activity that YES supporters are more enthusiastic, what if that is also reflected in responses to opinion polls? What if the yes supporter, full of zeal and keen to share their view, happily agrees to do the phone interview while the less enthused No supporter doen’t want to interupt their tea? Eagerly clicks on the email when the No voter doesn’t bother?

Ultimately, while I think that social desirability could be overstated in this context, if it does have a role to play at the margins, it will be to over-estimate pro-independence support.

Additionally, while we’re in somewhat uncharted territory with the unique composition of this electorate and the difficulty in constructing likely voter models, we do have some historical precedence to draw upon, as discussed on Monday by Stephen Fisher here, with the takeaway:

So overall the evidence is mixed, but not balanced. It seems more likely that the headline poll figures are over- rather than under-estimating the vote for Scottish independence – and that this might be especially true of the final polls published between now and polling day.

Finally, Fisher highlights something that has been downplayed in both the media and in polling aggregators: the interpretation of the don’t knows:

The tendency for final polls to differ from the actual result does not necessarily mean that referendum polls are biased towards Yes responses. It might be that the Don’t Knows split disproportionately towards No, that those in favour of the proposition tend to be less likely to turnout to vote, while late swing is also a possibility. Whatever the reason, the experience of referendum polls in the UK and internationally suggests that the findings of final polls (from which the Don’t Knows have been removed) are typically flattering for the Yes camp.

We do have empirical evidence to make some reasoned, if imprecise, estimates regarding the don’t knows. As the ICM poll released yesterday still reports 14% Don’t Knows, this remains a significant chunk of the potential electorate. The literature on direct democracy, specifically referenda and initiatives in the United States (the literature about which I’m most familiar), suggests that in a yes / no dichotomous decision, the No option has some of the advantages of incumbency. I strongly suspect that of the DKs that do turn out to vote, they will break significantly to No. This makes sense. Given this is the most important and far reaching election in Scotland in a lifetime, if a voter has yet to make up their mind 48 to 96 hours before the election, the odds of them sticking with the safety of the status quo rather than the riskier unknown of independence is compelling. People tend to attempt a minimisation of maximum regret. Information about the status quo, even with the promises of “devolution max”, is readily available. Information about how an independent Scotland will operate, including basics such as the currency, the status within the European Union, the armed forces, and uncertainty of the disposition of companies currently based in Scotland is, at best, murky as hell. Hence, it’s safe to assume that those DKs that do vote will significantly favor the No side.

Considering the weight of the above, it’s safe to suggest that the extant polling data is overestimating support for Scottish independence.