Social Desirability and Response Validity in Current Polling

Several days ago in these very pages, discussion ensued regarding the latest conservative attempt to rewrite reality through re-weighting polls to one guy’s liking.  Of course, polling is not an exact science, but it is a science, and the latest wingnut delusion has no grounding in theory or empirical evidence.  Like any science, survey research continuously attempts to improve upon the validity and reliability of its measures and findings.  While I’m not at all concerned about some nefarious (and successful) attempt by the MSM and that paragon of power, the Democratic Party, to turn otherwise professional and reputable polling houses into duplicitous shills.

However, I have been somewhat interested (note, not concerned) if there might be something else going on that causes the polls to over estimate support for Obama.  Social desirability bias is something I’ve published on in the past (direct link to the paper here).  While that article suggests a contextual effect that causes variance in social desirability across countries (regarding accurately reported turnout in survey research), relevant here is what is colloquially known as the Bradley Effect.  It’s possible (though I consider it unlikely in the specific context of the 2012 Presidential election) that this helps explain Obama’s consistent polling advantage in an election where many if not most structural conditions suggest an incumbent defeat.

It’s difficult with the data I have available to examine this hypothesis to any satisfaction, but that’s not going to stop me from trying.

To begin with, we have the current state of the polls.

RCP’s running average has Obama up 4 points, Nate Silver’s “nowcast” model up 5.7, and his current prediction for election day 4.1 points.

If social desirability is at work here, a poll respondent will state that she or he supports the President because internally, our not entirely sincere respondent is seeking the socially desirable response, and not supporting the black guy might be racist.  However, this is done knowing that they will ultimately support the white guy.  Practically, this would mean that Obama’s support in these polls is inflated.

I’m approaching this from several directions.  First, I’ve averaged the final month of polls for Presidential elections going back to 1976 (an arbitrary cut off) to examine how accurate the polls were in predicting the final outcome between two white men, with 2008 to serve as a benchmark for 2012.  Shift represents how wide of the mark the final polls were, to the benefit or detriment of the incumbent party.

Social desirability response bias in an election can take on many forms, not just race.  To wit, the 1992 general election in the United Kingdom is a good case study (one I lectured on here at Plymouth about six or seven years ago, shame I have no clue where those lecture notes now reside) as the polls largely predicted a narrow Labour outright victory or a hung parliament with Labour having the plurality of seats, yet the Conservatives under John Major easily won by 7.5%.  This is called the Shy Tory Factor on this island, which is simply another manifestation of social desirability.  But in 2012, I’m primarily considering race, and comparing 2008 to past elections does not support the hypothesis that this might be a problem for Obama’s numbers in 2012.

I also considered several of the primary elections in January, 2008.  This was the beginning of a primary where Obama was a somewhat unknown junior senator only four years into his Congressional career, going up against the assumed nominee.  For these data, I average all polls from the last week of the campaign in the given state (there were 20 in New Hampshire alone).

This evidence is more ambiguous than the examination of previous general elections.  Both Clinton and Obama received shifts in their favor, which isn’t surprising considering the undecideds presumably made a decision of some sort once voting.  However, in both New Hampshire and Nevada, the shift was stronger towards Clinton than Obama: a 7 and 2.25 point advantage respectively.  Both are dwarfed by Obama’s advantage in South Carolina.

These are the wrong data to be analysing this with, of course; ideally we’d have individual level data.  While not individual level data, the following figure, by Greenwald and Albertson, offers a more holistic view of the 2008 primaries.

The above shows that among 32 states where data were available, the “Bradley effect” was only evident in three states, yet 12 states demonstrated what has been termed (erroneously, in my opinion) the “reverse Bradley effect”: states where Obama’s support in the primaries was under, not over, estimated (see South Carolina above).  I consider this an erroneous classification because where the theoretical explanation for the Bradley effect hinges on social desirability, the reverse has been hypothesized as a function of systematic sample bias, through either the under-representation of African Americans in polling samples, or the cell-phone effect.  However, some have hypothesized that “black voters might have been reluctant to declare to pollsters their support for Obama”, and the link above does discuss that

After the Super Tuesday elections of February 5, 2008, political science researchers from the University of Washington found trends suggesting the possibility that with regard to Obama, the effect’s presence or absence may be dependent on the percentage of the electorate that is black. The researchers noted that to that point in the election season, opinion polls taken just prior to an election tended to overestimate Obama in states with a black population below eight percent, to track him within the polls’ margins of error in states with a black population between ten and twenty percent, and to underestimate him in states with a black population exceeding twenty-five percent. The first finding suggested the possibility of the Bradley effect, while the last finding suggested the possibility of a “reverse” Bradley effect in which black voters might have been reluctant to declare to pollsters their support for Obama or are under polled.

There are numerous possible explanations for the “reverse” effect, including faulty likely voter models, under sampling of blacks, sampling bias due to cell phones, to name a few.  There might be some sort of contextual effect at work here, but to ascribe it to behavioural motivations (rather than factors exogenous to the individual, such as sampling bias) such as blacks being shy to state their support for Obama fails the face validity test to me.

Ultimately, given the wide array of mediocre data presented here, I am not concerned about social desirability biasing the estimates of support for Obama in any significant, substantive manner.  However, much as I’d like to, I wouldn’t say that the Republican conspiracy is more likely, if only because that is so creatively ludicrous I initially thought it was an Onion piece.