Notes on a Couple of Polls: Pew and RAND

First, let’s discuss the new Pew release, and attempt to do so without freaking out.  It presents grim reading for at least two reasons: the four point Romney lead, and that Pew has had a consistent Democratic lean, relative to the aggregate averages, this entire cycle.  Indeed, Silver notes

The Pew poll, however, may well be the single best polling result that Mr. Romney has seen all year. It comes from a strong polling firm, and had a reasonably large sample size. Just as important is the trendline. Pew’s polls have been Democratic-leaning relative to the consensus this year; its last poll, for instance, had Mr. Obama 8 points ahead among likely voters. So this represents a very sharp reversal.

As it’s important to not fall into the trap of focusing on confirmatory evidence to the exclusion of equally valid contrary evidence, it’s likewise not healthy to overly concentrate on one negative national poll to the exclusion of the weight of extant evidence.  In other words, I didn’t start doing back flips following the September Pew release showing an eight point Obama lead, and I’m not marching to the liquor store to buy out their stock of whisky because of this one.

There are legitimate, and less legitimate reasons to be weary of this one poll; John Sides does a solid job of pointing out some examples of both.  One reason attracting attention is the underlying partisan identification of the sample.  The most recent, October Pew sample had 31% Democrats to 36% Republicans.  Therefore, at best, Republicans were over-sampled accidentally; at worse, purposefully, or so goes the narrative.

A comparison to the September (12-16) Pew release is illustrative here.  The underlying partisan composition of that sample was 39% D to 29% R.  Does this mean that there has been a swing of 15 points in the underlying partisan composition of the American electorate from a ten point Democratic lead to a five point Republican lead in three weeks?  Of course not.  There are (at least) two dynamics at work here.  First, partisan identification is not a static attribute.  It’s a dynamic attitudinal measure that can be affected by context, and following the debate, it’s possible to conceive that claiming a Republican allegiance is marginally more attractive than it was prior to the debate.  Second, polls vary, even from the same polling houses, for random reasons.  Again, Sides: “Poll results vary for random reasons—that is, because of sampling error.”  Indeed, considering both the extreme (in comparison to other polls) top lines, and equally extreme shift in both the top line and underlying partisan composition of the sample, I’d suspect that both samples were unlucky.  In other words, while October is possibly an over-statement of Romney’s support in October, likewise September was possibly an over statement of Obama’s support.

There’s a second comparison to make: the LV N of the September sample was 2,192.  October, 1,112.  This does not make the former automatically better than the latter, but assuming everything else in their sampling methodology remained the same, the error bands in October are larger.  Meaning, we can be less confident in the validity of the four point lead in October than we could in the eight point lead for Obama in September.  The MoE of October for the LV figures is +/- 3.4%, meaning that there is a 95% chance that the “true” figure for Romney’s support lies within a band of 45.6% to 52.4%.  While this simplifies matters, and I feel that MoE can be overstated, it is illustrative of the random error that naturally exists even in a perfectly drawn probability sample.

An intriguing alternative to the Pew release is the ongoing panel survey at RAND.  Wang discussed this survey yesterday, suggesting that the data show that the Romney debate bounce resulted from inspiring the base and not from converting undecideds or siphoning directly off of Obama’s support.  This is notable because of empirical evidence that suggests viewers largely watch the debate to cheer their team and have their own underlying decision confirmed, yet is surprising because Romney gave a very non-base performance highlighted by a rapid shift to the center.

Before looking at this a little closer, there are some strengths, and weaknesses, to the RAND methodology to discuss.  The notable strength of this survey is that it is drawn on a panel, not a cross section.  They’re “interviewing” the same 3500 respondents every week (500 each day so the daily result is a composite rolling average).  In terms of research design, this puts us in a much stronger position to make causal inferences.  If we consider the debate performance a temporal “treatment”, we’re on safer ground suggesting that “the debate caused X” than we would be comparing a bunch of cross sections.

There are several weaknesses specific to using a panel, however, but in my estimation these weaknesses pale in comparison to the inherent strength of the design.  First, any initial errors in drawing the sample are locked in.  This can be, and is, partially offset through weighting, but it is something to consider.  Second, there’s the issue of sample attrition.  Finally, something I just considered today, it’s possible that respondents might feel internalised pressure to stick with their initial decisions.  An example might be a hypothetical respondent consistently answering with “Obama” from early September to the present, but internally is starting to lean Romney, yet doesn’t want to publicly admit this, as he or she does not want to appear fickle or superficial or inconsistent (or choose any of a number of possible awkward adjectives here).

There’s also a minor weakness, in my assessment, of their LV model.  This said, there are at minimum minor weaknesses in any LV model, so this should not be interpreted as a critique of this instrument in isolation.  One of the three items this survey asks is “What is the percent chance that you will vote in the Presidential election?”.  This is self reported, which suffers from questions of both validity and reliability.  Social desirability gets to the theoretical heart of the matter in both.  People will tend to over-estimate the chances that they will vote, because it’s the socially desirable thing to do, and before an election a respondent will feel this both externally and internally.  The former, they’ll feel that whoever is reading and coding their responses will look down upon them for claiming only a 50% chance of voting, so will inflate the number.  The latter, sure, they believe that they’re going to vote, it’s the right thing to do after all.

Thus, these figures are going to be inflated.  However, in terms of reliability, social desirability effects different groups of people differently, in systematically predictable ways.  Individually, those of socio-economic categories more predisposed to participate to begin with will be more influenced by their own perceptions of the socially desirable response (both internally and externally).  Basically, Romney supporters should be more susceptible to this phenomenon than Obama supporters.  This should present a silver lining to most of us here: we should discount the self reported figures for Romney supporters more than for Obama supporters.  Likewise, electoral context with higher levels of turnout over time will likewise be more susceptible to social desirability bias.

That said, they weight for this as well based on national participation figures from 2008, which might explain why the current estimates show only a 1.6% gap between Romney and Obama supporters probability of voting.  I’d expect this estimate to be larger.

Below are the rather precise daily estimates from RAND since the October 3 debate.  Inferring from these numbers, Romney’s bounce was in the neighborhood of 1.15% in his own support, 2% in the spread, and this has rather quickly stabilized.  Of course, this is out of line with what most are reporting.  That said, the October 8 rolling average only represents the updating of 2500 out of the 3500 overall sample, so it’s possible that Obama’s numbers could still deteriorate further.  Additionally, the debate doesn’t appear to have had a big impact on the number of undecided respondents: on October 4 undecideds were 6.15%, yesterday 5.69%.

Examining these daily numbers present a picture different from the Pew estimates, and indeed from most of what we’ve been seeing in the past 72 hours.  RAND estimate a 3.83% Obama lead.  RCP’s aggregator: Obama 0.5%.  Pollster’s aggregator: Obama 2%.  Gallup’s daily tracking: Obama 5% (RV, it should be noted).  Rasmussen today: a tie.  Sam Wang’s meta-margin is Obama +2.3%.  This tells us two things: Pew’s an outlier, and RAND is at the high end the other way.