Outliers to the fore!

Because I teach rhetoric, not statistics, I chose not to have my students run standard deviations and what-not on the results of their final surveys. (Discipline-specific research classes further down the line will teach them the correct arcane maths to perform upon their data.) Instead, I thought I’d teach them how to present their oversimplified results in a professional manner, i.e. how to couch the results of their small surveys in language that acknowledges the limitations of generalizing from scant data. Since the majority of their citations came from the Ed Diener-edited Journal of Happiness Studies, that seemed like the best place to filch examples from. But what I found there dismayed me as both an academic and a former baseball player.

My baseball career was not illustrious: I was a defensive specialist whose slugging percentage mirrored a batting average that occasionally threatened to break the Mendoza line. Despite rarely making contact and having no power to speak of, during the summer of my junior year I slugged a Ruthian .640 and flirted with a .500 batting average … for 52 non-consecutive at bats against pitchers whose deliveries I knew intimately from having played behind them for three seasons.* To extrapolate my true talent from that five-game series would be the height of folly for the simple reason that the sample size would be hilariously small. Even for something as statistically quantifiable as hitting ability, a data set of fifty-two captures nothing of statistical significance.

Or does it? Because today I learned that I could have been the greatest hitter in the history of the game: the aforementioned journal considered the results of a survey of fifty-two people important enough to warrant publication. If that’s all that’s required to establish statistical significance in academia, it should be more than enough to demonstrate the same in a silly children’s game. By those standards, proof of the illustrious career I would have had can be derived from the self-reported data above so long as I include a paragraph like this one in the “Discussion” section:

The study has limitations which must be taken into account when findings are interpreted. According to common statistical guidelines, the present sample size was small, resulting in modest effect sizes, so the findings should be viewed as preliminary, and requiring replication in a larger sample. Interpreting this evidence is by no means straightforward, but the way we interpreted our findings were consistent with the theoretical position we entertained. In sum, preliminary investigation indicates that Scott Eric Kaufman likely would have been the best hitter in the history of professional baseball.

So my assignment for my students is to isolate ten examples of the rhetoric of dubious-but-professionally-acceptable hedging in the Journal of Happiness Studies and employ them in their discussion of the results of their surveys. If their pool of 52 students happens to contain two homosexual women of African-American descent who are also on the dean’s list, they can generalize about the academic achievement of all African-American lesbians so long as they couch their conclusions in the speculative rhetoric found in the journal. This may seem irresponsible of me, but remember that my purpose here is to teach them how to write well and read critically, and if they’re parroting the prose of established scholars and learning to mistrust certain rhetorical tics, I’ve done my job.

*I went something like 26-52 with 10 doubles, all of which were pulled down the line. They tipped the breaking pitches they couldn’t throw for strikes, so I sat on the two-seamers that never really sank, started swinging outrageously early and yanked them down the line.

Update: Read the comments before you decide to SMASH. Apparently my undergraduate statistics teacher pushed the anecdote-is-not-data argument a tad too enthusiastically, so my annoyance with a sample size of 52 is unjustified. Fortunately, as they say: Blogs is for learning.

Update 2: Despite the manner in which I flubbed the discussion of sample sizes, the exercise described in this post had its intended effect (which just goes to show why I should not be teaching statistics, as was suggested in the comments, and stick to what I know how and was trained to do).