Steroids, performance, and evidence

Moralistic hand-wringing about steroids is frequently accompanied by somewhat extravagant claims about the dramatic impact they have/had on individual player performances and the game itself. In the thread below, we get this from IB (who generally manages to avoid drug war moralism about steroids in baseball):

1) Steroids pretty clearly give a bigger performance boost than speed. To wave one’s hands and say we can’t know this for sure seems intentionally obfuscatory to me. I agree that admitting this doesn’t, in and of itself, tell us what to do about steroids in baseball. But that’s all the more reason to be suspicious of arguments that seem to be based on pretending it isn’t the case.

I have no idea whatsoever what impact, if any, speed had on performance, so for all I know I might agree with this post. Nevertheless, I confess to being entirely unaware of what the persuasive case for steroids’ substantial impact on player performance is. This isn’t denial that such an impact might exist; such claims are well within the realm of plausibility. But I’ve never seen anything approaching compelling evidence for it. I’ll also concede that gathering evidence about whether it exists or not is remarkably difficult, given our lack of reliable data about who was using steroids and who was not. So the following is offered in part as an effort to convince IB I’m not being intentionally obfuscatory, and in part as a plea for better evidence in favor of this proposition, if it does or even can exist.

The first type of argument for the proposition involved the tilt toward offense in the offense/defense balance beginning in 1987, retreating for a couple years, and coming into force in the 1990’s. But offense/defense balance changes all the time for all kinds of reasons, and now, in the wake of Clemens, people are claiming it’s self-evident that steroids can substantially improve pitcher performance as well, rendering the original claim incoherent.

But the bulk of evidentiary claims amount to hand-waving in the direction of players with unusual career paths who’ve been tied to steroid use. This is useless as evidence because some players have career paths that are unusual. It’s akin to saying that because you guessed the number between 1 and 10 I was thinking of, you must be psychic. And employed in a haphazard manner, it’s deeply susceptible to confirmation bias. Since a master-list of steroid users and non-users can’t be had, we can’t test whether steroid users were more likely to have particular kinds of unusual career paths than non-users. Useful possible evidence would begin with an examination of the extent and frequency of deviations from standard career paths in the steroid era vs. other eras, with particular attention to late-career spikes. If these kind of unusual career paths were occurring at a notably higher rate during the steroid era, we’d have some solid but not conclusive circumstantial evidence for the claim. If someone has actually designed such a study, which seems doable for a competent sabermetrician, I’m not aware of it.

Weirdly, this same reasoning is sometimes used with players who have not been linked to steroid use: Nolan Ryan in previous comment thread on this subject, Brady Anderson’s 50 HR year, etc. In these cases, the evidentiary claim is hilariously circular–the unusual career path is evidence of steroid use, the steroid use, knowledge of which is adduced from nothing but an unusual career path, is said to have caused the career path aberration. The problems here are too obvious to belabor.

A variety of other claims pop up from time to time, such as “it must do something if so many of the players were doing it” which would work equally well as evidence for the effectiveness of various superstitious rituals successfully improving performance. I don’t find it all particularly implausible that large number of people convince themselves that something is causing a substantial change when it’s not. The placebo effect if very real, and Enzyte, I’m told, has millions of loyal consumers. We also see people focus on certain player’s changes in physique. But Baseball players are not offensive linemen, and their success doesn’t necessarily come from increased strength and muscle size. While this (whether steroid-aided or not) might improve some hitters power, it might mess with another’s swing or inhibit their defensive skills. Batting eye and bat speed are more important than bulk, and from what I understand there’s no medical reason to link steroids to improvements in either of those categories.

This is not a defense of quasi-defense of steroid use by baseball players. Since I’m not interesting in moralizing drug use and I have no good reason to consider myself a stakeholder with respect to steroid use by baseball players, it’s not really my business. But the social scientist in me rebels at being told that rejecting wholly unsatisfactory and unsystematic evidence makes me “intentionally obfuscatory” with regard to steroids effect of player performance.

….just looked up Nate Silver’s essay “What Do Statistics Tell Us About Steroids” in Baseball Between the Numbers. After dispensing with the “power spike” theory, he examines all the players, major and minor leagues, who were suspended prior to and after the suspension. The potential problems with this approach are manifest, but at least it gives us something. Position players saw a post suspension drop of .009 EQA, and pitchers saw a post suspension increase of .13 ERA. In other words, the change was in the ‘right’ direction for a steroids matter hypothesis, but only very slightly (and in for the pitchers, below the threshold of statistical significance). This doesn’t, of course, measure the possible effect of improved injury recovery, which might be particularly important for relief pitchers (allowing them to pitch more frequently). But as far as performance, this study suggests a minimal impact, and Silver correctly concludes it gives us no compelling reason to reject the null hypothesis.