The most reactionary Court ever, and the limits of statistical models of Supreme Court votes

The end of a contemporary Supreme Court term generally involves reporters straining to cherry-pick a few cases to show a Surprising Moderation from the reactionary majority/supermajority. As Steve Vladeck points out, these arguments (as well as many statistical models) fail to adequately take the Court’s control over its own docket and the shadow docket into account:

Statisticians call this phenomenon the “tyranny of averages” — the fact that averaging a data set tells us nothing about the size, distribution or skew of the data. But these kinds of “judge the Supreme Court by its data” assessments are even worse than just ordinary statistical errors.

First, they fail to account for the Supreme Court’s own role in choosing the cases it decides — so that the data isn’t random to begin with. Second, they ignore all of the Supreme Court’s significant rulings in other cases — those that don’t receive full briefings and arguments. Finally, even within the carefully cultivated subset of cases on which these claims generally focus, these commentaries both miscount the divisions and treat as equal disputes that bear no resemblance to each other. It’s not that this data is completely irrelevant, but anyone relying upon it should take it with a very substantial grain of salt.

Let’s start with the court’s docket. With one tiny exception (which accounted for exactly one case during the justices’ current term), the court chooses each and every one of its cases (and, even within those cases, which specific issues it wants to decide). This docket control, which is entirely a modern phenomenon, means the justices are pre-selecting the cases they decide — including technical disputes on which they may be likely to agree (or, at least, not disagree along conventional ideological lines). Thus, from the get-go, the entire data set on which too many commentators rely is biased toward the justices’ own behavior.

Thus, statistical claims about the court tend to neglect the thousands of other rulings the Supreme Court hands down every term — on what has become known as the “shadow docket.” These rulings are unsigned and almost always unexplained, and they run the gamut from agreeing or refusing to take up an appeal to agreeing or refusing to block a lower-court ruling while the appeal runs its course.

Many of these rulings are relatively insignificant, but some are just as important as — if not more important than — cases that receive plenary consideration.

Consider the April ruling that preserved nationwide access to mifepristone or the December ruling that left in place a controversial Covid-related border control policy. Indeed, there have been 35 shadow docket orders from the court since October from which at least one justice publicly dissented — including six from which all three of the Democratic appointees registered their opposition. (That’s in contrast with a total of seven argued cases in which all three dissented.) Shouldn’t that data figure in any putatively comprehensive summary, too?

Finally, even within the skewed subset on which these statistical claims rest, there are serious false equivalency issues. It’s not just that a 237-page ruling invalidating race-based affirmative action policies at virtually every college and university in the country has a far greater impact (and is far more important in almost all respects) than a 16-page technical resolution of a question of bankruptcy procedure; it’s that the way we count votes doesn’t necessarily reflect the true divisions among the justices.

Consider Sackett v. EPA — a major decision in which the court dramatically curtailed the federal government’s ability to prevent pollution of wetlands. Raw data treats that ruling as unanimous — because all nine justices agreed that the lower court applied the wrong test. But with regard to the rule going forward, the justices divided 5-4 — with Justice Brett Kavanaugh breaking from the other conservatives and writing for himself and the Democratic appointees in a sharp separate opinion that embraced a broader reading of the statute. No statistical summary of the court’s work treats that decision as 5-4 — even though, for all intents and purposes, it was.

The caseflow point is particularly important. Models will generally score Moore v. Harper as a “liberal” outcome. But in a federal judiciary controlled by moderate liberals, the “independent state legislature theory” would probably have gotten nowhere near the Supreme Court, and had some rogue Circuit Court come up with it the Supreme Court would have unanimously overruled it. The fact that there were almost certainly three votes on the merits for the theory that gerrymandered Republican legislatures can manipulate the outcome of elections irrespective of either the will of the voters or their state constitutions shows, and that the majority felt the need to make some concessions to it to maintain its majority, shows that this is an extremely conservative Court.

To be clear, the best political scientist who work on judicial behavior are aware of these problems and trying to adapt their models accordingly. But relying on basic data can produce very misleading results, especially if they’re the results you’re trying to find in the first place.