The ‘Stats Guys’ vs. Cinderella, or how Minnesota Wild regressed to the mean
(Ed. Note: Kent Wilson it the managing editor for Flames Nation, who has written for Hockey Prospectus and Houses of the Hockey. We're honored to bring his intelligent and thorough analysis of the NHL to you here on Puck Daddy.)
"Whatever you think you know about your team's supposed ability to maintain high shooting and save percentages, they are very likely to crash back to league average regardless of how many shots you've observed. Internalize this…and you can make a lot of money betting against people who are convinced there's mysticism in scoring goals." - Gabriel Desjardins, Why You Should Ignore Shooting Percentage
Unless you are a Minnesota Wild fan or a numbers-inclined hockey analyst, you're probably only vaguely aware of the on-going battle between these two factions. Here is some background on the matter:
After the first 31 games of the season, Minnesota was leading the Western Conference in points. Their Cinderella-like rise from the West's basement was an apparent confirmation of the organization's various off-season moves, from hiring bench boss Mike Yeo, to dealing Brent Burns and Martin Havlat for Dany Heatley and Devin Setoguchi. A Church of Yeo sprung up in worship of the new bench bosses uncanny ability to squeeze success out of a line-up that was predicted by most to miss the playoffs.
Minnesota's record was unlikely for a numbers of reasons. Not the least of which was the fact they were getting routinely outshot. In fact, despite boasting one of the best records in the league at the time, the Wild had surrendered 173 more shots on net at even strength than they had generated up to that point. They had also blocked 145 more shots than the opposition. Again, that's only at even strength.
Their total shots for/against (or "CORSI ratio") to that point was just .419, one of the worst in the league. Nevertheless, the underdog Wild were "finding ways to win," to borrow a cliché, so any skepticism was dismissed out of hand.
After all, pointing to the standings could readily silence any unbeliever.
There were some persistent heretics, however — statistically oriented writers and bloggers who acknowledged the Wild were living off of sky-high save percentages that were unlikely to continue in perpetuity. Truly great teams, it was argued, tend to control puck possession and outshoot their opponents. As such, Minnesota's success was likely a mirage. Regression was inevitable and with it, a fall from grace.
Raining on a parade is never popular. The Minnesota faithful understandably bristled at suggestions their team was merely lucky.
"Regression to the mean" became a punch line in Wild fan circles.
Of course, with Minny currently sitting 12th in the Western Conference heading into Thursday night, the next chapter of this story is an obvious one. Daniel Wagner of Backhand Shelf summarizes here what has happened since.
The purpose of this article isn't to dance on the grave of the Wild's short-lived elite status. Nor is it to point and laugh at Wild fans. The episode is an object lesson in how percentages can vary wildly around a mean in small samples and why that is so counter-intuitive to the fan experience.
In his recent book "Thinking, Fast and Slow," psychologist Daniel Kahneman notes how poorly people tend to grasp statistical truths like regression or the influence of sample size on results. In his chapter "Regression to the Mean," Kahneman details how apparently foreign the concept is to us:
Whether undetected or wrongly explained, the phenomenon of regression is strange to the human mind. So strange, indeed, that it was first identified and understood two hundred years after the theory of gravitation and differential calculus.
Humans prefer patterns and causal chains to abstract notions of variance or probability. In fact, people tend to identify apparent patterns in randomness without effort and to fit noisy, complex events with tidy narratives that make them easier to understand and more coherent. In the same chapter, Kahneman shares this anecdote:
I happened to watch the men's ski jump event in the Winter Olympics while..I (was) writing an article about intuitive prediction. Each athlete has two jumps in the event and the results are combined for the final score. I was startled to hear the sportscaster's comments while athletes were preparing for their second jump: "Norway had a great first jump; he will be tense, hoping to protect his lead and will probably do worse" or "Sweden had a bad first jump and now he knows he has nothing to lose and will be relaxed, which should help him do better."
The commentator had obviously detected regression to the mean and had invented a causal story for which there was no evidence. The story itself could even be true…And perhaps not. The point to remember is that the change from the first to the second jump does not need a causal explanation. It is a mathematically inevitable consequence of the fact that luck played a role in the outcome of the first jump. Not a very satisfactory story — we would prefer a causal account — but there it is.
The tendency to apply ex post facto rationalizations to outcomes is common and obvious in hockey. Visit any struggling team's message board and marvel at the number of fans who are apparently intimately aware of the club's locker room environment and how various personalities are affecting it. Coaches, systems, "will to win", "playing for each other" and other such explanations were earnestly proffered to defend the Wild's incredible first quarter of the season by fans, for instance.
As Kahenman notes, some of these stories may even be true, but by and large the uncanny winning streaks and unsavory losing streaks that we see often have as much to do with the inscrutable ebb and flow of variance and probability as the team's true talent level.
Over a long enough timeline, a club's talent level will emerge, but over relatively small samples such as 10, 20, or ever 30 games, lousy teams can post a winning record and vice versa as a matter of chance.
Perhaps the greatest and most frequent charge against the spreadsheet analysts was that they were assessing the Wild's abilities without "watching the games."
Direct observation is indeed the most data rich method of analyzing hockey teams, but it is also potentially the most deceptive; particularly through the rose-colored glasses of a fan.
Further, some events are predictable even without direct observation. For instance, if a friend asked you to predict what would happen if he threw a baseball as high he could in the air, your natural response would be that it would fall back down to earth. It wouldn't matter if you had seen him throw a baseball once or a thousand times: some things are inexorable. Unless he's a superhero or floating outside a space station, that ball will eventually peak and then fall from the sky, no matter how apparently good an arm your friend might have.
Fights between a Cinderella club's fan base and "the stats guys" aren't new. Last year the Dallas Stars had a similar flight up the standings and then a subsequent fall from grace in the second half. The Colorado Avalanche's incredibly unlikely run to a playoff appearance in 2009-10 was also a bone of contention between the bean counters and Avs fans. In fact, a cult of personality sprung up around Joe Sacco similar to Mike Yeo, although he was later targeted as a scapegoat when the team failed to replicate the feat one short year later.
Each season has at least one surprising team come out and ride the percentages for a few months before regression eventually hits.
Hockey is a complex game and all sorts of things can happen that makes predicting the future nearly impossible. However, regression is as persistent as gravity and a team dependent on unusually high percentages for success will inevitably fall back down to earth — regardless of how many of their games you watch.
