The limitations of evaluating goaltender performance (Trending Topics)

Getty Images
Getty Images

Timely saves.

That's what Henrik Lundqvist and Jonathan Quick have made at various points throughout this Stanley Cup Final to keep their teams in it when all else seemed lost and things just weren't going their teams' way. 

That's what people would have you believe, anyway. That great goalies make the saves when they need to be made. Quick and Lundqvist have now traded stellar performances in Games 3 and 4, respectively, after being statistically lacking in the first two games of this Cup Final. In Game 3, a contest dominated by the Rangers, Quick stopped all 32 shots he faced to put the Kings up 3-0 in the series. Lundqvist got a game back with a 40-of-41 performance on Wednesday night to keep his team alive.

Great games, we can all agree. But Quick's was lumped in with his performances in the first two games of the series (25 of 27, and 34 of 38) as being “great” by the Watch The Games crowd, when they were in fact merely “very good” and “rather poor,” respectively, from a statistical point of view.

That's the tricky part about evaluating goaltender performance on really small levels. You can say so-and-so did well in giving up four on 35 (an .886 save percentage) and be right, and you can say someone was only okay in conceding two on 26 (.923), and be absolutely right as well. The tools we have available to us to evaluate goaltenders are extremely limited. Despite the fact that Corsi, Fenwick, zone starts, quality of competition, and so on have all popped in the last several years as ways to more effectively evaluate skaters' performances beyond simple points, penalty minutes and (if you're particularly forward-thinking) shooting percentage, we really only have three stats available to us to evaluate goaltenders.  

Each of these — wins (often with a capital W), goals-against average, and save percentage — is limited in its own way.

The most awful of these stats, one that's more or less useless in telling you anything about a player's quality, is “wins,” and you still see it cited a whole hell of a lot. A winner wins and a loser doesn't win and that's the end of it no matter what. Doesn't matter that Dominik Hasek, the greatest goalie of all time, has 12 fewer wins the Chris Osgood, who's maybe the 250th-best goalie in league history. Wins, I've been told, is “the ultimate stat.” This is, apparently, most often true when it comes to playoff success.

People argue that Osgood (career save percentage of .905, worse than Ondrej Pavelec) should be in the Hall of Fame based on this stat alone. But we all know in our functioning brains that wins is a measure of team success and not individual success, obviously. Do some players contribute to them more than others? Of course. Osgood likely hurt his teams' totals more often than not. Didn't stop them from winning two Stanley Cups with him between the pipes, though. 

Then there's goals-against average, which people — even if they are smart enough to dismiss wins as being a junk statistic cited only by idiots or intellectually dishonest water-carriers — cite quite often to indicate a goaltender's quality. This is getting closer to the truth behind what makes a goaltender good, at least, but it still falls short in a number of ways. First and most important is that it's still a team-based stat at the end of the day. 

Suppose Goalie A who only faces 25 shots a night because his team is good, and he has a goals-against average of 2.50. That means he has a save percentage of .900 (allowing five goals on every 50 shots he faces). Meanwhile, Goalie B has a goals-against average of 3.00, which is obviously 20 percent worse than Goalie A's number, but he faces 40 shots a night. Three goals allowed on 40 shots every game is a .925 save percentage. He's getting more work and he's doing more with it, but he's more likely to lose because he's giving up more goals, because his team is allowing more shots. Pretty simple concept, but that's why it's not a good stat for evaluating very much at all.

And that brings us to save percentage. It's the very best stat we have for evaluating goaltenders on an individual basis because it levels out many team factors (though not all of them) and simply tells you what percentage of the shots a goalie faces over any given period of time end up being stopped. Again, straightforward, and yet not enough to tell you everything you need to know. What if a team is bad defensively? What if a team is good defensively at 5-on-5 but bad on penalty kills? And so on. There are still factors to consider, but the noisier ones are at least filtered out for the most part.

(By the way, that has to be done over a considerable length of time, usually at least about 16,000 shots; goaltenders usually face about 8,000 per season if they're a full-fledged starter, and even bad goaltenders are capable of good years. But over time, those are leveled off, and until that point it's tough to know for sure what a netminder's true quality is.)

This is helped even more by the fact that the stats community now more or less determined that an even better way to evaluate goaltenders is through even-strength save percentage, rather than overall, because it gets rid of chances where they would be both more vulnerable overall (opponents' power plays) or they would be more likely to face higher-quality chances (when their team is on the man advantage, when they face greater risk a breakaway). 

As it stands right now, if you want to look at a single statistic to tell you whether a goaltender has been good, then “ESsv%” should be the only stop you consider.  But even 5-on-5 save percentage has its limitations, and the difference between Vezina-worthy and AHL-worthy is probably about one or two saves a game, on average. The biggest problem, opponents say, is that it does not account for “shot quality.” This is true, but there are so many things that go into determining that kind of thing.

The most common trope when people argue against the citing of save percentage for determining a goaltender's value is that shots “from center ice” aren't the same as those “in close.” Correct. But how many shots do you think come from the neutral zone, to go with that ludicrous premise, versus those from high-percentage parts of the ice? So you have to account for shot location, fine. 

However, not every shot from the same spot on the ice — let's say in this case it's six feet in and nine feet down from the right faceoff dot, a very good place to shoot from — is of the same quality. A Brian McGrattan shooting from there is far less likely to have success than a Patrick Kane. We all know that, too. 

And not every Patrick Kane shot from that one spot there is going to be the same, either. While even the majority from that spot might be crisp wrist-shots, some might be weak backhanders designed to get a rebound and nothing more. Others might be an attempt to have the goalie glove the puck and get a whistle for a change. So that has to be taken into account too. 

And what about his coverage? A wrist shot from that spot while Kane harried by a defender is probably less likely to go in than one for which he's wide open. What if it's a breakaway? Or what if there's a man between him and the goal, what are the odds that it hits the defenseman's stick and goes into the back of the net, versus hitting a stick and going into the pads, or hitting a stick and going into the glove, or hitting a stick and going into the stands? And what if there's a screen? What if there's a guy on the doorstep waiting for a rebound? What if there's a defender with a broken stick?  How does “bad ice” at the end of a period, or because of humidity, factor in? How does the “pressure” of the situation factor in? What about quality of competition? What about the number of saves he was asked to make earlier in the game?

People who want us to buy what they're selling when it comes to goaltending — and believe me, those who claim to goaltending experts but were not, say, high-level NHLers are always trying to sell you something — will also remind you that “shot quality” also relates to “shot quantity” in some way, because goalies who don't get a lot of work have a tougher time staying focused. Or something. That's why it's hard to be that Goalie A who only faces 20 shots a night. That's why you have to forgive a low save percentage sometimes, but not all the time.

These are all things to consider when determining “shot quality.”

Which is why “shot quality” cannot be quantified or uniformly understood in any way currently available to us. When SportVU, or whatever company wins the NHL's or individual teams' contracts, finally brings its technology to hockey — two years from now, probably — that might begin to give us an idea of who was good and who wasn't starting at that point, but there will be no way to go back in time and examine past goaltenders along the same criteria. Maybe the statistical models and simulations those systems spit out can put a hard and fast definition on “shot quality,” and maybe they can't. 

When people argue against save percentage, they are inherently asking all these questions. They want those who think this stat tells you something to give them concrete answers. Unfortunately there are none. Not any good ones, anyway. No one will ever be able to normalize every shot to conform to a single statistic that says, “Yes, this is how much better Henrik Lundqvist is than Jaroslav Halak.” It's hard to imagine skeptics would buy it even if that formula existed. “Put your calculator away, nerds.”

People who want you to believe in mumbo jumbo like timely saves point out that if you're going by save percentage alone, you're investing in a “flawed” view of the art of goaltending. That's the problem with the way goaltending is interpreted: It's little understood, even by the so-called experts, and a lot of mysticism comes with the territory. 

People are generally familiar with a lot of the terms and concepts that come with open play. Forecheck, backside pressure, and so on. Many observers of the sport could even point out to you, perhaps without knowing the terms themselves, who is the F1 or F2 on any given breakout. People can often tell, even before things happen, whether a zone entry attempt is going to succeed or fail depending upon any number of given factors they're looking at in a split second. Hockey makes sense to us in this way.

On the other hand, there is not a lot of familiarity with goaltending terms or theory. We hear goalies are technically sound, or that they position themselves well, and we know what that means in general, but we don't know what it means-means. Where should his glove be on a 3-on-2 with the puck coming down the left wing? Where should he be if a right-handed forward has the puck behind the net moving right to left without impediment from a defender? These are all factors that must be considered, and most people don't notice them because they're not watching the goaltender. Most times, they only leap into the collective conscious of the observers, whether at home or in the arena, when a shot attempt is made, but so much more goes into the position of goaltending than that split-second between release and save, or goal, or block, or miss, or post, that we don't understand, even if we've played the position ourselves. We're all guilty of this not-knowing. 

That's what leads to analysis like, “The book on him is to shoot high,” which was repeated about Quick throughout Game 4, because “he takes away the lower part of the net so well.” So, in 2014 when the butterfly style is and has been basically the only one practiced league-wide in years, does everyone else. Remember all those jokes about Corey Crawford's glove hand, and how it was pathetically weak? You only notice the goals that get blown by a guy, over and over again. Every glove save he made before that, no one noticed. Goalies are supposed to make stops, and the vast majority of those in the NHL do so at least 90 percent of the time, even if they flat-out stink.

So I'm really very sorry that “stats people” often cite save percentage, especially at 5-on-5, as the defining statistic in measuring goaltending quality. It's the best we have right now, and it's not like people haven't tried to improve on it. Lots of very smart people have examined this closely, and right now there's nothing they can do to give naysayers a better tool to evaluate goalies. No amount of asking people to consider sample size is going to help them stop their worrying about what exactly has gone so wrong with this sport that we're judging goalie quality on something as specious as the one stat that actually tells you something about their performance.

It's strange, though, that those who see things the other way don't try to come up with anything better to evaluate them. Even if they don't see every save a goalie has made in the course of a sample as small as a postseason, they're more than happy to stick the eye test and their gut. It's a lot harder to be proven wrong that way.

Ryan Lambert is a Puck Daddy columnist. His email is here and his Twitter is here.

What to Read Next