What We Learned: What NHL GMs don’t know about quality goaltending

(Hello, this is a feature that will run through the entire season and aims to recap the weekend’s events and boils those events down to one admittedly superficial fact or stupid opinion about each team. Feel free to complain about it.)

There is a huge shortcoming when you talk about goaltender evaluation at just about any level of hockey:

We only have one number that really tells us anything about their quality.

All we have is save percentage. That's it. There's not a lot more we can really do at this point to objectively understand their efficiency, efficacy, and so on when it comes to doing anything involved in their job, except for what it ultimately boils down to: Stopping the puck.

For example, who's the best in the NHL at getting from one post to the other? Who takes up the most net? Who best takes away scoring chances in 1-on-1 situations? These are all measures that probably could be measured, but not right now, and that means that while we can have opinions as to who does all those things better than anyone else, it's very, very subjective.

[Yahoo Fantasy Hockey: Sign up for a league today]

The good news is that we are at least getting better at this sort of thing. Shot location data is easier to come across now than it ever has been, so we're getting a pretty good idea of how likely a shot from x part of the ice is to go in, generally speaking. Obviously, Steven Stamkos taking a shot from the hashmarks has a much better chance of going in than Brandon Bollig shooting on the same goalie from the same spot. Likewise, Henrik Lundqvist is statistically going to be far more likely to stop that shot than Ondrej Pavelec.

But again, we have a general idea of how “shot quality” figures into this sort of thing.

Which is why the quality-adjusted save percentage stat that has been developed in the last few years is so valuable. As the name implies, it adjusts for the overall quality of shot the goaltender faces — i.e. giving them more credit for stopping higher-percentage, quality shots — to somewhat level the playing field (it does not, however, account for shot volume, meaning busier goalies get no real benefit here).

This was something that became rather hard to ignore when Craig Custance released his now-annual ranking of the league's 30 likely starting goalies on a scale of 1-5, with 1 being the best. This is a poll of six GMs, one assistant GM, three head coaches, and four goalie coaches. These are, then, guys you can safely consider “hockey people,” who watch a lot of games and have a lot of experience in the sport.

And yet, the actual rankings they handed in two years running really didn't really make a lot of sense. How, for example, is Jonathan Quick better than Tuukka Rask? How are there 11 goalies in the league rated more highly than Braden Holtby? How is Jonathan Bernier worse than Ondrej Pavelec? There are a lot of questions you can ask here, but even if you check out their justifications (and you should, because the whole thing is really interesting) you're left asking how Mike Smith, for instance, “has as much talent as any guy on the list.” He, in fact, has a long and demonstrated statistical history of exactly the opposite, save for one season in which he was inexplicably a .930 goalie over 67 games.

Well, funny you should ask.

“It comes down to confidence and consistency,” said one coach. “And being about to go out night after night and reproduce what his strengths are.”

And that is just the kind of mumbo-jumbo you hear a lot with goalies because, again, they are incredibly difficult to evaluate. Even statistically minded people will repeat over and over that “Goalies are voodoo,” but what that really means is that they have a position that is more likely to be ruled by randomness than any other in the sport.

For goalies, the difference between a good season and a bad one is failing to stop an extra 10 shots out of 1,000 in a lot of cases. But over time (say, the last three years) quality-adjusted save percentage is going to tell you a lot about a goaltender's quality in a lot of cases.

What made things a little tricky this year is the fact that Edmonton and San Jose are both using goalies with very good statistical profiles — Martin Jones and Cam Talbot rank first and third, respectively, in adj. 5v5 sv% over that time — with very little actual in-game experience. Most goalies on this list have more starts in a given season than they have in their brief careers as backups, as you might expect. Buffalo and St. Louis are both likewise going with inexperienced hands who were ranked on this list, but whose stats were a little less rosy. Even Minnesota's Devan Dubnyk is little-used in comparison with a lot of these guys.

But still, looking at these ratings versus adjusted save percentage highlights that there's a lot of poor evaluation going on out there these days from some very smart hockey people.

When looking at this chart, great goalies should be far to the right, and bad ones low to the bottom. And yet you see a lot of guys who are quite high that don't get very far over (the one in the 3 range is Jones, the one closer to 1.5 is Rask), as well as some quite low ones who aren't nearly as far left as they should be (the lowest guy in that far-right group is Pekka Rinne.)

First things first, let's note that at the extremes, they mostly seem to get things right. GMs are going to have good idea of who is really good or really bad; No. 1 Carey Price, No. 2 Henrik Lundqvist, No. 30 Cam Ward, etc., all make sense. There are only four goalies inside of the 1.5 rating, and six are below 3.

However, none — not even unanimously awful Cam Ward — could be evaluated as being less than 4 overall? All the goalies are at least “a little less than fair?” That seems crazy, especially given the evidence we have to suggest Cam Ward shouldn't be a starting goalie in the league at all. One GM polled even specifically said, “I think he might be done,” but we can't get a better consensus than “he's pulling about a D+ grade?”

But there's a lot of crazy crap in the middle (not even counting “Jonathan Quick is better than Tuukka Rask,” which is absurd). Again, it is difficult to evaluate goaltenders but the fact that there's only about a 14 percent correlation between what these hockey lifers think and what the actual performance is crazy.

And on one level, you can say “who cares,” right? Who cares what a small sampling of people, even those with a wall full of credentials, might think about an individual guy they may only see in person a handful of times per season?

Fair enough. But here's the thing: Seven of the 14 guys polled were either GMs or AGMs, and three more were head coaches.

These are the guys that sign the deals or put together the lineup. So if they're misevaluating goaltenders — which they are, by and large — then that hurts their teams' chances of winning. Especially if those goalies are signed to fat paychecks. Which many of them are, as discussed by Corey Masisak on Friday. For example, 18 goalies carry a cap hit of at least $5.2 million, and seven more are between that number and $4.1 million.

Indeed, here's the same quality-adjusted save percentage numbers, but this time compared with those goalies' cap hits for this season.

GMs perform even worse here: There's only a 3.88 percent correlation for three-year adjusted save percentage at full strength and salaries. Which leaves a lot of room for some very bad contracts. In an ideal world, you're getting contracts high and to the left on this chart, but probably don't mind paying for the ones high and to the right. When you're down and to the right, that's where the problems start.

Every dollar is valuable, and you certainly get more bang for your buck out of a goalie than any other single position in the league, simply because they play the full 60 and are often the difference between winning and losing. In terms of dollars spent per win, you basically can't overspend on an elite goaltender, but you can very quickly spend too much money on mediocre goaltending (ask Dean Lombardi). And that doesn't even begin to get into the term on many of these deals, which are often four or five years at the minimum. It's crazy, and typically not good valuation.

But at least these talent evaluators are pretty consistent in how they examine these players; there's a more than 36 percent correlation between AAV and evaluator rating, meaning that they generally believe guys with big contracts also happen to be the best goalies, far more so than they think the guys with the highest save percentages are the best in the game.

Seems they're big fans of their own work, even if that work isn't very good.

What We Learned

Anaheim Ducks: The Ducks have won the Pacific Division three times in a row and a fourth seems very much in the cards. And man, it used to be frickin hard to win the Pacific. Turns out this Bruce Boudreau guy is good.

Arizona Coyotes: The Coyotes went 0-fer in the preseason, having scored just one (ONE!) goal at 5-on-5. Goal differential across six games was 20-4.

Boston Bruins: “Bruins can't muster enough offense even as Tuukka Rask is incredible” sounds like a pretty plausible storyline for the entire season, doesn't it?

Buffalo Sabres: Are we really already saying “Bylsma for Jack Adams?” With zero games played? Slow down.

Calgary Flames: It's going to be fascinating to see what the Flames give Kris Russell. Because he isn't very good, and Calgary seems to think he very much is. And with the contracts they're going to have to give out over the next few years, overpaying a borderline No. 5 defenseman would be disastrous. Get your popcorn.

Carolina Hurricanes: Carolina has only nine players signed for 2016-17, plus five RFAs. You can probably say so long to plenty of guys on the current roster in the very near future.

Chicago Blackhawks: Why it's almost like you shouldn't give bad players big-money contracts based entirely on one postseason run. I don't know.

Colorado Avalanche: To other teams, you mean?

Columbus Blue Jackets: William Karlsson's nickname is Wild Bill. Hard not to be very into that.

Dallas Stars: Very sorry to hear about this, Stars fans. You have our sympathies.

Detroit Red Wings: How dare Dylan Larkin challenge Detroit's assumptions that all players under the age of 22 toil for years in the AHL, regardless of their talent level? Doesn't he know they want to play Darren Helm and Drew Miller 14 minutes a night?

Edmonton Oilers: The Oilers' defense isn't very good? The devil, you say!

Florida Panthers: The Panthers largely say they like the idea of a 3-on-3 overtime. They seem like a team that could do better in it than the shootout, given the amount of high-end skill they have up front and that 19-year-old defenseman at the back.

Los Angeles Kings: Drew Doughty really just ought to have the full-on captaincy, right?

Minnesota Wild: This is not a good choice to have to make.

Montreal Canadiens: Well guys I have some bad news about this: Michel Therrien is your coach.

Nashville Predators: Pekka Rinne (and probably Cory Crawford) are probably going to be poster boys for the “goaltender's workload” problem in this league.

New Jersey Devils: Another judicious deal in a growing line of them for Ray Shero.

New York Islanders: What if being in a building that's actually easy to get to is good? Hmm. Hmmmmm.

New York Rangers: The Rangers got plenty of luck last year and didn't win anything. How much more would people realistically like them to have?

[Yahoo Fantasy Hockey: Sign up for a league today]

Ottawa Senators: Raise your hand if you remembered Clarke MacArthur played for the Senators. Well, now he doesn't because he's injured. At least a little bit.

Philadelphia Flyers: Oooooo they should try this with most of their other blue line veterans too.

Pittsburgh Penguins: Yeah, having Sidney Crosby, Phil Kessel, and Kris Letang as your 3-on-3 unit seems bad.

San Jose Sharks: If Martin Jones can actually stay good all season, that would be some kind of thing. Have to doubt it happens, though.

St. Louis Blues: Do you think this is just copied and pasted from the last two seasons?

Tampa Bay Lightning: This is a really good look at the Stamkos contract situation as the season looms large here. You gotta wonder how much negotiating will really happen after the season starts.

Toronto Maple Leafs: Mike Babcock got to Toronto expecting Nazem Kadri to be The Worst, simply because of how the local media talks about him. Turns out: “All the things I heard about him, none of them were true.”

Vancouver Canucks: “Brandon Sutter with the Sedins” seems very much like a thing that isn't going to work out.

Washington Capitals: Hey, remember when the Capitals were playing third-line center Jay Beagle with Alex Ovechkin? Man, what a time that was.

Winnipeg Jets: It shouldn't be that hard for the Jets to make the playoffs. Not getting creamed in the first round, on the other hand...

Gold Star Award

Wow, it's quite a world when a team can sign its starting goalie for two years at the Eddie Lack price point. Very shrewd move in Carolina.

Minus of the Weekend

Andrew Hammond being out two weeks is going to be rough. As long as those two weeks are at the end of last season. Otherwise, probably not a big deal.

Perfect HFBoards Trade Proposal of the Week

User “legitbeauty” is feeling good.

michalek and lazar for okposo

Signoff

He’s a can of vegetables. He doesn’t have to know.

Ryan Lambert is a Puck Daddy columnist. His email is here and his Twitter is here.

(All stats via War On Ice unless otherwise noted.)

MORE FROM YAHOO HOCKEY: