If there's one thing I've learned as a longtime sabermetrics geek and sixth-year member of the Football Outsiders staff, it's that if you don't fully road-test a new statistic through a series of probabilities and permutations before foisting it upon an unsuspecting populace, there's a very good chance that stat you're so proud of is going to bite you right in the butt.
So it seems to be with ESPN's new Total QBR rating. The metric, which the Worldwide Leader unleashed on America before the 2011 season, has been fairly shoved down our throats as the end-all/be-all of quarterback rating systems. ESPN claims that its metric weighs play importance and clutchiness in ways that no other stat possibly could, completely ignoring the fact that Total QBR is basically a rating version of Football Outsiders' nearly decade-old percentage-based DVOA stat — except that QBR doesn't adjust for opponent, and DVOA does.
Another thing that QBR doesn't seem to take seriously is the fact that it's a bit tougher to rack up quality percentage stats when you're throwing the ball 40 times than when you're throwing it 10 — apparently, the all-knowing QBR doesn't weigh that aspect of the game. We know this because on Monday, it was revealed that Denver's Tim Tebow(notes) — he of the 4-for-10 passing day against the San Diego Chargers — had a higher total QBR than did Green Bay's Aaron Rodgers(notes), who completed 26 passes in 39 attempts for 396 yards and two touchdowns against the Atlanta Falcons last Sunday.
And before you think that the lab dorks at ESPNU broke in and fiddled with Tebow's rating to make the thing "buzzier," hold up! ESPN actually went to the trouble to explain just exactly WHY their prized system aligned those two quarterback performances as it did.
We'll get into that in a second, but here's what Rodgers had to say about it this week when he was asked. The best quarterback in the game today was more bemused than rebuffed, which seems to be an appropriate reaction when discussing the stat.
"The only time I ever see the QBR is on the bottom line on ESPN," Rodgers said. "I saw that [Tuesday morning] and chuckled to myself. I played a full game, he played the half. He completed four passes, I completed 26. I think it incorporates QB runs as well.
"The weighting of it doesn't make a whole lot of sense," Rodgers continued. "They said a third down in the fourth quarter should mean more than in the first quarter — OK, I get that. A touchdown pass late in the game against a team that's up by a bunch of points isn't going to be worth as much. I think when you start putting values, and judging the effort levels a team may be giving up 14, as opposed to tied, or up even ahead by 28, I think you're getting into some gray areas."
So, on to ESPN's explanation(s) here and here, which actually go a long way to explaining the ways in which QBR is uniquely flawed. That would be understandable and acceptable under any circumstances — there is no perfect stat — except that it's a bit funnier to see that result after the WWL has gone out of its way to trumpet QBR as the single greatest quarterback evaluation tool in the history of mankind. You know how some people are — nobody invented anything before they did.
First, the ESPN analytical team tries to explain that "Tebow posted an 83.2 QBR in his partial comeback Sunday. This -- along with Week 15 last season -- is as good a game as he has had, but a very small sample. It's like a pinch hitter hitting a home run in his only at-bat. That 83.2 is better than Aaron Rodgers' 82.1 against the Atlanta Falcons because Tebow was efficient. It doesn't mean, though, that he is a star. The San Diego Chargers weren't ready for him, and they were winning by two touchdowns when he entered the game. Tebow was good for a quarter and a half."
Problems 1 and 2 right there. First of all, QBR doesn't have a performance frequency floor any player has to hit before he can be rated along with other players who had more chances to either succeed or fail. ESPN admits that its "all-encompassing" metric doesn't differentiate between the NFL versions of leadoff and pinch hitters. How is it possible to take such a metric seriously when analyzing performance trends over time? Second, Tebow wasn't efficient. He completed 40 percent of his passes — one for zero yards, one that required a miracle catch from Brandon Lloyd(notes), another that came against a San Diego defense that was playing prevent (to assume that the Chargers "weren't ready for him" is beyond ridiculous). When the research team says that "QBR is explained as a measure of efficiency at any given moment, not necessarily over the same time period," that's more an expose of a flaw in the system than it is an explanation.
OK, next — QBR places a certain elevated weight on non-passing plays for quarterbacks, which is very problematic in this case. When it's revealed that "QBR evaluates a wider spectrum of quarterback play than completions and yards, including sacks, scrambles and passing yards after the catch," that's a lead-in to the fact that "Rodgers took four sacks. Tebow did not have a play that resulted in negative yardage. Quarterbacks are given a share of blame for sacks," and "Almost half of Rodgers' passing total, 197 of 396, was judged to come after the catch, for which quarterbacks receive less credit."
Well, if you're going to give Rodgers less credit for yards after catch gained (which seems odd, considering that you're setting up an automatic bias against quarterbacks in offenses, like Rodgers', that are based very much on YAC), why aren't you giving him more credit for the fact that he took some of that pressure without his starting left tackle? And as far as Tebow not having a play that resulted in negative yardage, remember also that he had far fewer chances to create a negative play, which goes back to the whole concept of weighing participation.
Here's the real corker: QBR appears to weigh a quarterback's performance relative to the game situation -- on offense and defense! Here's how the difference between Tebow's performance and Rodgers' performance was described: "Rodgers performed about the same as Tebow in bringing his team back from a 14-0 deficit (QBR of 83.6 from start of 2nd qtr). However, unlike Tebow, Rodgers was on the field when [his] team went into as 14-0 hole. Since those plays count towards his overall [performance], his QBR was 82 for the game."
And that's where I dropped the narrative and started cracking up. Basically, QBR is penalizing Aaron Rodgers for the 14 points his defense gave up before his own offense could score a point. Did Matt Ryan(notes) get less of a "clutchiness" grade when his defense gave up 25 unanswered points?
Look, it's less about bashing ESPN and its precious new stat and more about understanding that you can't just assign numbers to performances and expect them to work all the time. People generally understood that. Where ESPN is erring, and why Rodgers found QBR to be so skewed, is that ESPN seems to want to alter the story whenever events don't fit the numerical setup. And it's things like this that set the cause of advanced statistical research back — things like this that forward the causes of people who align themselves with the old quote which put forth the proposition that "There are three kinds of lies: lies, damned lies, and statistics."
I don't believe that for one second, which is why I find the QBR "explanations" so offensive.
Posted Jul 2 2012
Posted Jul 3 2012
Posted Jun 21 2012