Home Unfiltered Articles Players Baseball Prospectus
Basketball Prospectus home
Click here to log in Click here for forgotten password Click here to subscribe

Click here for Important Basketball Prospectus Premium Information!

<< Previous Article
Bracket Breakdown (03/26)
Next Article >>
Bracket Breakdown (03/27)

March 26, 2009
Evaluating APBRMetrics
The State of the Art

by Kevin Pelton


Recently I did one of the most dangerous things any columnist can do: I read through some of my old work. There on the Hoopsworld.com archive, in between touting Mike Sweetney and questioning the length of the contract Phoenix gave Steve Nash, I found a three-part series entitled "The Future of NBA Statistics." We're coming up on the five-year anniversary of that column, so the future, as they say, is now.

Here's the funny thing: When I read through that three-part series, though the names and numbers are outdated, I was struck by how familiar the concepts seemed.

I've been thinking a lot lately about where NBA statistical analysis is headed, starting when I read Michael Lewis' feature on Shane Battier, Daryl Morey and the Houston Rockets. The MIT Sloan Sports Analytics Conference earlier this month provided plenty of thought-provoking conversation. The topic then returned to my attention this week after reading SlamOnline.com's interview with Roland Beech of 82games.com as well as Bill Simmons' treatise on the state of NBA statistics in the latest ESPN the Magazine.

Ultimately, the discussion seems to center largely on the two questions that were the focal point of my columns five years ago:

  • Can we boil a basketball player's value down to a single number?
  • What better describes a player's value: his individual statistics or plus-minus data describing his impact on the team?

If you would have told me five years ago that neither question would yet be answered to anyone's satisfaction, I would have been disappointed and somewhat surprised. What happened? Simmons notes one key explanation in his column: Teams got interested in this stuff. Five years ago, no NBA team employed an analyst on a full-time basis. Oliver was attempting to get just such a position, a process that culminated in his hiring by the Seattle SuperSonics that fall. While full-time statistical analysts remain the exception rather than the rule, at least nine teams use at least one person, with Morey's Rockets employing a full roster.

This is, in general, a great development for the statistical community and a validation of its work. It has also been an enormous setback for a couple of reasons. First, whatever progress Oliver, the Houston team and others have made toward answering the important questions of APBRmetrics has remained under lock and key and will at least until Simmons manages to get a tipsy Morey to reveal his secrets. (He was kidding about that plan--I think.) Second, when a member of the community starts doing new and interesting work, like Eli Witus or David Sparks, it is usually a matter of time before they end up being hired by a team (Witus now works for the Rockets, Sparks the Boston Celtics).

With the benefit of hindsight, the way baseball teams ignored the sabermetric revolution for years and years turned out to be a blessing in disguise, allowing a growing community to freely advance on each other's work. By the time a large number of teams got wise to the value of statistical analysis, sabermetrics was entrenched to the point where it could easily survive teams poaching some of the best talent.

None of this is to say APBRmetrics has failed to advance at all in the last five years. The mere fact that Simmons, one of the world's most prominent sports columnists, has taken such an interest in our work is a remarkable sign all by itself. ESPN.com regularly features John Hollinger's work on not only its NBA page but also its front page. Basketball-Reference.com has emerged as a treasure trove of advanced historical statistics. We also have a better understanding of many aspects of the game. Thanks in large part to Witus, we know more about the critical relationship between usage and efficiency. We also now have several years' worth of adjusted plus-minus data, which has helped shed light on its benefits as well as its limitations.

In the context of that progress, let's return to those two key questions.

Can we boil a basketball player's value down to a single number?

I would answer this question by saying both "no" and "yes." Let's start with the "no." As Simmons points out, one of the biggest complexities for statistical analysts in basketball is the importance of fit. In some of his rare public work in the Journal of Qualitative Analysis of Sports, Oliver used the game of frescoball to illustrate how powerful fit can be in a team sport. This is especially problematic for those coming from the world of baseball, where you can reasonably look at a player's statistics and draw a safe conclusion about what his addition or loss will mean to a team. There's a desire for the same kind of tidy analysis of NBA trades or free-agent additions, but things are simply too complicated, as most recently illustrated by Elton Brand flopping this year in Philadelphia after people like me touted his signing.

Everyone involved agrees that it would be ideal to use a dynamic method of player valuation that takes into account a player's teammates and his role. Figuring out how to turn this into reality has proven much more challenging, and this remains an area with the potential for huge advances in our understanding.

At the same time, I'm also capable of arguing for single-number methods as more than a purely academic exercise. From the fan's perspective, especially in the cases of awards and All-Star teams, it's helpful to be able to make head-to-head comparisons. Using focused skill-based statistics like True Shooting Percentage and rebound percentage in these situations is a noble idea, but it's simply too difficult to mentally weight players' strengths and weaknesses against each other.

The problem isn't the existence of single-number methods; it's their prominence in the discussion. I wrote five years ago that too much time had been wasted arguing the relative merits of different systems, a debate that has only grown louder with the entry of the Wages of Wins metrics into the fray and PER's growing fame. Each system has its merits and its supporters, and certainly I think that my own WARP system is useful. However, the discussion tends to be charged and subjective when frankly I'm not sure the differences are all that noteworthy. Past a certain threshold of reasonableness, each method is probably about equally useful. The danger lies in the true believers who take any one of them to be gospel and ignore its shortcomings and blind spots. These systems are one tool to rate players, not an evaluation in and of themselves.

What better describes a player's value: his individual statistics or plus-minus data describing his impact on the team?

More than any other issue, I think this question divides the APBRmetrics community. There are zealots who believe individual statistics are entirely meaningless and a player's value can only be divined through adjusted plus-minus, others who are totally dubious of the inherent noisiness of plus-minus measures and even more scattered between those two extremes. I've struck a typically moderate position. As I noted in my follow-up to The New York Times' Battier story, I think there are elements of the game--primarily at the defensive end--that at this point can only be quantified using plus-minus data. At the same time, I take adjusted plus-minus from a single season or even combining multiple seasons with a shaker's worth of salt.

The question raised by both Beech and Simmons is whether tracking additional statistics outside the box score can achieve this same goal in a more reliable fashion. I certainly figured five years ago that game tracking would have made strides by now, but even getting data on a regular basis for something as simple as charges drawn has proven difficult. It would be nice to see the league take the initiative to begin counting some of these categories on its own, but to get detailed, freely-available information may require a volunteer effort along the lines of Project Scoresheet or the Game Charting Project run by our counterparts at Football Outsiders.

The more numbers we have at our disposal, the better, but I do see some problems with the "count everything" philosophy. The first is the issue of subjectivity. If we get to the point of trying to measure picks or assess the quality of a player's help defense, that is very different from counting deflections or charges. With a large crew of volunteers, subjective measurements create any number of potential headaches.

Valuing these new statistics is another issue. Take the popular "deflections." Is a deflection as good as a half a steal? A quarter? Three-quarters? These questions are nearly impossible to answer from a logical perspective. In fact, the best way to value them might be using adjusted plus-minus in the same way as Dan Rosenbaum did in creating statistical plus-minus, bringing the two schools together.

Ultimately, I feel like there are areas of the game that are simply too subtle to be quantified at the individual level, like a player's positioning. If being at the right place at the right time forces an opponent to pass the ball out for a more difficult contested shot, that's virtually impossible to track. Where it does show up is at the team level, which is why I feel that even with more detailed statistics at our disposal there will still be value to looking at plus-minus numbers.

As compared to five years ago, I'm less certain now where APBRmetrics is headed. In part, the statistical revolution, no matter the sport, is a generational issue. It's no surprise that in addition to Morey, the league's other young GMs--like Oklahoma City's Sam Presti and Portland's Kevin Pritchard--tend to be especially interested in synthesizing scouting and statistical information. Especially as the teams with analytics departments continue to enjoy success, the demand for analysts at the team level figures to grow steadily.

That will challenge the outside community to keep up. If we're to make more rapid progress in answering the important questions, the statistical community can't get caught up in the minutiae of pitting rating systems against each other. Innovation and ingenuity is needed. I'm optimistic, but now cautiously so.

Kevin Pelton is an author of Basketball Prospectus. You can contact Kevin by clicking here or click here to see Kevin's other articles.

0 comments have been left for this article.

<< Previous Article
Bracket Breakdown (03/26)
Next Article >>
Bracket Breakdown (03/27)

State of Basketball Prospectus: A Brief Anno...
Tuesday Truths: March-at-Last Edition
Easy Bubble Solver: The Triumphant Return
Premium Article Bubbles of their Own Making: Villanova, Temp...
Tuesday Truths: Crunch Time Edition

Bracket Breakdown: Memphis Is Under a Micros...
Around the Rim: The Wildcats Pounce

2009-04-03 - Every Play Counts: Blake Griffin
2009-04-01 - Marginal Dollars Per Marginal Win: Hoops Sty...
2009-03-30 - His Rightful Place: Goran Suton Leads the Sp...
2009-03-26 - Evaluating APBRMetrics: The State of the Art
2009-03-18 - Measuring Greatness: The Statistics Say...
2009-03-17 - Every Play Counts: Tyler Hansbrough
2009-03-16 - Every Play Counts: James Harden

Basketball Prospectus Home  |  Terms of Service  |  Privacy Policy  |  Contact Us
Copyright © 1996-2017 Prospectus Entertainment Ventures, LLC.