I've been thinking a lot about player evaluation lately for a few reasons, including our Drew Cannon's excellent piece on the kerfuffle involving Anthony Davis' plus-minus statistics. A different question has really stuck with me. When I appeared on the ThunderGround Radio podcast a couple weeks ago, host Joel Ashbee asked me about the difference between looking at Nick Collison statistically and watching him on the court.
My answer was important not so much for what it says about Collison but because it explains a lot of how I evaluate players. You see, there are two very different ways to talk about a basketball player's statistics. One is to look at individual, or box-score statistics. While we've come a long way from the time when points, rebounds and assists per game were the three most important numbers, our advanced metrics are merely more polished ways of using the same information from the box score we've always had.
Nearly everyone at either extreme when it comes to using statistical analysis can agree that the box score is badly limited. More than 80 percent of defensive possessions are recorded in the box score by merely a defensive rebound or nothing at all in the case of a made field goal. Even on offense, the box score doesn't reflect the value of non-assist passes, screens and the influence shooters have on their teammates by spacing the floor.
To get a more complete picture of how players influence their teams, statistical analysts have turned to plus-minus statistics in various forms. These range from as simple as the player's plus-minus in a single game to as complex as adjusted plus-minus, which attempts to control for how the other nine players on the court influence a player's plus-minus with their own performance.
Among NBA general managers who value analytics, the phrase "eyes, ears and numbers" has become a popular way to explain how they balance scouting with statistical analysis. Since we aren't concerned with how a player fits into a team but simply the value of his performance, the "ears" portion isn't particularly meaningful here. Instead, I propose a different set of three data points: scouting, individual statistics and plus-minus statistics. Like eyes, ears and numbers, these three factors can be thought of as a Venn diagram:
Let's apply this to Anthony Davis:
Both scouting and Davis' individual statistics show him to be an impact defender, shown here in red. Davis ranks second in the country in block rate (the only player ahead of him, per Kenpom.com, is 6-6 Manhattan reserve Rhamel Brown), and qualitatively he's made life in the paint painful for opponents. Then we have Davis' net defensive plus-minus, which shows him to have a relatively small benefit to the Kentucky defense, even when adjusted for level of competition. Putting this category in blue is a bit of an exaggeration, since nothing indicates Davis hurts the Wildcats on D, but it's clear this evaluation doesn't quite match up with the others.
The easiest scenarios to understand, naturally, come when all three methods of evaluation line up. All available evidence suggests that Dwyane Wade is one of the top five players in the NBA. Going the opposite direction, both kinds of statistics and the eye test agree that Stephen Graham was one of the NBA's worst players last season. Most of the time, it's not so easy, which is both frustrating but also why I find basketball analysis so interesting. There is truly an art to understanding why the various measures disagree and what that means.
Let's go back a second. By drawing the Venn diagram as I did, I made the misleading implication that all three measures are equally meaningful. This is not the case. How relatively important are they? That depends, which is not a copout answer but a recognition that the quality of the information we have in each category varies widely.
Returning to Davis, this is a crucial part of the discussion. To me, the most obvious explanation for Davis' relatively poor net defensive plus-minus is the small sample size. At the time SI.com's Luke Winn wrote about this statistic, Davis had been off the court for 202 possessions--less than three games' worth. That's just not nearly enough time to make meaningful declarations. Even the entire NBA regular season, nearly three times a long, is not sufficient for the noisiness in plus-minus to filter out. There are plenty of examples of players' net plus-minus ratings bouncing wildly from year to year.
This doesn't invalidate plus-minus statistics. It merely means they must be used with more caution than individual numbers. To be clear, Winn didn't use plus-minus to say anything negative about Davis. He merely made note of the plus-minus numbers in the process of pointing out how effective Eloy Vargas has been defensively as Davis' backup.
Collison provides a different example. His Venn diagram is the mirror image of the one we showed for Davis' defense:
In terms of individual statistics, Collison doesn't impress. Because he uses so few possessions on offense and rarely blocks shots, Collison rates worse than replacement in WARP and little better in PER. Basketball-Reference.com's Win Shares provide a superior estimate of Collison's value but still put him barely better than average.
Meanwhile, Collison's net plus-minus of +11.1 last season ranked eighth in the league, per BasketballValue.com. Every player ahead of him was an All-Star. The year before, the Thunder was 9.4 points better per 100 possessions with Collison on the floor.
In this case, the issue isn't one of reliability. Collison's individual stats are accurate to the extent they describe his skills. However, they are incomplete. Like Shane Battier, another plus-minus All-Star, Collison excels in the areas of the game that aren't tracked in the box score. He's a terrific post and position defender, draws charges by the bushel and sets good screens on offense. In this case, watching Collison play explains the discrepancy. We can also support that observation using supplementary statistics, like charges (according to Hoopdata.com, Collison led the league by drawing 57 of them in 2009-10), Synergy metrics and information from StatsCube.
We could come up with any number of additional examples to show possible outcomes of this model of player evaluation and how it depends upon the available information. In a single game, for example, scouting should get more consideration than the other two factors. As Dean Oliver once aptly noted, "Individuals see a basketball game better than the numbers, but the numbers see all the games."
The crucial takeaway is that none of the evaluation methods is sufficient on its own. Plus-minus statistics are often unreliable, individual statistics are incomplete and we can be fooled by what we see (or don't see). We've long moved past the outdated notion that scouting and statistics are opposed to each other. Instead, at their best, they work together to help us form more robust evaluations that are more likely to prove correct.
Get your copy of the most comprehensive guide to the new NBA season. Pro Basketball Prospectus 2011-12 is now available in .PDF and paperback format.
Kevin Pelton is an author of Basketball Prospectus.
You can contact Kevin by clicking here or click here to see Kevin's other articles.