There are few things I enjoy more than thought-provoking writing, and that's what Phil Birnbaum delivered last week at his Sabermetric Research blog. Primarily a baseball analyst, Birnbaum has turned his attention to advanced statistics in the NBA from time to time. Recently, he's been considering diminishing returns to rebounding, a topic that is relevant to discussion of Dave Berri's Wins Produced metric because of the heavy weight it places on rebounding. In combination with the discussion here and elsewhere about Carmelo Anthony's influence on his teammates' shooting, this led Birnbaum to a provocative conclusion:
You know all those player evaluation statistics in basketball, like "Wins Produced," "Player Evaluation Rating," and so forth? I don't think they work. I've been thinking about it, and I don't think I trust any of them enough put much faith in their results.
That's the opposite of how I feel about baseball. For baseball, if the sportswriter consensus is that player A is an excellent offensive player, but it turns out his OPS is a mediocre .700, I'm going to trust OPS. But, for basketball, if the sportswriters say a guy's good, but his "Wins Produced" is just average, I might be inclined to trust the sportswriters.
I don't think the stats work well enough to be useful.
Birnbaum also noted that APBRmetricians would disagree with him, given that they--we--continue to make use of these box-score statistics. And he's right. But I think there is also plenty of room to agree with some of Birnbaum's points without reaching the same bottom line. Let's give them consideration.
Rebounding at the Margins
As noted, this whole thing started with the issue of rebounding and a question: Do good rebounders take boards away from their teammates? The preponderance of evidence seems to indicate that this is the case, especially on the defensive glass. Birnbaum has a series of posts looking at the issue, while Houston Rockets consultant Eli Witus used a slightly different method to demonstrate diminishing returns. In an entirely different manner, I've found the same thing when attempting to project team defensive rebounding as part of SCHOENE.
This theory is demonstrated anecdotally (go check out Ben Wallace's rebounding with and without the notoriously poor rebounder Clifford Robinson as his teammate) and is completely logical. There are many defensive rebounds--those off of missed free throws, for example--that could be grabbed by multiple defensive players and essentially come down to choice as much as skill.
That said, I'm still not sure this is meaningful for most players. It is only at the extremes, with specialists like Reggie Evans or dominant glass-cleaners like Kevin Love, that this really becomes a notable issue. In general, despite the issue of diminishing returns, rebounding tends to be one of the most stable statistics from season to season.
Using players from the last decade who played at least 500 minutes in consecutive seasons, I looked at the year-to-year correlation between performance in the two campaigns for the basic "skill" statistics that are used with SCHOENE. Rebound percentage has the highest correlation of all.
Stat R Stat R
TR% .946 PF% .845
OR% .931 FT% .832
Ast% .928 FTA% .817
3A% .919 Stl% .775
DR% .910 TO% .721
Blk% .900 2P% .584
2A% .890 3P% .358
Historically, year-to-year correlation has been used to gauge whether a statistic reflects a skill as opposed to something outside the player's control. I don't think that is appropriate, because random noise can overwhelm the underlying skill, as it does for three-point percentage (even limiting the pool to players with at least 100 three-point attempts each year, as I have; the same is true of free throw percentage). What this can tell us, however, is how much a player's performance in a given season tends to represent his true skill level as opposed to noise or interaction with teammates. Here, rebounding fares well--especially at the offensive end.
The counter-argument is that most players tend to stay on the same team, in which case we might not be able to isolate the impact of teammates. So the second cut of my study looked only at players who changed teams from one year to the next. While this lowers the year-to-year correlation, rebounding stays near the top:
Stat R Stat R
TR% .937 PF% .810
OR% .916 Usg .781
3A% .901 FTA% .761
Blk% .893 Stl% .721
DR% .891 TO% .665
Ast% .889 2P% .487
2A% .870 3P% .299
An interesting question to ask about these two sets of numbers is which statistics tend to be most affected when players change teams. That is, which saw their correlation drop the most from the first table to the second. As a percentage, here is that change:
Stat Drop Stat Drop
2P% -.166 2A% -.022
3P% -.165 DR% -.021
Usg -.099 3A% -.020
TO% -.078 OR% -.016
Stl% -.070 FT% -.012
FTA% -.069 TR% -.010
Ast% -.042 Blk% -.008
Shooting statistics, already unreliable from year to year, become even less predictable when a new team is added to the mix. Same with turnover percentage. By contrast, defensive statistics--including rebounding--seem to stay relatively steady. If players really had a significant effect on their teammates' rebounding percentages, I think we would expect to see more inconsistency from year to year--especially among players who changed teams.
But What About Shooting?
The more interesting aspect of Birnbaum's post to me was his quick study of the impact players' field-goal percentages have on their teammates. He found an increase of one percent to improve each teammate's shooting percentage by 0.26 percent, which is notable. This could be an artifact of the well-established relationship between usage and efficiency, or even the value I found (using adjusted plus-minus) to spacing the floor, but it is also possible this might be an indication of yet another way in which teammates affect each others' shooting.
In general, the attitude of the APBRmetrics community has been that defense is difficult to measure because so little is tracked but that we have a pretty good handle on how to rate players offensively. I'm beginning to question that assumption. It's based, I think, in part on the fact that the statistics we use for individuals perfectly measure offense at the team level--something that is not remotely true for defense. However, that doesn't necessarily mean that we are apportioning credit correctly between players. If a player's shooting percentage is highly dependent on his teammates--and there is some evidence for that when we consider the year-to-year correlations of shooting percentages for players who change teams--then players may be getting too much credit (or blame) for their own shooting.
We can account for some of these factors, as I do with the average tradeoff between usage and efficiency, but that method is a blunt tool for the job. Individual players tend to be affected by changes in their usage rates differently. Go-to guys, for example, usually have that role because they are not particularly sensitive to usage, which means they tend not to gain much efficiency when put in smaller roles (as in the case of LeBron James and Dwyane Wade in Miami).
Defending the Alphabet Soup
So where does that leave us with WARP, PER, WP, WS and the rest of the acronyms? Well, if you're putting your complete trust in any single statistic to measure player value, that is surely a mistake. Each metric has its own biases that can be seen most easily in comparison to the others. I believe that WARP does a better job of reflecting value than anything else; otherwise I would use the others. Yet I still blanch every time I see Jason Kidd ranked in the league's top 10 last season, which seems excessively kind. WARP tends to give too much credit to defenders who pile up steals and blocks while neglecting individual defense, so to consider them using only WARP would be a mistake.
The other useful reality check for me is net plus-minus and, with the appropriate caution, adjusted plus-minus. A couple of years ago, when I pondered the state of APBRmetrics, I argued that the choice between box-score statistics and plus-minus statistics divided the community. Increasingly, however, I see people using both in combination. When they agree, they allow us to make a stronger conclusion about a player. When they disagree, that's when basketball analysis becomes an art rather than a science. And that, to me, is the most interesting part of the whole process, especially when it becomes clear that there really is no such thing as player value in a vacuum. Everything is contextual based on role and system.
My biggest disagreement with Birnbaum is in his use of a study from the 2007 paper "The Pot Calling the Kettle Black - Are NBA Statistical Models More Irrational than "Irrational" Decision-Makers?" by David Lewin and Dan Rosenbaum of the Cleveland Cavaliers to support his conclusion. While part of the research by Lewin and Rosenbaum showed advanced statistics performing no better than minutes per game (with a team adjustment) at predicting future team wins, they later go on to find more positive results when it comes to explaining adjusted plus-minus.
In fact, Lewin and Rosenbaum ultimately drew precisely the opposite conclusion as Birnbaum despite the fact that their background lies more in the plus-minus realm than in the use of box-score statistics.
"The results of this paper should not discourage the use of statistical analysis in basketball," they write. "John Hollinger's PER metric, for example, performs quite well. The important thing to take away is that statistical analysis must be done carefully and rigorously, with an appreciation of the complex and dynamic interactions that are at the heart of the game of basketball."
That rings as true to me now as it did then. It's not that box-score statistics don't work in basketball. It's that box-score statistics by themselves are insufficient to describe the game. It's up to us to fill in the gap.
This free article is an example of the kind of content available to Basketball Prospectus Premium subscribers. See our Premium page for more details and to subscribe.
Kevin Pelton is an author of Basketball Prospectus.
You can contact Kevin by clicking here or click here to see Kevin's other articles.