The ‘advanced’ stat in this article from Nate Silver, is Wins Produced. And here is what Silver argues Wins Produced is missing.
Popular VideoPeople were so furious about this Pepsi ad that Pepsi pulled it after just one day. Watch it here and decide if it's offensive:
What is missing from formulas like Berri’s is an account of what Anthony does to the rest of the Nuggets. Because he is able to score from anywhere in the court, Anthony draws attention and defenders away from his teammates, sometimes leaving them with wide-open shots. He also allows them to be more selective about the shots that they choose to take, since they know that Anthony can usually get a respectable shot off before the 24-second clock expires if needed.
Silver goes on to provide evidence – detailed in the following table — supporting his contention.
Popular VideoPeople were so furious about this Pepsi ad that Pepsi pulled it after just one day. Watch it here and decide if it's offensive:
And then Silver argued…
The effect of a player who improves the rest of his team’s TS% by 3.8 points is extremely substantial: it is works out to their scoring about 5 or 5.5 additional points per game solely on the basis of this efficiency gain. That, in turn, translates into about 15 additional wins per season for an average team, according to a commonly-used formula. This is how Anthony creates most of his value — not in the shots he takes himself, but in the ones he creates for his teammates – and some of the “advanced” formulas completely miss it.
Wins Produced argues that Carmelo Anthony has produced 33.5 wins across his eight seasons. And that means Anthony is hardly an elite player. But Silver argues that Anthony –because he increases the shooting efficiency of his teammates — is worth 15 additional wins per season. So Melo – according to Silver’s analysis – is clearly elite.
Silver’s argument certainly reflects conventional wisdom and it appears supported by some empirical evidence. Unfortunately – as people have noted since Silver’s story appeared – there are some problems with this analysis.
Before I get started, let me first apologize for the length of this post. Explaining the issues with this analysis takes more than a few words, so this post may take up some of your time (that is, if you wish to read all of it).
And before I get to the problems, let me also note two basic issues one needs to think about in considering such a study. The first is statistical significance. Or more simply, can we confidently declare the estimated relationship actually exists (i.e. is different from zero)? The second issue is economic significance. Or (again) more simply, what is the size of the estimated effect?
If we determine that the estimated relationship probably isn’t real (i.e. isn’t statistically significant), than the second issue isn’t important anymore. And as I will note, I do not think this relationship actually exists. Nevertheless, I want to start by noting that even if one insisted that Silver’s simple approach was ‘best’ (as I will note, this approach is not the ‘best’), his calculation of the size of the effect failed to take into account two obvious differences in the players included in the sample.
Calculating an Average
The reason I want to start with the size of the effect is that I think Silver does something ‘odd’ in his calculation. The table above presents the change in TS% for sixteen players. Some of these players – like Marcus Camby, Nene Hilario, and Kenyon Martin – have played more than 10,000 minutes with Carmelo Anthony. Others – like DeMarr Johnson, Voshon Lenard, and Aaron Afflalo – played less than 3,000. Given these difference in time spent with Melo, one might expect the analysis to take this into account. But from what I can tell, all Silver did was calculate the simple average change in true shooting percentage. So whether they played 10,140 minutes (like Kenyon Martin, whose TS% appears to improve 1.8% because of Melo) or 3,276 minutes (like Greg Buckner, whose TS% appears to improve 8.9% because of Melo), the impact each player’s change has on the overall Melo Effect (the 3.8% impact estimated above) is the same. One would expect, though, that the analysis should at least present a weighted average. And if we weight these numbers by minutes played, the reported effect does fall to 3.5%.
This is not much of a decline.
Then again, weighting by minutes is not the best approach. This issue here is shot attempts, so a better weighting scheme is to adjust by how many field goals a player is taking. I didn’t feel like gathering all this data, but I would note that the three of the four players with the largest Melo Effect – Anthony Carter, Greg Buckner, and Chris Andersen – are not known for taking many shots. Specifically, in the eight seasons where these three players appeared in at least 50 games, none of these players ever averaged more than 6.8 field goals attempted per game for a season [6.8 is the mark Carter posted in 2007-08]. And in five of these seasons, the field goals attempted per game was 4.3 or less.
Again, according to Silver, this trio has three of the four largest Melo Effects. But even if we could argue that increased shooting efficiency we observe for these players is entirely about Carmelo Anthony (and again, I will note in the moment that this is unlikely), if these players don’t really shoot much then the change in shooting efficiency noted can’t matter much. Given this observation, one might expect a simply adjustment for the number of shots each player takes. But again, all we have is a simple average.
Is it All About Melo?
Unfortuntely, even if the weighting of the average was correct, there is a much bigger issue to consider. As a number of people noted, Silver’s analysis doesn’t consider any other factors. He argues that the changes we observe in each player’s TS% is entirely about Carmelo Anthony. But player performance could change for other reasons. And because other factors could matter, the analysis of the Melo Effect is incomplete – and quite misleading – if no effort is made to control for the other factors.
To illustrate this point, let’s briefly talk about the study I published on NBA coaching (with Michael Leeds, Eva Markova Leeds, and Michael Mondello). The purpose of this study (discussed in Stumbling on Wins) was to examine how coaching impacted player performance. At the onset of the study we first report how player performance changes when the player comes to each coach in our sample. This analysis did not initially consider any controls. And the coach that we report having the largest effect was Dan Issel. Of the fifteen players who came to Issel, twelve posted higher per-minute performance.
If we were following Silver’s example, we would have stopped at this point and declared Issel the greatest NBA coach [across our sample from 1977-78 to 2007-08]. As we note in the paper, though, other stuff matters. And when you control for past performance, age, injury, etc…) the impact of Issel vanishes (i.e. Issel’s impact was not statistically significant) and the top coach – according to our analysis – is Phil Jackson.
Of these ‘other factors’, age appears to be one factor Silver should have considered (and people noted this issue as well). After Buckner, Carter, and Andersen, the top seven players in the Melo Effect rankings includes J.R. Smith, Nene Hilario, DerMarr Johnson, and Aaron Afflalo. Here is how old each player was when he first became Melo’s teammate.
Smith: 21 years
Hilario: 21 years
Johnson: 24 years
Afflalo: 24 years
Player performance in the NBA – as reported in Stumbling on Wins – tends to peak in the mid-20s. So each of these players was at an age when improvement in performance was still likely to occur. To estimate the size of the Melo Effect, the impact of age needed to be considered.
And that is what I attempted to do. Utilizing the same data set employed to study coaching [i.e. data on players from 1977-78 to 2007-08], I looked at the factors that explained a player’s TS%. The explanatory factors I considered included past TS%, age, game played (to capture injury), etc…. In addition, I considered a dummy variable, equal to one if a player was in his first year as Carmelo Anthony’s teammate. If the estimated coefficient for this dummy variable is statistically significant (and positive), then we can conclude that Silver is on to something. When the model was estimated, though, the Melo dummy variable was clearly insignificant. In sum, it doesn’t appear that a player’s TS% — when we consider a number of factors that impact player performance – is impacted by joining at team with Carmelo Anthony.
One should note that even if the estimated coefficient was significant we still wouldn’t have been able to conclude that there is a Melo Effect. Again – as people noted – Melo is not the only factor unique to Denver. The Melo Effect – if it existed – could have been the George Karl Effect. Or it could have been the Dean Oliver Effect (Oliver is the author of Basketball on Paper and he does statistical analysis for the Nuggets). Or it could have been the altitude in Denver, or any other factor unique to Denver.
Although this exercise failed to uncover evidence of a Melo Effect, it does serve to highlight an important point about statistical analysis. If we wish to understand how one factor impacts another, an effort must be made to control for other explanatory factors. Silver’s analysis didn’t control for anything. As a consequence, his estimate for the existence and size of the Melo Effect appears to be incorrect.
Quoting from Others
As noted, I was not the only one to note problems with this analysis. So let me close by noting some of the other points people have made (some of this echoes what I said above).
Dre — Silver and Gold: Prospecting Melo’s Past — looks at Melo’s history in Denver. And Arturo — in Fanservice: Followup notes on Melo, Rookies and A simple response to Mr. Silver – looks at how TS% for players in Denver changes with and without Melo. For this interested in more on this topic, these are excellent reads. Dre’s point that Denver’s success with Melo is not all about Melo is especially important.
Beyond these posts, let me also reports some of what I have seen in the comment section (and in the interest of space, these are all just partial quotes; please read the comment section for all that people had to say):
- Don’t you need to look at a bigger population of scorers who might make their teammates better before coming to this conclusion? If you examine n guys who fit the Melo profile and the results in general argue against this teammate effect then isn’t this effect likely to be happenstance (or unpredictable)? And then what about the same effect from lesser (presumably more numerous and cheaper) versions of Melo, so that we can determine the marginal value? What about the effects of Melo’s backups with Denver over all these years? This analysis has all the drawbacks of plus-minus. Of course if the results from a statistically significant sample size then confirm this effect then it’s certainly a valid statement.
- I’m stunned at this particular statement [“In taking all of those shots, however, Anthony has also done something else: he’s made his teammates much more efficient offensive players”]. How can he make a statement on causality based upon the evidence that he presents? It would have been so much better for him to just say: ‘When Anthony is on the floor his teammates are much more efficient offensive players’ and then let his audience make of that what they will.
from John Giagnorio
- Why does Anthony get credit for improving his teammates even when he is not on the court? Was it that difficult to break down the data further?
- Why use TS% instead of eFG%? Does Anthony deserve credit for his teammates becoming better free throw shooters? Look at Iverson’s age 22 and age 32 seasons. The eFG% is identical, but he’d become a much better foul shooter.
- Take a look at Nene’s career on basketball-reference. He’s played all of 1 year without Anthony, yet his TS% didn’t really improve until 08-09.
from Italian Stallion
- High usage scorers that often get doubled should be in a position to get a lot of assists, but Melo doesn’t.
- The article gives all the credit for the improvement in the TS% of his teammates to Melo when it could easily be partially be Billups (an underrated PG), better coaching, a combination of players, or random.
- IMO he should not have compared a players lifetime TS% to his TS% with Melo because the one thing most good players do over time is improve their shot selection and shot. So most good players improve their TS% also as they develop. He is probably giving Melo credit for the natural improvement of the players.
- TS% is only a part of what determines wins, acquiring the ball and preventing your opponent from doing so also are important, or so I’ve read in one of the basketball blogs that I frequent. So even if had demonstrated that Anthony improves his teammates’ scoring efficiency, it’s a narrow view; what if Melo’s teammates are rebounding less and turning the ball over more?
- While I agree that it’s pretty laughable that Melo could impact his teammates’ FT% (what, does he give better high-fives between shots?), he could conceivably improve their FT rate by passing them in a situation where they are more likely to be fouled while shooting. This would improve their TS%, though not their EFG%. However, Silver has failed to show any correlation between playing with Anthony and an increase in FT rate, much less a causative link.
from Peter (commenting at Nerd Numbers)
- Even with the changes in true shooting percentage, as a stat major, there is also the concept of statistical significance. In a nutshell, yes, most of the players that played with Anthony improved their true shooting percentage. But it is possible that, at least for some players, it is highly likely that their improvement is not “significant”, that is, they could have had that performance with or without Anthony based upon measuring their previous performance. And if they could have had those performances with or without Anthony, then Anthony was not the reason why they shot so well.
- Expounding on the previous point, two of the players with the biggest gains, Nene and J.R. Smith, played some of the fewest minutes of the group before Anthony arrived. Even with their great gains, at least some of their improvements may have been age-related, not Anthony-related.
- The author only addresses shooting. Winning basketball games entails great shooting, obviously, but it also requires rebounding, assists, etc. When you look at Anthony’s non-scoring statistics, for example, he is below average with respect to the average shooting guard in turning the ball over and fouling, which are activities that do not help the Nuggets win, let alone help his teammates shoot. The author even admits that Anthony’s assists are below average with respect to other scorers such as Bryant and James. Besides, the reasoning behind Berri’s metric is that scoring *isn’t* all that there is to winning games, and as such, he tries to reward players who contribute in ways other than scoring, which hurts Anthony.
When we look at these comments we certainly see some similarities. A number of people have noted that Silver failed to show causality (so he overstated his case), failed to show statistical significance, and failed to control for other factors (like age).
Let me close by repeating something others have noted. During the past two elections I have enjoyed reading Silver’s analysis (and the analysis from other people at FiveThirtyEight). And I want to emphasize that what ever you think of Silver’s analysis of Carmelo Anthony, the analysis of the Melo Effect doesn’t tell us anything about the quality of analysis offered on other topics. In other words, it is incorrect to argue that because Silver may not have gotten this story right, all the other stories he tells also have problems. Such an approach would be drawing an inference from a sample of one. And yes, a sample of one isn’t statistically significant as well.
P.S. Again, sorry for the length of this post. It is more than 2,700 words and if you got to the end… well, I am not sure this was the best way to spend your time. For my next post I will try and say less (and I hope not to use the word “Melo” at all).