Monday, October 15, 2012

Jose Calderon: An Example of How Wins Produced Fails

Update: See similar articles for Boozer here and Rondo here.

Wins Produced, a model by David Berri, attempts to capture player value through box score stats by assigning each measurement a number in terms of how many wins it adds to a team. There are numerous problems with the method, some of which I described here, but despite its rejection in most NBA stats circles (they have the best parties) it's heavily supported by TrueHoop, whose influence and view count allow the flawed stat to be marketed heavily toward a large audience. This isn't a pointless issue because for many people it may end up being one of their first experiences with advanced stats in the NBA, which is negative for the advanced stats movement. I realize that all models have their flaws; one of the main issues I have is the author's, and his cronies', insistence that it is nearly perfect and they regularly disparage other analysts.

The method of Wins Produced overvalues efficient shooters, high rebounders, and players who keep turnovers low, while they undervalue shot creation and defense. If you shoot a percentage below their league average (with positional adjustments), then your Wins Produced number is below average, even if you're the only player on the team who can create a shot. There's no context or nuanced understanding of the game. Last season for instance James Harden "created" more wins than Dwight Howard and the year before Landry Fields somehow was rated higher than Tyson Chandler, Ginobili, Blake Griffin, and Durant.

But who's the prototypical example of a player overrated by Wins Produced?

How about someone who was apparently the fifth best player in 2009 behind only Chris Paul, James, Howard, and Billups? (Hm, one of those four is not like the others.) It's yet another point guard ... Jose Calderon, who somehow put up 14.9 wins for a Toronto team with only 33 wins. This was when they still had Chris Bosh, so it's not like they were so completely devoid of talent that without Calderon they would have been atrocious.

This is someone who's lead the league multiple times in assist to turnover ratio, some years over 5. He's broken the single season free-throw percentage record at 98.1%. He joined the 90-50-40 club (though he didn't have enough attempts.) He's had years of accuracy from all over the court only bested by Nash. However, he's also one of the worst defenders in the league, especially after injuries piled up, and he doesn't shoot enough that he can be considered a star given that the only other thing he can do is pass. He's a low mistake point guard whose prowess on the offensive end is nearly erased by his lapses on the other end. So how would they play without Calderon, who according to Berri and friends should have been an all-league, multiple time all-star?

Unfortunately, Jose's never been traded, so we can't see if there's a LeBron-leaving-Cleveland type of situation on a smaller scale, and he played all 82 games that year. Luckily, he normally misses 10 to 20 games a season. Obviously, there are more advanced methods to estimate player value, but one focusing on the performance of a team when someone plays versus when he misses a game is simple and transparent, allowing anyone to understand the numbers.

Jose Calderon in 2008-09

Playing alongside Chris Bosh, Jose Calderon averaged 12.8 points, 8.9 assists, 2.1 assists, and 34.4 minutes per game on stellar percentages of 61.3 TS%, 49.7% from the floor, and a record-setting 98.1% from the line. Toronto, however, finished 33-49 behind three teams in their own division. He missed 14 games, lending to a nice little sample of games to analyze. Wins Produced says he was worth 11.6 wins at 0.239 wins per 48 minutes where his rank that year was 14th and 11th, respectively (among players with at least 1000 minutes.) That is definitely all-star worthy, and if he misses a game Toronto should predictably perform worse. A win per 48 minutes level over 0.200 is elite where 0.100 is "average". What happens in those 14 games when he's out?

Toronto was 5-9 when Calderon didn't play, and 28-40 with him. That's a win percentage of 35.7 versus 41.2, meaning they won more games with him. Do we drop the conclusion there? Of course not. It's a terrible small sample size and the difference isn't huge. If they had gone 6-8 without him, just one more game, that percentage moves from 35.7 to 42.9. Also, the competition was a bit steeper during those 14 games. Once you make some adjustments and look at the numbers in the table below, in Calderon's second best season ever his team, a fairly crappy team, performed better when he was off the court. That is not congruent with a player apparently worth about 12 wins over the season.


Record
Win%
Margin of victory
Off. rating
Def. rating
Strength of schedule
SRS
Wins Produced
With Calderon
5-9
41.2
-2.99
106.5
109.8
-0.55
-2.94
11.6
Without Calderon
28-40
35.7
-2.00
109.5
111.2
1.45
-0.55


The most important column is SRS. It's a basketball-reference measure in terms of points that includes margin of victory, strength of schedule, and home-court advantage. An average team is 0, a very good one +5, and a poor team -5. For example, during the 2009 season there were five teams with an SRS -5 and below and five teams at +5 and above with the best mark at +8.68 (Cavaliers) and the worst -8.59 (Kings). Homecourt advantage is assumed as a +3.2 advantage for calculations.

Once more comprehensive information is included, the results are rather damning. Against a tougher schedule without their starting point guard, the Raptors improved their average margin of victory and consequently their simple rating system score (SRS). Notice that despite his stellar shooting their offense actually improved when he wasn't in the rotation. One piece of evidence against his favor is that his team had to use their backups to replace his production, which is harder to do at point guard where the guy you're replacing runs much of the offense averaging near 10 assists a game.

If you think there are extenuating circumstances and shouldn't be used to discredit Wins Produced, don't worry because there are many other seasons to consider.

Jose Calderon 2006 to 2012

Excluding the year he played every game and his rookie year (adjustments during the first year are notorious, especially coming from a Euro league, so it wouldn't be fair to include it), Calderon's results are shown below. The team's win percentage is noticeably better when he's not playing, and the effect appears to be on both the offensive and defensive ends of the court. The SRS, again, is a rating system that adjusts for your opponents, home court, and by how many points you beat a team; it's like an adjusted average margin of victory.

Season

Record
Win%
Margin ofvictory
Off. rating
Def. rating
SRS
Wins Produced
Total
With Calderon
115-219
34.4
-3.75

105.1

109.2

-3.87

38.7
Without Calderon
26-34
43.3
-2.13

106.6
108.8

-2.05

2006-07
With Calderon
3-2
60.0
-3.35

100.8

104.6

-3.79

6.5
Without Calderon
20-57
26.0
-2.60

100.7

103.2

-1.83

2008-09
With Calderon
5-9
41.2
-2.99
106.6
109.8
-2.94
11.6
Without Calderon
28-40
35.7
-2.00
109.0
111.2
-0.55
2009-10
With Calderon
9-5
64.3
-2.31

111.7

114.3

-2.04

5.4
Without Calderon
31-37
45.6
0.79

109.1

107.8

-0.80

2010-11
With Calderon
4-10
28.6
-6.41

105.3

112.3

-6.89

7.4
Without Calderon
18-50
26.5
-5.64

108.6

114.8

-3.32

2011-12
With Calderon
5-8
38.5
-3.75

100.7

104.9

-3.65

7.6
Without Calderon
18-35
34.0
-1.46

101.2

103.0

-3.74



It's best to view the table as paired data within each season where a gray row is for when Jose "Wins Produced Star" Calderon plays and checking the cell below in the white for when he doesn't play. There was only one season where they played better with him on the court, and it wasn't by much; Wins Produced that year basically considered him a near all-star while his effect on the game was middling. Be careful, however, looking at the yearly results because with the small number of games the numbers are prone to noise and chaos, but the aggregated totals have enough of a sample size for some trust. And the results aren't good.

Synthesizing the Information

But how do you estimate how Toronto should have played without Calderon? Well, first assume that Calderon is being replaced by an "average" player; given the backups they normally employed this is generous and a conservative assumption that helps Calderon look better. Use a Wins Produced per minute number and his average minutes per game for the season to estimate the "wins" lost. For a statistical test, a standard deviation is needed, and unfortunately that's a difficult measure to find for such a specific use. What I used was the standard deviation for Russell Westbrook's Wins Produced each game in the 2011-12 season, which the author claimed was a fairly high variance. I realize that's a hefty assumption, but I'm trying to err on the side of caution here and I'm also assuming his backup is an "average" player. If anyone has a better number or wants to calculate the standard deviation for Jose Calderon in each season tested, go right ahead. The problem is that given the standard deviations I've seen the gap is too large to cover.

Actual wins without Calderon
Wins Produced predicted wins
Standard deviation of wins produced
t-test 
p-value
26
17.4
0.24
1.08E-41


The p-value is saying there's an extremely low probability that by mere chance the Wins Produced predicted wins number just happened to be that much lower. Think about the standard deviation as a naturally occurring spread of numbers. I'm looking at each game Calderon missed and trying to estimate whether there's a win or loss by the team's win percentage with Calderon and Win Produced's guess at the average "loss" in wins each game from losing him. The standard deviation then should be rather low because it's the Wins Produced each game. It's important to consider because natural variation of his score can skew the results by sheer chance, but again the gap is so big between the actual wins they had in the 60 games without Calderon and the predicted wins from Wins Produced that I'm comfortable in stating there appears to be statistical significant difference. 

Wins Produced is unsuccessful at explaining the value of Jose Calderon.

Conclusion

You can play around with the numbers all you want, but over 60 games the evidence is hard to ignore: even on a crappy team like the Raptors, Calderon is is a negative force, not an all-star point guard like Wins Produced suggests. If you think it's unfair to judge just on the games he missed and there could be complex teammate effects behind everything you can take a look at his yearly ridge-regressed +/- score or his current ranking; or maybe you can cross-reference another adjusted +/- model that also states he usually has a negative effect; or hey here's another one; and if you think yearly models are too noisy (though one link has a two-year model) here's a six-year study. If you don't want to click through those links, here's the summary: he's a below average player according those methods normally in the -1 to -2 range; the six-year study is particularly alarming where it says he's a -1.69 because it's hard to explain away any bad results when many seasons are included. 


With most models, authors are hesitant at declaring complete confidence in the results, but the fun with the Berri crew is their cult-like insistence at the perfection of their magic recipe. Here's where Berri implied Calderon should have been considered for all-star and MVP voting in 2008. Here's an article where he says Calderon was better than Bosh. There are also many arguments like this in the comments section where the adherents try to ignore any criticism by how, hey, they used numbers, so of course it's infallible. The most robust argument I've heard, which I've made in a previous article, is that Wins Produced derives wins from offensive and defensive efficiency (and possessions), and then assigns a piece of that win to each box score stat. Then they claim validity by summing Wins Produced and saying, Hey! Look at how close it is to the team's actual wins! The problem is all they're doing is stating team off/def efficiency correlate well with wins, and there's no proof offered at how that applies to the players. They give all the credit to the rebounder and none to the wing defender, unless he gambles and gets a steal.

When Calderon played over five seasons his team's SRS (adjusted point differential) was -3.87, but when Calderon wasn't playing during those same seasons his team's SRS improved to -2.05. He's an amazing shooter who can shoot like Nash and pass and protect the ball like Chris Paul, but his defense is atrocious and he doesn't shoot enough ... or appear to have any positive effect on the basketball court. And ask yourself this. Do you really think box score stats are all you need to understand the game (or specifically player value)? And that the stats completely cover all ends of the court? Try tracking a play and see who's involved, what passes are made, the picks, the boxing out, and then realize you can only give credit when someone completes an action recorded in the box score sheet. I'm sorry, professor Berri, but Jose Calderon is not worth 12 wins in a single season.

Note: good discussion of Wins Produced here

17 comments:

  1. Excellent article, I am a wins produced devotee, so to see the numbers analyzed from a different perspective is both healthy and refreshing. One small criticism is that people always discuss the idea of "shot creation" which WOW often dispels. You state that Calderon is a low turnover player but not a shot creator. My question is that isn't a low turnover player by definition a shot creator? Not turning the ball over allows your team to shoot, pure and simple. The quality of the shot can be debated, but any possession that does not end in a turnover ends with a shot.

    Also is it possible that Calderon is so bad at defense that he is an extreme outlier? Could this same analysis be done with other WOW stars such as Kevin Love, Kenneth Faried, or Andre Kirilenko. Those players are average to above average on defense so in theory wins produced should in theory either properly or undervalue value their efforts.

    ReplyDelete
    Replies
    1. I only mean shot creator in the strict sense: he doesn't have a lot of FGA's per minute. I wish he did because he's an amazing shooter. But yes, keeping your turnovers down is an underrated feat, and few know just how proficient Calderon is at this.

      I would like to do another one like this. Calderon is an extreme case in that he's great at what Wins Produced focuses on, but downright terrible in what it ignores like man-to-man defense. Kevin Love would be interesting, but you probably couldn't do it until after this season because you need enough missed games to have a real sample size. Keep an eye on how the Wolves play without him to start the season. It won't completely prove how valuable he is, but it's worth noting.

      Delete
  2. It looks like you have your with and without Calderon rows the wrong way...based on that it looks like for 4 of the 5 years the SRS shows the Raptors are better off with Calderon, and in the 5th year it is marginal. How have the overall totals been calculated? It looks like you have just averaged each of the seasons without accounting for number of games in each of the seasons which would surely affect the maths.

    ReplyDelete
    Replies
    1. In the same table near the top is a totals row. There's also a single statistic for everything in the "Synthesizing" part of the article below that. I didn't average by year; I based the averages on the number of games.

      Delete
  3. Nice work here. The curious case of Jose Calderon should be enough to convince most people that wins produced is more of a fun distraction than the bible of player value its adherents insist that it is. If you're looking for another player to take a look at, I would suggest Marvin Williams. WOW insists he was an "all-star" caliber player for much of his career and "the Hawks best player" last year.

    As a person who has watched most of Marvin's professional career either in person or on television, I can shoot a million holes in such statements, but I'd be interested to see what your numbers show.

    ReplyDelete
  4. Your analysis fails to take one thing into account, while WOW says that Calderon is amazing, it also says that another player that Raptors have employed since 2006-07 and given heavy minutes has been downright TERRIBLE! That player is Andrea Bargnani. See his career stats from a WOW perspective here: http://www.thenbageek.com/players/314-andrea-bargnani

    Bargnani has managed to earn his team -18.8 wins since he was drafted. Also in the '08-09 season you mention, 2 of the top 5 minute receiving players produced NEGATIVE wins (http://www.thenbageek.com/teams/tor?direction=desc&season=2008&sort=minutes). It is hard to win a lot of games when 2 of your top 5 most played players are costing you games not just compared to average players but compared to players who produce 0 wins. Had they replaced Bargnani and J. Kapono with players who produced 0 wins (not even average players) the Raptors in theory produce 4 more wins and had they replaced them with average players(who played the same minutes) they produce 8.8 more wins. With average players the Raptors win about 42 games that season which is good for the 6th seed in the East that year (http://en.wikipedia.org/wiki/2008%E2%80%9309_NBA_season).

    So while you may not believe that Calderon is as good as advertised it may help to look at who he plays with when citing a team record. No matter how good someone plays, it does not matter if there is no one around them is playing well: see Kevin Love 2010-11 (http://www.thenbageek.com/teams/min?direction=desc&season=2010&sort=minutes).

    ReplyDelete
    Replies
    1. I'm sorry, but your cult leader for WOW fails to take into account 90% of defense and indirect effects on offense.

      Here's one question:
      Do you really think individual defense is ENTIRELY covered by rebounds, blocks, fouls and steals? That's delusional. Every analyst with his own stats mentions the limitations. Hollinger will state PER doesn't cover defense well. Plus/minus guys will say there's a lot of noise and you need lots of samples.

      What WOW does is put a huge amount of credit into rebounding. That's Bargnani's weakness, and it's why he's so negative. Yes, gaining a possession is good, but you don't completely get that possession from a rebound. That's stupid. Someone has to defend the other player to force a miss, and he gets no credit.

      Sure, plus/minus is noisy, but if you do, say, a model with nearly a decade you can get some pretty good results, and Calderon still isn't the positive force WOW makes him out to be.
      http://godismyjudgeok.com/DStats/2012/nba-stats/quick-chart-aspm-vs-rapm-2003-2011/

      You also didn't read this article very closely. I'm not simply using team record. I'm using team record when he plays versus when he doesn't. That's very different. The fact that he has a bad roster doesn't affect that.

      Delete
    2. Having a bad roster does affect that. Who was the backup PG? Was it a steep drop-off in talent, or none at all? You can't look at on-court/off-court data without taking the other specific players into account. That's why +/- struggles with small sample sizes. You have to control for all the other variables (aka players).

      That said, I agree with your point about steals, rebounds, blocks and fouls (or lack thereof) not being the sum total of defense. It's easy to think of additional defensive stats such as passes denied, post position refused, screens fought over, shot attempts prevented, shots altered, tipped balls, fouls that result in free throws vs. layups, etc. Do you know of anyone who is collecting them?

      Thanks for the article.

      Delete
    3. The analysis is basically sound. Or at least it is if we assume the the roster is otherwise the same whether or not Calderon is on the floor.

      The Raptors played better when Calderon sat than when Calderon played. Bargnani played with the Raptors when Calderon sat, and Bargnani played with the Raptors when Calderon played.

      If Calderon played on a team with LeBron and CP3, and that team played better when Calderon sat than when he played, the conclusion would be the same: Calderon's team plays better when he sits than when he plays.

      Delete
  5. I agree that WOW is not properly acknowledging defense. I think what is really missing from their formula is a spatial analysis. That being said... Wins produced is incredibly better than PER. If you have to chose one of the two go Wins produced every single time. Taking more shots (upping you usage) does not make you better if you don't hit them!

    ReplyDelete
    Replies
    1. That being said, having the skill to create a low-percentage perimeter shot at the end of the shot-clock is far more valuable than the alternative (letting the shot clock expire; or as a big-man that can't necessarily create their own shot, having their shot blocked or otherwise disrupted by a collapsing defence with the shot clock running down).

      WOW doesn't account for this (I'd be interested to see the percentage of possessions where the defence forces a shot in the last 5 seconds of the shot-clock) - in the WOW world, there's no difference between the opportunity of Tyson Chandler catching an oop 12 seconds into the shot clock and that of DeMar Derozan trying to bail out a well-defended possession via a high-skill (albiet necessarily low-efficiency) turn-around jumper.

      Delete
  6. The new trade will be an interesting test of the Calderon effect and the WOW formula. I hope you do a follow up at the end of the season.

    ReplyDelete
  7. I think WP is the best overall metric if you look at it with some common sense, of course.

    5 Centers can't win a basketball game, but you'd rather have a point guard that's athletic enough to rebound and pad his WP than not.

    Overall, picking 1-5 (pg - center), I think you'd be better off with the top 5 in WP than the top 5 in any other advanced stat.

    For example I think if you put Calderon next to a strong wing defender at the 2 or 3, and a shot blocker at the 5, his record would match his WP a little more. Any stat is just a part of an overall system of thought.

    ReplyDelete
  8. WP is the worst stat I've ever seen. It's absolutely pointless, and if it was so valuable, then why don't NBA people use it? I don't have a problem with developing advanced stats, but to think they can accurately rate every player to a decimal point is absolutely ludicrous. Berri thinks it's pointless to even watch the games, and instead looking at a spreadsheet tells him everything he thinks he knows. They think rodman is better than jordan, and matt barnes is better than kobe, just 2 of an endless # of awful analyses.

    It's refreshing to see an article like this trying to dispel these blatant fallacies.

    ReplyDelete
  9. Dude, before you post a table, at least make sure that you didn't get the row labels messed up. Based on your values, if the labels had been correct for with/without, your explanations are screwed big time, 'cause the Raps look better with JCalderon.

    This 3-team trade with Calderon to the Pistons should hopefully clear things up (if a good sample size can be drawn). Will be checking back on the site by season's end. Interesting stuff here.

    ReplyDelete
    Replies
    1. It's not just the row labels that are messed up...there is a 5-9 record lined up with a 41.2% W-L record, and a 28-40 lined up with a 35.7% record. But 5-9 is 35.7% and 28-40 is 41.2%...

      Delete
  10. Raptors are now 7-5 since the trade. Small sample, sure, but they were at 16-30 when the trade went down.

    And not only did the Raps give up Calderon, they gave up Davis (who WP loves) and got Rudy Gay (who WP hates).

    ReplyDelete