Jose Calderon: An Example of How Wins Produced Fails

Update: See similar articles for Boozer here and Rondo here.

Wins Produced, a model by David Berri, attempts to capture player value through box score stats by assigning each measurement a number in terms of how many wins it adds to a team. There are numerous problems with the method, some of which I described here, but despite its rejection in most NBA stats circles (they have the best parties) it's heavily supported by TrueHoop, whose influence and view count allow the flawed stat to be marketed heavily toward a large audience. This isn't a pointless issue because for many people it may end up being one of their first experiences with advanced stats in the NBA, which is negative for the advanced stats movement. I realize that all models have their flaws; one of the main issues I have is the author's, and his cronies', insistence that it is nearly perfect and they regularly disparage other analysts.

The method of Wins Produced overvalues efficient shooters, high rebounders, and players who keep turnovers low, while they undervalue shot creation and defense. If you shoot a percentage below their league average (with positional adjustments), then your Wins Produced number is below average, even if you're the only player on the team who can create a shot. There's no context or nuanced understanding of the game. Last season for instance James Harden "created" more wins than Dwight Howard and the year before Landry Fields somehow was rated higher than Tyson Chandler, Ginobili, Blake Griffin, and Durant.

But who's the prototypical example of a player overrated by Wins Produced?

How about someone who was apparently the fifth best player in 2009 behind only Chris Paul, James, Howard, and Billups? (Hm, one of those four is not like the others.) It's yet another point guard ... Jose Calderon, who somehow put up 14.9 wins for a Toronto team with only 33 wins. This was when they still had Chris Bosh, so it's not like they were so completely devoid of talent that without Calderon they would have been atrocious.

This is someone who's lead the league multiple times in assist to turnover ratio, some years over 5. He's broken the single season free-throw percentage record at 98.1%. He joined the 90-50-40 club (though he didn't have enough attempts.) He's had years of accuracy from all over the court only bested by Nash. However, he's also one of the worst defenders in the league, especially after injuries piled up, and he doesn't shoot enough that he can be considered a star given that the only other thing he can do is pass. He's a low mistake point guard whose prowess on the offensive end is nearly erased by his lapses on the other end. So how would they play without Calderon, who according to Berri and friends should have been an all-league, multiple time all-star?

Unfortunately, Jose's never been traded, so we can't see if there's a LeBron-leaving-Cleveland type of situation on a smaller scale, and he played all 82 games that year. Luckily, he normally misses 10 to 20 games a season. Obviously, there are more advanced methods to estimate player value, but one focusing on the performance of a team when someone plays versus when he misses a game is simple and transparent, allowing anyone to understand the numbers.

Jose Calderon in 2008-09

Playing alongside Chris Bosh, Jose Calderon averaged 12.8 points, 8.9 assists, 2.1 assists, and 34.4 minutes per game on stellar percentages of 61.3 TS%, 49.7% from the floor, and a record-setting 98.1% from the line. Toronto, however, finished 33-49 behind three teams in their own division. He missed 14 games, lending to a nice little sample of games to analyze. Wins Produced says he was worth 11.6 wins at 0.239 wins per 48 minutes where his rank that year was 14th and 11th, respectively (among players with at least 1000 minutes.) That is definitely all-star worthy, and if he misses a game Toronto should predictably perform worse. A win per 48 minutes level over 0.200 is elite where 0.100 is "average". What happens in those 14 games when he's out?

Toronto was 5-9 when Calderon didn't play, and 28-40 with him. That's a win percentage of 35.7 versus 41.2, meaning they won more games with him. Do we drop the conclusion there? Of course not. It's a terrible small sample size and the difference isn't huge. If they had gone 6-8 without him, just one more game, that percentage moves from 35.7 to 42.9. Also, the competition was a bit steeper during those 14 games. Once you make some adjustments and look at the numbers in the table below, in Calderon's second best season ever his team, a fairly crappy team, performed better when he was off the court. That is not congruent with a player apparently worth about 12 wins over the season.

	Record	Win%	Margin of victory	Off. rating	Def. rating	Strength of schedule	SRS	Wins Produced
With Calderon	5-9	41.2	-2.99	106.5	109.8	-0.55	-2.94	11.6
Without Calderon	28-40	35.7	-2.00	109.5	111.2	1.45	-0.55

The most important column is SRS. It's a basketball-reference measure in terms of points that includes margin of victory, strength of schedule, and home-court advantage. An average team is 0, a very good one +5, and a poor team -5. For example, during the 2009 season there were five teams with an SRS -5 and below and five teams at +5 and above with the best mark at +8.68 (Cavaliers) and the worst -8.59 (Kings). Homecourt advantage is assumed as a +3.2 advantage for calculations.

Once more comprehensive information is included, the results are rather damning. Against a tougher schedule without their starting point guard, the Raptors improved their average margin of victory and consequently their simple rating system score (SRS). Notice that despite his stellar shooting their offense actually improved when he wasn't in the rotation. One piece of evidence against his favor is that his team had to use their backups to replace his production, which is harder to do at point guard where the guy you're replacing runs much of the offense averaging near 10 assists a game.

If you think there are extenuating circumstances and shouldn't be used to discredit Wins Produced, don't worry because there are many other seasons to consider.

Jose Calderon 2006 to 2012

Excluding the year he played every game and his rookie year (adjustments during the first year are notorious, especially coming from a Euro league, so it wouldn't be fair to include it), Calderon's results are shown below. The team's win percentage is noticeably better when he's not playing, and the effect appears to be on both the offensive and defensive ends of the court. The SRS, again, is a rating system that adjusts for your opponents, home court, and by how many points you beat a team; it's like an adjusted average margin of victory.

Season		Record	Win%	Margin ofvictory	Off. rating	Def. rating	SRS	Wins Produced
Total	With Calderon	115-219	34.4	-3.75	105.1	109.2	-3.87	38.7
	Without Calderon	26-34	43.3	-2.13	106.6	108.8	-2.05
2006-07	With Calderon	3-2	60.0	-3.35	100.8	104.6	-3.79	6.5
	Without Calderon	20-57	26.0	-2.60	100.7	103.2	-1.83
2008-09	With Calderon	5-9	41.2	-2.99	106.6	109.8	-2.94	11.6
	Without Calderon	28-40	35.7	-2.00	109.0	111.2	-0.55
2009-10	With Calderon	9-5	64.3	-2.31	111.7	114.3	-2.04	5.4
	Without Calderon	31-37	45.6	0.79	109.1	107.8	-0.80
2010-11	With Calderon	4-10	28.6	-6.41	105.3	112.3	-6.89	7.4
	Without Calderon	18-50	26.5	-5.64	108.6	114.8	-3.32
2011-12	With Calderon	5-8	38.5	-3.75	100.7	104.9	-3.65	7.6
	Without Calderon	18-35	34.0	-1.46	101.2	103.0	-3.74

It's best to view the table as paired data within each season where a gray row is for when Jose "Wins Produced Star" Calderon plays and checking the cell below in the white for when he doesn't play. There was only one season where they played better with him on the court, and it wasn't by much; Wins Produced that year basically considered him a near all-star while his effect on the game was middling. Be careful, however, looking at the yearly results because with the small number of games the numbers are prone to noise and chaos, but the aggregated totals have enough of a sample size for some trust. And the results aren't good.

Synthesizing the Information

But how do you estimate how Toronto should have played without Calderon? Well, first assume that Calderon is being replaced by an "average" player; given the backups they normally employed this is generous and a conservative assumption that helps Calderon look better. Use a Wins Produced per minute number and his average minutes per game for the season to estimate the "wins" lost. For a statistical test, a standard deviation is needed, and unfortunately that's a difficult measure to find for such a specific use. What I used was the standard deviation for Russell Westbrook's Wins Produced each game in the 2011-12 season, which the author claimed was a fairly high variance. I realize that's a hefty assumption, but I'm trying to err on the side of caution here and I'm also assuming his backup is an "average" player. If anyone has a better number or wants to calculate the standard deviation for Jose Calderon in each season tested, go right ahead. The problem is that given the standard deviations I've seen the gap is too large to cover.

Actual wins without Calderon	Wins Produced predicted wins	Standard deviation of wins produced	t-test  p-value
26	17.4	0.24	1.08E-41

The p-value is saying there's an extremely low probability that by mere chance the Wins Produced predicted wins number just happened to be that much lower. Think about the standard deviation as a naturally occurring spread of numbers. I'm looking at each game Calderon missed and trying to estimate whether there's a win or loss by the team's win percentage with Calderon and Win Produced's guess at the average "loss" in wins each game from losing him. The standard deviation then should be rather low because it's the Wins Produced each game. It's important to consider because natural variation of his score can skew the results by sheer chance, but again the gap is so big between the actual wins they had in the 60 games without Calderon and the predicted wins from Wins Produced that I'm comfortable in stating there appears to be statistical significant difference. 

Wins Produced is unsuccessful at explaining the value of Jose Calderon.

Conclusion

You can play around with the numbers all you want, but over 60 games the evidence is hard to ignore: even on a crappy team like the Raptors, Calderon is is a negative force, not an all-star point guard like Wins Produced suggests. If you think it's unfair to judge just on the games he missed and there could be complex teammate effects behind everything you can take a look at his yearly ridge-regressed +/- score or his current ranking; or maybe you can cross-reference another adjusted +/- model that also states he usually has a negative effect; or hey here's another one; and if you think yearly models are too noisy (though one link has a two-year model) here's a six-year study. If you don't want to click through those links, here's the summary: he's a below average player according those methods normally in the -1 to -2 range; the six-year study is particularly alarming where it says he's a -1.69 because it's hard to explain away any bad results when many seasons are included. 

With most models, authors are hesitant at declaring complete confidence in the results, but the fun with the Berri crew is their cult-like insistence at the perfection of their magic recipe. Here's where Berri implied Calderon should have been considered for all-star and MVP voting in 2008. Here's an article where he says Calderon was better than Bosh. There are also many arguments like this in the comments section where the adherents try to ignore any criticism by how, hey, they used numbers, so of course it's infallible. The most robust argument I've heard, which I've made in a previous article, is that Wins Produced derives wins from offensive and defensive efficiency (and possessions), and then assigns a piece of that win to each box score stat. Then they claim validity by summing Wins Produced and saying, Hey! Look at how close it is to the team's actual wins! The problem is all they're doing is stating team off/def efficiency correlate well with wins, and there's no proof offered at how that applies to the players. They give all the credit to the rebounder and none to the wing defender, unless he gambles and gets a steal.

When Calderon played over five seasons his team's SRS (adjusted point differential) was -3.87, but when Calderon wasn't playing during those same seasons his team's SRS improved to -2.05. He's an amazing shooter who can shoot like Nash and pass and protect the ball like Chris Paul, but his defense is atrocious and he doesn't shoot enough ... or appear to have any positive effect on the basketball court. And ask yourself this. Do you really think box score stats are all you need to understand the game (or specifically player value)? And that the stats completely cover all ends of the court? Try tracking a play and see who's involved, what passes are made, the picks, the boxing out, and then realize you can only give credit when someone completes an action recorded in the box score sheet. I'm sorry, professor Berri, but Jose Calderon is not worth 12 wins in a single season.

Note: good discussion of Wins Produced here.