There has been much debate if Carlyle's defensive system that pushes shots to the outside has been beneficial and effective to the leafs. That is, a typical comment is that even though the leafs are getting outshot, they are giving up more of the "lower quality" chances from a further distance and minimizing quality scoring chances close up.
How To Measure Effectiveness of Carlyle's Strategy?
Let's try to quantify the "effectiveness" of Carlyle's strategy. There are a number of imperfect ways to measure effectiveness of the defensive system. Two of them are
1. How well the system/coach minimizes shot against (SA) as a proxy for how well the team is minimizing scoring chances against
2. How well the system/coach maximizes the ratio of shots for to shots against (SF/(SF+SA) as a proxy for how well the team is "outchancing" the competition.
Using Ninjastat's supershot tracker the shots for and shots against the leafs in road games for 2012 and 2013 were outputted. Note that, fenwick or corsi events (blocked shots, shots on goal and missed shots) from various distances are not available and so in this study we will use shots on goal as a "proxy" for Fenwick Close (actually scoring chances).
Skinnyfish has done a thorough analysis of shots against here. The focus in this article is primarily looking at the shot differential change in 2012 vs 2013 leaf team. This analysis didn't compare earlier leaf teams as the team roster was different (Kubina, Kaberle, Poni, Antropov, Moore, Stajan etc) who were actually quite effective at outshooting competition. Looking at the 2012 vs 2013 leaf results (see spreadsheet) we see this current team has given up fewer shots as a percentage below 30 feet at a cost of many more shots at a distance greater then 30 feet. On a per game basis, they have also given up fewer shots between 0 to 10 feet and 20 to 30 feet at a cost of more shots between 10 to 20 feet and dramatically more shots beyond 40 feet.
The leafs have been effective at decreasing the proportion and quantity of shots from high quality areas (close up 0 to 10 ft) at the cost of a large increase of shots from low quality areas (far away). This fact is probably known to most or at least evident to those that looked into it with some detail. In fact, even HNIC and Mirtle and a number of bloggers have discussed this topic. What is not answered is does it matter and if so how much? More on that later.
An alternate method is shot differential defined as SF/(SF+SA) which is a proxy for scoring chances (or Fenwick Close). In the spreadsheet linked above, we see again the leafs have improved their SF% (shot differential) by 3.7% within 0 to 10 feet. That is impressive. However, any gains in fewer shots and outchancing close up that Carlyle and the team gained at 0 to 10 feet is completely overwhelmed by the team being massively out shot beyond 10 feet. The problem again is does this tradeoff of allowing fewer high quality scoring chances close in at the cost or more low risk scoring chances from further out matter? And if so how much?
Weights and SH%
At this point, we should mention that all typical shot differentials are unweighted in that they treat shots, corsi or fenwick events as the same regardless if they are from 0 to 10 feet or beyond 30 feet. This treatment would be questioned by most coaches and many fans as a shot from 0 to 10 feet is much more dangerous then a shot beyond 30 feet. However shot differentials don't care about the idea of the quality of scoring chance. And they could possibly penalize a team that allows excessive shots from further out. To get by the problem with the using raw shot data, most NHL coaches track scoring chances. The leaf internal scoring chance team data is something which is not available though some bloggers track and publish this data so that is a possible avenue to investigate.
The method I use to determine the "effectiveness" of Carlyle's system and adjust shots for their quality is from this gem of entry by Hawerchuk that has an interesting graphic.
The point to note about the graphic is how SH% (shooting percentage) varies with distance of the shot. Basically the closer the shot the higher the quality of the scoring chance. And the further away the shot the lower quality the shot. [I don't have access to the raw data and I tweeted Ninjagreg to get an excel dump to get more accurate data but for this analysis I eyeball the SH% each of the distance ranges as shown in the table below].
The weights imply for example, that one shot from 0 to 10 feet equals approximately 3.9 shots from greater then 30 feet or 2.9 shots from 10 to 20 feet or 1.4 shots from 10-20 feet. Or said differently, a shot from 0 to 10 feet is 3.9x more likely to enter the net or 3.9x higher quality. The difference in SH% gives us something to weight or adjust the shots from different distances to produce something that is more representative of the quality of the scoring chances. Not that a coach's system can work this mechanically, but if their system could have their team allows up 2 more shots from beyond 20 feet which by allowing those two far out shots prevents 1 shot (scoring chance) close in, then in the long run the average SH% is in teams favor and they will outchance the competition in terms of quality. If however, the system generate 4 additional shots beyond 20 feet and only limits 1 shot from 0 to 10 feet then the SH% is not in the team's favor and they will be handily outchanced in terms of the quality of scoring chances.
A Look At Different Shot Differential Measures
In the table below are various shot differential measures are shown. The first four rows are the leaf shots differential for 2012 vs 2013 (current season) at the various distances. A score greater then 50% indicates the leafs are "outchancing" the other team. In addition, the Change % column is the difference in the 2013 shot differential minus 2012 shot differential and represent if the leafs have improved (bold) or worsened (red) in being "outchanced" (outshot).
Shot Differential (all distances) is the traditional unweighted shot differential [(SF/(SF+SA))] which treats each shot whether from 0 to 10 feet or 30 feet as having the same weight or same possibility of leading to a scoring chance. Not surprisingly, the results of the Shot Differential (all distances) matches more closely with the Fenwick Close which also treats all corsi events as being equivalent. The final shot differential measure is Shot Differential SH% Weighted that weights shots at various distances to better describe the quality of the scoring chance. The specific weights were presented above but again as an example a shot from 0 to 10 feet is 3.9x more likely to end in goal and so is weighted 3.9x higher then a shot from beyond 30 feet.
What does it mean?
Using weighted SH% to estimate the quality of the shot we see the leafs are considerably better then the traditional shot differential or Fenwick suggest. The way I interpret this is that the Carlyle is coaching an above average team and Fenwick and shot differentials may overstates how bad the team is. However, it also appears that this current team is "worse" overall then the last years team at being outchanced measured by the quality of scoring chances. And so in that sense Carlyle's coaching by limiting scoring chances has not been particularly effective or beneficial. That is, any gains achieved from his coaching of the defensive system has marginally worsened the team in terms of being quality outchanced compared to last season.
- Compared to traditional shot differential measures that indicate the leafs and coach are a completely ineffective, the weighted shot differential suggests the leafs and possibly the coach are slightly above average (in terms of outchancing the competition).
- Compared to 2012 team, the lower results in SH% shot differential this year by the leafs suggest that Carlyle maybe in some ways causing the team to underperform.
A couple of criticism of this
1. The thought behind this is the fact that a shot from close is in (0 to 10 ft) has a higher likelihood of scoring then a chance from further out. If that is incorrect then the analysis is invalid.
2. Ultimately, the analysis needs to be completed for each team and shown to correlate equal to or better then Fenwick or shot differentials. The SH% shot differential measure is designed to compensate for the quality of scoring chances and in that sense maybe "better" at least in theory then shot differential or Fenwick but that is to be seen. That said, it is unfortunate corsi and fenwick events by distance are not available.
3. They eyeball method to get SH% weights is not rigorous and a more analog method that doesn't group shots into some arbitrary distances is possible.
4. The shot data is available as 5 on 5 data and maybe subject to "scoring effects" that Fenwick Close does not.
5. Breaking the shot data into sub buckets distance may lead to small sample problems that skew results (especially with 0-10m distance) that is weighted more heavily as it is high scoring chance area.