I have long been aggravated by references to Toronto's supposedly terrible defence post-lockout.  People look at goals against, see a very large number, and automatically conclude the defence is terrible.  Left out, of course, is that the Leafs have received historically bad goaltending, at least according to save percentages. Advanced hockey analytics have taught us that given the relatively narrow band of NHL shooting talent, for the most part shots for and against are the best predictor of future success.  The Leafs have been merely mediocre at preventing shots post-lockout (rank for the past five years: 22nd, 7th, 17th, 17th, 13th); they meanwhile they are perennial bottom dwellers in goals against.

But is there more to it?  Is it possible that while the Leafs may not give up an unusual number of shots, we do give up an unusually high quality of shots?  That's the question I'm going to tackle here.  The bad news is, the evidence suggests that the answer is yes.

First of all, I have to acknowledge that others have ploughed this ground to some extent already.  Way back in 2004, pioneering stat guru Alan Ryder published this paper which found that shot quality allowed could account for a +/- 19 goals allowed swing over an entire year (note the Leafs of '02-'03 were actually 5th best in the league in this metric).  More recently, Gabe Desjardins at www.behindthenet.caand SBNation's www.behindthenethockey.ca has come up with a metric called "expected save percentage", and friend of the blog James Mirtle wrote about it while running From the Rink, here(not surprisingly,  in '08-'09 the Leafs did give up better than average scoring chances, but so did teams like the Wings according to this metric).  Basically, the lower the expSv%, the better quality chances you were giving up. This work suggests that yes, Vesa Toskala did have a bit of an excuse, but he was still horrendous.

I wanted to look at things a little differently though.  Could the measures described above just be a function of luck?  Is there a skill to allowing lower quality shots, not just preventing shots in general?  One measure is to look at a stat called on-ice save percentage (sv%on), which measures the save percentage recorded by the team's goalies when a particular player is on the ice.  Obviously there are a number of confounding variables here: quality of teammates, quality of competition, and quality of goaltending being the most obvious.  But perhaps there is something to be learned.

I therefore decided to look at the difference between a player's sv%on and the team's overall save percentage, and do so over a three year period.  If a player consistently outperforms his teammates, perhaps that tells us something.  If results vary wildly year-to-year, perhaps it's more a function of luck.

I should note at the outset that I have not eliminated confounding variables.  Even if sv%on is a repeatable skill, the data will be skewed by quality of teammates and competition.  So consider this a starting point--a way to determine whether there's actually anything there to study. Perhaps someone with a more statistical bent can take things further.

Before the results, let me give a quick summary of my methodology.  First, I decided to focus on defencemen because I thought they were much more likely to have an effect on shot quality than forward. Second, I don't have the computer programming skills to run data on all NHL defenders.  I therefore looked at the Top 20 defenders in 5 on 5 ice time for '07-'08, and tracked them through to last year.  Finally, I took sv%on figures from 5v5 figures available at www.behindthenethockey.ca.  I then compared it to team save percentages from www.nhl.com.  Note this means I am, to some extent, comparing apples to oranges.  Individual players' sv%on is measured 5v5; team sv% are measured in all scenarios, and was calculated by simply dividing goals by shots, thus including things like empty netters and shootout goals. Since I am only looking at the comparative difference between individual and team sv%, and not absolute numbers, hopefully these factors do not have too much impact.  

Here are the combined results for the three years starting in '07-'08 (higher numbers are better, and reflect a greater difference between the teams' 5v5 save percentage when the player is on the ice compared to the team's save percentage overall):

*Note for Phaneuf I had to manually calculate team sv%on for last year given the mid-year trade.  I think this issue also affects one of Kuba's years but I'm too lazy to do another manual year.

So some predictable results, and some surprises.  Falling in the predictable category: Nik Lidstrom is really, really good.  Also predictably impressive are the likes of Rafalski, Foote, Hannan and Mitchell.  I don't think it should surprise anyone to see Brewer, Ranger, Johnson and Kuba at the bottom (keeping in mind we're only looking at guys who, at least as of '07-'08, were big minute players).

There are also some surprises, but I think most of them are explainable.  For Leafs fans, it's noteworthy that Kaberle almost matches Beauchemin, and easily outpaces Phaneuf.  Remember, however, that it's a relative stat.  Phaneuf and Beauchemin played on defensively strong teams in Anaheim and Calgary, making it harder to exceed the team average (which makes Lidstrom that much more impressive).  That also explains why a guy like Martinek does so well playing for the putrid Islanders.

Bouwmeester surprised me--he played mostly for weak Florida teams, so perhaps Michael Peca is really on to something about the "easy to play against" angle.  Shane O'Brien also surprised me--Karina may have some insights here.

I wouldn't read too much into these figures, but they do suggest that individuals can have a material effect on shot quality, not just quantity.  And that means if the Leafs have another year of putrid goaltending maybe we really do need to reconsider whether our "great on paper" defence is really so great.