Predictably, the article I wrote last week on what the Leafs needed to be mindful of as they enter this new phase of rebuilding brought a lot of criticism on one specific point; the identification of Director of Amateur Scouting Dave Morrison as the person most responsible for the Leafs current situation, after Dave Nonis and Randy Carlyle. A case could obviously be made quite easily for Claude Loiselle and the woefully inadequate job he did managing the Leafs salary cap, and that's perfectly fair. But Loiselle's major mistakes were confined to a one year period, while Morrison's have permeated throughout the organization for years.
Nevertheless, this seemed like a good opportunity to really examine just how Morrison's performance stacks up to both his criticisms and his supporters.
The defence of Dave Morrison typically relies on four key facets:
- It's not his fault that so many of his highest draft picks were traded away, making it much more difficult to improve the team through the draft.
- If you look at his overall body of work, the total number of games played by his picks are among the best in the league over that period.
- He has a number of great prospects currently on the way up.
- It's his GM's fault for pushing him to follow certain strategies, or push for particular prospects who would wind up becoming busts.
We'll get to his performance relative to his peers in another post, because there's much more data to process. Today, we'll focus on whether the record stands up to people's beliefs about some of the excuses for poor performance.
Not all draft picks are equal. Obviously a 1st round pick carried more weight than a 2nd round pick, and a 3rd round pick more weight than a 5th round pick. But even within each round, there can be significant difference in the overall quality of a pick. If you were to say "I'll trade you a 1st round pick", that sounds great, but what 1st round pick? The top pick in the draft? A pick in the top ten? A pick in the late 20s? As the draft progresses and more players of high quality are claimed, picks carry less and less value. The field flattens out and the number of players that are of significantly greater likelihood to become NHLers diminishes rapidly.
This is a logical rationale yet it always bothers me that the Dave Morrison Truthers (a term I'm using tongue in cheek) out there don't understand the difference. By virtue of failing to make the playoffs all but once in Morrison's tenure, the Leafs picks are almost always in the upper half of each round of the draft and in several years they've been in the top ten. Holding the 8th pick in the 6th round may not have much more additional value relative to the 18th pick in the 6th round but there is a marginal difference.
Scott Cullen of TSN has done some tremendous work to quantify this change in value, going through nearly a decade's worth of drafts to evaluate and assign probabilities to every pick in the draft. in the process, he arrives at the expected value from a particular pick (that is to say, the type of player you could reasonably expect to draft at that position) as well as the probability that a player picked at that point of the draft plays more than 100 games in the NHL. It's an elegant method to both assess the relative quality of each teams' drafting ability, their ability to draft ahead of expectations, and their ability to produce NHL players.
What I've done is compile the Leafs' history of draft picks under Dave Morrison, from 2006-2011* and, using Cullen's work, determine the expected results of each of Morrison's draft as well as the cumulative expectation for his tenure in charge. Then I've determined how he did relative to expectations based on what the players are today.
*I chose to cut this off at the 2011 draft because many players from the 2012 Draft or later have just begun their professional careers, so it would not be representative of their results. The 2011 draft is I feel an appropriate cut-off because I think we have a pretty general idea of what most of the players are at this stage of their development. Obviously the 100 game threshold can still be met, but this is the information we have today and I don't think leaving this off or making optimistic assumptions changes the overall picture significantly.
We'll use the 2008 draft to illustrate the methodology I'm using. These were the picks made by the Maple Leafs during that draft:
We cross-reference the placement of the Leafs' picks against Cullen's expected value for that pick. i've also included his reference of what each number on the 1-10 scale represents.
Obviously, the 5th overall pick is historically expected to produce a quality NHL player compared to a lower pick, and as you get down to the lower rounds, there is virtually no difference, just fluctuations based on the historical sample Cullen's used. We calculated a simple average for all of the picks from this draft; this number represents the "expected value" of the draft; without knowing anything about the players that were selected, based on the order at which the Leafs selected and the historical results of people selected at those positions, this is the overall value we'd expect out of this crop of prospects.
Quick tangent; what this does is that it places the overall value of the team's draft picks in context. A team drafting 1st overall and concurrently at the top of every round will have a significantly higher expected value based on historical results compared to a team drafting 27th overall and at the end of every subsequent round. It allows us to put a value to the relative strength or weakness of the particular group of picks that were available to the team in a given year. In effect, we've "normalized" the draft.
The next thing we'll do is determine the historical likelihood of a player reaching the 100 games played mark, in order to look in more detail at whether the draft is producing a relatively good proxy for "NHL players" (This is also why I used the 2011 draft as the cut-off for this exercise; there's still enough development to expect some of these prospects to reach this threshold but I think with most you have a relatively decent idea of whether they are going to get there or not). Similarly to above, we've referenced where the Leafs picked with the historical results of picking an "NHL player". We then calculate the same average of all picks to determine the expected number of "NHL players" that should come out of this draft. If you had 8 picks and an expected value of 25% NHL players, you'd expect 2 of your draft picks to play over 100 games.
Then we fill in the "actual values"; the assignment of a grade on the 1-10 scale based on that particular players' career, as well as whether they have met the 100 GP threshold. We do the same calculations of an average to get the draft's Actual Value, that we can compare against the Expected Value. I also calculated cumulative averages from 2006 onwards to gauge performance against expectations over multiple drafts.
The first graph below represents results against expectations for each particular seasons' draft (the bar graphs) and the cumulative results against expectations for all drafts up to that year since 2006 (the line graphs).
The higher the blue bar graph in a given year, the better the quality of picks the Leafs had and the better based on overall NHL performance we would have expected Morrison to do. You can see this as 2007 (where the Leafs didn't pick until the 3rd round) and 2010 (didn't pick until mid-2nd round) are lower, while 2008 (5th overall) and 2009 (7th overall) are higher. As we add more and more picks to the cumulative total, the expected value flattens out, since most picks are in later rounds and have minimal chances of success.
What this graph tells us is that they did quite well in the 2006 draft that produced a number of key NHL roster players, and met expectations in 2008 and 2009 when the top picks they chose, Luke Schenn and Nazem Kadri, became NHLers. However, as time has gone on the overall performance has not been particularly impressive, and as of 2011, the results have declined to the point that even when accounting for a weakened draft class, the results are only slightly above what would have been expected.
Okay, so what about on games played? That's a common point in Morrison's favour, including a study done by a PPP user that determined Morrison had produced the 3rd most games played from his draft picks over his tenure. I think that this is a bit of a red herring; Morrison's results in this field rely on the contributions of the earliest draft classes of his tenure, so there's an inherent advantage of getting his best results coincided with the players that have the most games available to play in.
So let's check with Cullen's expected probability of producing an "NHL player" (>100 GP) to see if this is true.
Not... really. The results generally match what we saw above; the Leafs results are largely dependent on the earliest draft classes, but worsen over time and fall below expectations of producing an adequate number of NHL players. Again, I concede that it's likely 2011 draftees like Stuart Percy and Josh Leivo meet the 100 games played threshold, but given their injury history and consistency issues, respectively, it's far from given and I feel comfortable relying on what they actually have achieved instead of projecting what they might become.
One confounding factor; Dave Morrison took over his current role on June 23, 2006. One day before the 2006 draft took place. So is it appropriate to give him full control over a draft that somebody else oversaw the preparation of the draft list? We can't possibly attempt to determine that answer, so in interests of more information, I produced another set of these data, starting from the 2007 draft, one that we can all agree Morrison oversaw.
Without the 2006 draft to prop up actual results over expectations, the degradation of results to expectations happens much sooner. The 2007 draft produced a couple of useful players but was generally poor in quality which was to be expected given they didn't pick until 74th.
- The overall quality of the Leafs picks during Dave Morrison's era definitely suffers due to the volume of high picks traded away during his tenure. However, even when accounting for this fact his results have frequently not met expectations.
- Going by cumulative games played in the NHL to validate the drafting record is a red herring, as it suffers from survivorship bias. Based on a consistent threshold that represents a player making the NHL, Morrison has not produced NHLers at the rate that would be expected of him.
- The data examined covered a period of time with 2 different General Managers and 3 different strategies for acquiring talent, and it's not entirely clear the level of Morrison's involvement in leading the draft that unfortunately happens to be the most successful over that period. To say that his performance is negatively influenced by the message from his superiors doesn't seem to hold up to scrutiny.