Yesterday, James Mirtle published an interview with Brendan Shanahan wherein it was reported that the President of the Toronto Maple Leafs "expects the organization to find a way to incorporate analytics into what it does, noting that he had already spoken with several other management types around the league about how they use data to make decisions." Mirtle followed up that article with the following tweet:
Shanahan also said today he intends for the Leafs to use their analytics budget this year.— James Mirtle (@mirtle) May 14, 2014
So it seems the Leafs have finally decided to spend the money they have set aside for years. As a reminder, it wasn’t too long ago, back on November 11, 2013, that James reported this quote from Dave Nonis:
"We’re constantly trying to find solid uses for it," Nonis said on Monday as he took part in a sports analytics panel at the PrimeTime Sports Management Conference. "The last six, seven years, we’ve had a significant dollar amount in our budget for analytics and most of those years we didn’t use it.
"We couldn’t find a system or a group we felt we could rely on to help us make reasonable decisions."
This is what bothers me about the Leafs. Nonis is talking about buying a system that would help him make decisions and Shanahan is talking to other executives about analytics. Well, this is the thing. The system doesn’t exist, or at least it’s not for sale. There has been some great work done showing the correlation between "advanced stats" (and in particular, Fenwick Close) and scoring chances, possession and wins. Of particular interest is that although Fenwick Close or other advanced stats aren’t the ones with the highest correlation to wins (that would be GF%), it is the most reliable/repeatable and therefore, focus in advanced stats has revolved around Corsi and Fenwick. And herein is the criticism of advanced stats. The correlation between them and the most important factor, wins or goals, isn’t high enough wherein you could make decisions based on these stats with a high enough degree of confidence.
In a recent article from the Chicago Sun-Times, staff reporter Mark Lazerus writes:
"Don’t ask for details. You won’t get them.
You won’t get terminology. You won’t get methodologies. You won’t get the names of the stats, nor the names of the people compiling them. And you certainly won’t get the numbers themselves.
But Blackhawks general manager Stan Bowman and his numbers-crunchers are watching. They’re tallying every shot — on goal, blocked, or simply missed. They’re tracking where each one was shot, where it was aimed, how it was shot, against whom it was shot. They’re tracking who brought the puck into the offensive zone, and in what manner — was it chipped in? Rimmed around? Carried in? Passed in? They’re noting who tends to start their shifts in the defensive zone, and how often they tend to finish in the offensive zone. They’re keeping an eye on the quality of competition each player is facing, and how they fare."
The Chicago Blackhawks, one of the most successful teams in the NHL, have embraced analytics. The key here is that they are not falling back and relying on the work done by the bloggers of the world (make no mistake, the work guys like Tyler Dellow are doing is incredible and compelling. In fact, Tyler Dellow is currently looking at new metrics like zone starts, zone entries and faceoff percentages.) The Hawks are instead forging ahead and compiling their own stats – stats that they have deemed to be useful. In short, what the Blackhawks have done is simply ask themselves: If Fenwick Close can’t tell us enough of the story wherein we can make decisions in reliance on this stat, then what data can we collect that would help us? They’ve created their own stats. They’ve created their own system. And with this system, they have iced a team that will be competing in the Conference Finals for the third time in five years.
With the kind of revenue that the Leafs are able to generate, they should be at the forefront of the analytics movement. They shouldn’t be looking for a system; they should be creating the system. They should be the ones collecting the unknown stats, developing a new metric and creating the proprietary algorithms. They should be ahead of the Blackhawks. With a salary cap determining how much you can spend on players, analytics can be your competitive advantage, but it is only an advantage if you are the only team (or one of a few) with access to it. With the lack of meaningful stats being collected by the NHL, there is an opportunity here for the Leafs to create a system that would offer them a much needed competitive edge and it is a real shame that this organization did not see it that way last season. It isn’t too late to catch up to the Chicago Blackhawks analytics department. The budget needs to be spent, but it also needs to be spent responsibly.
It is painfully obvious that there is a severe lack of statistical data readily available or published by the NHL. It is therefore vital that the Leafs start collecting data such as zone entries, type of shot, shot position, shot results, zone times, and any other stat that they think might be useful. This is the kind of data that will form the basis of your competitive advantage. Ideally, you would be keeping these stats for every player and every team in the league whilst going back through game tape for as many years as possible, but this is a monumental task that probably won’t be feasible. However, at the very least, it is definitely feasible for the Leafs to go back and collect these stats for their own roster and for that of the Marlies. With respect to future trades or UFA signings, the Leafs could simply collect the data for particular individuals that they may be interested in.
The Leafs need to also make sure they do the analyzing part of analytics and not let the narrative define the statistic. This is the biggest pitfall of statistics in general and in my opinion, the biggest danger facing the Leafs. This organization is saturated with media coverage with a very predictable narrative for the team and its players. As an example, last year, CBC tried to introduce a catch-all stat for "grit", which was promptly ripped apart by Cam Charron over at the Score. The stats that the team collects must actually measure what is meaningful and they must be reliable and repeatable. The analytics department cannot be sidetracked by the team’s intense media coverage.
However, that’s not to say that the narratives run by the mass media have zero merit. It does mean that the narratives should be tested for their accuracy. If toughness, grit, leadership helps win games, then there should be a stat that demonstrates this concept. As an example, the narrative surrounding Dave Bolland is that he is a solid two-way forward. The stats crowd, however, disagrees and points to his appalling possession numbers. Here’s the thing. If Bolland is a great defensive forward despite the fact that he’s not all that great at possession, then stats should be able to tell us that. If he’s great at preventing chances, then let’s show it. Let’s collect the stat that tells us his zone time, where he starts his shifts, where he ends them, where opponent’s shots are coming from, who he plays against, etc. Let’s compare this to other Leafs players and to other role players on other teams. Let’s weigh these attributes against possession and come to a realistic evaluation of the player.
The Leafs need to use their stats to create their own proprietary metric or algorithm for evaluating players. Currently, the metric being used by the vast majority of the advanced stats crowd only measures possession. Although important, there are other aspects to the game and certain players are asked to play roles that have nothing to do with possession. The metric you create can take this into account. There is nothing wrong with evaluating a third line role player on a different scale than that of a first line scorer. In other words, create a metric that compares apples to apples and that takes into account all the meaningful stats you have collected (including possession). Having done so, you could then compare salaries of similar players and come to a statistically sound evaluation of your team.
Finally, rinse and repeat. There are bound to be mistakes in this process. You may end up collecting the wrong stats. You may end up giving too much weight to one stat and not enough to another. Management’s hockey experience may very well avoid a lot of these mistakes, but there’s a chance that they will happen nonetheless. It doesn’t mean you throw out the system you’ve created. It means you go back, find the errors and improve it.
The Leafs do not have to re-invent the wheel. What they have to do is make the wheel better – a lot better. This is a team with the resources to do everything I’ve written and probably do it extremely well if they remain objective. If they accomplish this goal, the Leafs will have their own system. It will be proprietary and confidential. It will be their competitive edge. Ideally, in two years, when a reporter asks about analytics, Shanahan only replies with a knowing smile.