Lessons from HALO: How teams use data

The HALO Conference in Denver in April was the first NHL team-run analytics conference and is part of an initiative by the league. Just what its goals and activities will be is a little hazy yet. The timing of the event has been useful for confronting the changing management direction of the Leafs and for just figuring out what analytics even is now that it's back on the menu.

Kroenke Sports & Entertainment is the parent company of the Colorado Avalanche, the Denver Nuggets of the NBA and the Colorado Rapids of MLS among other related assets. The ownership structure is extremely similar to that of the Maple Leafs, and this panel at HALO shows how similar all three KSE teams are in their approach to using data. But the differences are meaningful lessons for Leafs fans who are about to get a datacentric approach back in hockey operations.

The panel is hosted by one of their broadcast people and features Chris MacFarland (Avalanche GM), Ben Tenzer (Exexutive VP of Basketball Ops for the Nuggets) and Pádraig Smith (president of the Rapids).

This is an incredibly interesting window on what teams are actually doing, as distinct from what fans use public analytic models for.

Three main areas of using data are discussed:

Evaluating players for the draft or for acquisition of current pro players.
Evaluating game play in general and also from specific players to measure against a game model of expectations.
Evaluating the physical ability and health of the players directly.

How that evaluation is used by the teams varies greatly.

Data as a source of collaboration

Ben Tenzer leads off by saying that analytics is a part of everything they do all the time, but he also mentions that the NBA allows things to be measured that other leagues aren't allowed to do. The NBA allows biometric data collection that the NHL does not, for example. Basketball teams began using cameras and GPS trackers over a decade ago, and teams began experimenting with wearable monitors – both on and off the court – at that time.

Five years ago that wearable tech was used during the Ryder Cup broadcast to show golfers' heartbeats in real time. Keith Pelley was head of the European Tour at that time, the co-sponsor of the Cup.

In describing how the Nuggets use analytics in evaluating draft prospects, Tenzer talks about an integrated process of looking at video, gameplay data and biometric data to decide if a prospect can carry his college skills into the NBA. This contrasts pretty sharply with a typical NHL org-chart view of hockey operations where the data science people are over there, and the scouts are over here and they might only ever meet by accident. How much the org-chart mirrors the actual processes used is, of course, going to vary from team to team, but it's clear that for the Nuggets, they aren't walled off.

Tenzer can't talk about analytics without talking about video review, coaching, and scouting as well as the data they use because, to him, they are all the same thing. Their in-house scouting portal is a website they built to allow scouts to instantly access all the information they have on a player. There's no concept of balancing competing paradigms, it's just one big integrated package.

Next is Pádraig Smith, the president of the Rapids, who is an interesting man. He used to run a branch of Ernst and Young (an accountancy firm) in Dublin before he moved to a business related role in Irish football. He's been sporting director and then president at the Rapids since 2015.

He talks about analytics in terms of team and player performance as well as recruitment. MLS gets a lot more players than the NHL does by signing foreign pros as adults, so it's much more important for them to have data on those players.

MLS also uses biometric data extensively to determine player usage, for training as well as for evaluating players for recruitment. Smith talks about how much data they have access to, and how they have their database of potential players purpose built to align with their game model and position-specific relevance all in the name of getting the right fit in players they target to acquire.

The one area where MLS is more a model for how the NHL can acquire useful data and use it to gain deeper understanding of players and teams is in how rare the goals are, and therefore how much sway randomness has over outcomes. Smith talks about the need to dig deeper into the way the game works, essentially, to give them a game model that is the best. They then test against that model on a game-by-game basis.

The software tools the Rapids use are mostly built in-house. And a wander through NHL data science teams listings on their websites will show you software and database engineers along with the analysts. There was a push from some teams to have robust tracking data long before the NHL got in the business, and a lot of NHL rinks have private data providers' cameras up in their rafters amongst the Cup banners. But teams absolutely are building out their own tools and reporting platforms that allow the people at the top of the decision trees to be more effective managers, not analysts themselves.

Chris MacFarland describes analytics for the Avalanche as a way to look under the hood. His example is the situation where they know the team didn't play well even if they won. So why exactly did they play poorly? He says something similar about a situation where scouts and analysts don't agree about a player. Then you dig deeper using all your tools to find out the truth. He also touches on how bespoke their data usage is.

Data as a competitive edge

Smith says something interesting about how MLS is not the top of the soccer world (although they do have access to some of Arsenal's data), so that what they are doing is closer to the problem solving in Moneyball. The NBA and NHL teams are in the elite league for their sport.

The tag line of Moneyball is "The Art of Winning an Unfair Game". What made baseball unfair for the Oakland Atheltics in the early 2000s was the amount of money teams like the Yankees could spend on their players. The thrust of the Moneyball ideas, and where the "money" comes into play, is in attempting to get wins with less monetary investment. That's their problem. They sought evidence-based solutions to solve it, which is really the nature of the revolution.

"It's easy to spot the really good players" is a truism for any sport. MLS soccer has a soft salary cap, which allows for some designated players who don't count against the cap. This sets up an unfair game in the real money sense in MLS, where they need to succeed by finding the players that aren't obvious because they cost less in real dollars.

There is a hint of the way in which the NHL is unfair in some of MacFarland's later comments. As a team deep into their competitive cycle they lack draft picks and have to look to things like the NCAA free agency period to find value in player additions.

What all these executives also discuss in very similar ways is the question of a player's fit in terms of their playing ability but also their personality – team culture ideas. Good in the room vs good on the ice.

They are, inevitably, debunking the persistent myth that analytics is used to dehumanize the process of player evaluation and demands that decision makers ignore those human traits that can't really be measured. Let's consider Scott Laughton for a moment. He's good in the room, this is clear. He's also got a particular on-ice affect that seems desirable. He's past his peak age, while he's clearly still an NHL player, but disagreements persist on how good he is.

The idea gets floated that you use his "good in the room"-ness to break the tie in some imaginary head-t0-head contest with some other player you're going to shop for. So the Leafs should not sign him just because of the cultural value he brings. But the just here is a fiction created for the purpose of arguing. Evaluating players, as all three of these executives make clear, involves all their varied attributes. If evidence-based decision making exists in the mind of the person at the top then there is no risk of them just doing anything.

As Smith puts it, their analytics are used to de-risk player acquisition. As Tenzer puts it, you have a Venn Diagram of your eyes, your ears and your numbers and where they overlap is where the good players are.

What is very clear is that both MLS and the NBA have a lot of data on draft-eligible players and pros from other leagues that the NHL really doesn't. MacFarland alludes to what sounds to me like player similarity comparison models, which is about where the NHL's draft analysis sits. There are some public versions of these modles that float around from time to time, and they are almost universally points-based, so they might do a better broadly based sorting of players than no model at all, but points-based systems will inevitably have some big glaring false negatives that show up someday when an NHL team gets a top pairing defender on waivers and wins a couple of cups with him.

The Rapids didn't become a datacentric team overnight. Smith has been with the team for over 10 years, and he mentions that when he started, the coaching staff wouldn't even use video recording and the players balked at wearable monitors.

As they gradually incorporated new technologies and new ideas, they gained a competitive edge. He is upfront that the company's Premier League team staff told them they weren't up to standard and they had to improve.

The Rapids got to be a better team by being able to show players the data. The point of this is to help them if they are struggling, but also to reassure the player when they are doing the right things. Players buy in when it helps their performance, but both Smith and MacFarland made it clear that the coach has to be the nexus of passing information to players when it's appropriate and that the goal has to be the psychological well being of the player as well as maximizing performance.

Data as a source of conflict

An audience question asked about players having their own analytics sources. Both Smith and Tenzer mentioned the possibilities that performance coaches or training consultants might be using data as tools and engaging with the play in that way, but it appears not to be widespread.

On another HALO panel (linked below) Meghan Chayka, Co-CEO and Chief Marketing Officer at Stathletes, talked a bit about selling their product directly to athletes. She made a remark about it being an easy sell to the 1%.

The commercialization of data analysis into products to be sold is definitely something Smith sees as a source of conflict, and of course soccer is way ahead of the curve on this because they have a business and media landscape that is very different to the NHL's.

Agents use data in negotiations, and that leads to teams having to counter a case made with something they might not consider valuable information. The media does this a well. Smith contends the analytics sold to the media are largely descriptive, and it gets used for story telling. But they are often using information that isn't significant.

He also raised concerns about coaches using bundles of commercial tools that they think of as all useful when some aren't. Within an organization there can be conflicts over how to use these varied sources of data. An argument again for decision making skills at the top level.

As Smith says, everyone wants a simple number that tells you how much better a player is than replacement, but it doesn't exist. The game is too complicated for these simple answers. That means coping with the complexity has to be part of the goal of the analytics systems that teams use.

In the NHL, very little of what you could legitimately call analytics can be used in arbitration cases – Corsi and Fenwick and that's about it. But any other negotiation is wide open to whatever teams and agents want to say to each other. As is every conversation between a player and a coach and a coach and a GM. As the NHL progresses, and more teams have GMs comfortable with these tools, there will be disputes with agents over whose numbers mean something important. Everyone needs to be ready for this.

The NHL is way behind

While this conference made clear that all NHL teams are trying to use analytical tools, it's also clear that the complete integration into team operations that exists on the Nuggets and the Rapids is not quite what the Avs are doing.

The biometrics other sports use as a matter of course are a no-go in the NHL, and would require some serious CBA negotiation. There are privacy concerns, two countries' rules about it all, litigation risk and the hill to climb of getting player buy-in.

The internal-facing use of data is going to stay confined to the development of a game model and evaluating the players against that model before finally communicating what's useful to the players. It's not that the NHL has no sports science or medical data collection, but it's nothing like what other sports take for granted.

The external-facing use for data is player acquisition. The draft information the NBA has and the data on pro players MLS has is well beyond the scope of what exists for NHL teams. I am guessing a lot of hand tracking is going on for draft analysis for teams looking to draft out of the CHL or NCAA. Although there are some stats that NHL teams can get from those leagues that you and I can't.

The NHL's hard cap doesn't create the same real-dollar moneyball issues that most teams want to solve. But what the NHL does have is teams looking to draft and sign players in a way that keeps the competitive window open. In any given year 15% of the NHLers are undrafted players. That's the sea to fish in if you're not cash poor but rather draft-pick poor.

If there's an area of NHL hockey that is absolutely ripe for finding those famous market inefficiencies it's defence. Turn the smartest minds to figuring out what defence is and what it is not and how you find it, and you'll be able to turn a few good forwards and a couple of goalies into a real team. Or so we hope.

This is the other panel, which I struggled to get much out of, but it might give you a feel for where commercial services offering data analysis are at. Chayka constantly refers to masses of data and the cost and difficulty of managing it all. The low level of what is discussed for broadcasters makes that heartrate monitor idea seem innovative, though.