Lessons from HALO: What is Analytics?

Sign Up to PPP Today

You have to be a member to comment at PPP. Membership is free and requires only an email address.

Already have an account? Sign in

The HALO conference in Denver in early April was the first NHL-funded and team-run analytics conference. Arik Parnass, the Director of Analytics for the Avalanche, opened the event with an overview of how the NHL got to this point where over 90 NHL team employees attended this conference.

I've set the video to where he gives his view on what it means to do analytics well, but if you want to back it up, you can see him run down the names of the blogosphere crew of yore who have turned into NHL AGMs today. My thoughts on what he had to say are below.

Too often, the idea of taking statistics in sports and using them to inform decisions is perceived as almost a faith based initiative – a competing religion to the existing belief system. Parnass's example from an old advertisement about baseball is apt. I'll repeat it here:

In baseball there's a stat for every situation. Tie Game, man on second, Ian Desmond at-bat. In day games, he's batting .219. That's what the stats say. Do you pinch hit for him? Absolutely not. Sometimes, you believe in the stats. Sometimes, you believe in the players.

The idea behind this oh so clever bit of boundary policing is endemic today, while I wonder if anyone knows who the hell Ian Desmond is. Just last week, I ran into someone who was favourably quoting yet another comment by Brian Burke telling everyone that you can use stats (thanks for the permission), but you have to start player evaluation with your eyes, with how they play, and on it went.

For a lot of hockey people, it's okay to measure things after the fact, but it's a bridge way, way too far to use what analytics actually is to understand players, the game and to also understand what can't be known. When someone grunts out the word truculence and is sure the nerds can't grasp why it matters, it's obvious to anyone they've reduced the game down to ... well, I have been known to call that your taste in men. It's not those people who are the problem to a wider understanding of analytics in sports.

The popularity of Burke's homilies about the streetlight effect, one of his classics, reveals that a great many people don't actually know what analytics is. This became very clear the minute Keith Pelley said the word datacentric. I wish Parnass had been a few days earlier with his decade-old wisdom, but it's never too late to catch up.

His simple explanation is that a statistic is just a piece of data, but analytics is the study of data to find meaning. So therefore datacentric is about meaning not data at all. The cautionary tales of the moderates who are okay with using numbers (so big of them) as long as you understand what they mean are actually saying something so basic, so elemental that it's impossible to take them seriously.

It's okay to use data as long as you analyze it. That's what they're saying. It's okay to have water, but you shouldn't snort it up your nose. It's okay to breathe, but you should exhale the carbon dioxide. It's okay to put on skates, but don't wear them on your hands.

With that out of the way, Parnass covers the concept of deterministic vs probalistic views of the game. We often speak similarly of descriptive vs predictive. I sometimes call it thing you are vs thing you do. All these ideas seek to recognize a fundamental dual nature of hockey.

There is a cause and effect for every action in a game. Human decisions, the laws of physics, the randomness that is the nature of the universe, and physical ability of players combine to cause outcomes. The player's mind, their intent, their years of dedication lead up to a goal or a hit or a giveaway. And then someone takes all that and plots a bunch of dots on little comic-strip hockey rink, and says here is what this player is likely to do, and for many that's not merely something they struggle to understand, it's outside their paradigm of what the sport even is.

Parnass talks about what a GM might do if they have a wholly deterministic view of the game, in what amounts to a pretty good demolition of the "hot hand" fallacy beloved by many, including a lot of GMs who want to be analytics moderates and measure with data, but not think with the meaning of the measures.

Parnass's next point is about randomness. This is one of the fundamental things that any person needs to get a grip on if they want to be able to make good decisions. You don't need to give a damn about that if you're a fan watching the game, but your way of seeing the game isn't going to be a valid measuring stick for who should be GM or coach or what players they should add or subtract.

Parnass uses a good story about iPods and shuffle to demonstrate that we all start out with a view of what randomness is that is very, very wrong. We live in a state of belief in a giant conspiracy theory about how the universe functions. And I've seen this brought to life in vivid colour in almost every single comment made about the draft lottery and what will happen. Understanding what randomness looks like is the necessary precondition to being able to recognize things that aren't random, and then find meaning in data.

I think getting your head around measurement and meaning, description and prediction, shadows of what is and what may be is pretty easy if you're motivated to. It is randomness where many fall. We often get back up again and try to rewrite our own ways of thinking, and try again. But randomness is also the hurdle where many will outright balk at or they'll try the buffet approach: "I'll admit some things are random, sure, but if Auston Matthews isn't scoring goals like he should, then there's something wrong with him and no bunch of dots on a drawing of a hockey rink will convince me otherwise."

I think a lot of very smart, experienced and well educated hockey people with open minds take the buffet approach. And I think this is the point within an organization where the beautiful tapestry of data, analysis, decisions, collaboration, and process starts to unravel. A successful NHL team is bound to be tattered around the edges to some extent. We aren't ever getting to perfection, but analytics, evidence-based thinking, and datacentric approaches cannot be optional.

Parnass's conclusion is a thought experiment about all of these ideas of what measurement is for, how you use it, and how you think about it and with it. He sounds almost angry, certainly impassioned. And I am there with him. This is the article of faith which divides, and it's fundamental to the way the game is described on television and by us as viewers, and also by the people actually playing and creating the teams. It's the big emotional moment, and it's the emotionally weighted means by which we tell lies about players and teams and the league.

Points.

He asks you if you were inventing hockey today, would you select the last three people to touch the puck before a goal, a thing that happens about six times a game, give each of them one unit of your measure and say there: I've defined the players, the team, the game.

If you can't give that up, give up points as the foundational stat of the game, then you will never think clearly about the game at a fine level of detail.

That sounds harsh, and it's rare for the people who publicly disavow points almost entirely to say it this bluntly. But it's the truth. Goals sure are the fun part, though.

I don't expect we'll see coaches stop saying things like what was said of Matias Maccelli by Craig Berube in what was otherwise a good summary of his progress this year – that he has to start getting points. He's not going to tell a reporter that Maccelli needs to keep doing the things that lead to winning, that lead to "scoring goals, more than them, do it again next time". The closest we'll get to that is talk about grinding it out when there's a "slump" or maybe a little process talk from someone like Sheldon Keefe.

If the deterministic view, based on no understanding of randomness and a commitment to an absurdly ineffective measure, is the most popular dish from the buffet in the executive dining room, the team is going to fail. No more pretending that analytics is an optional tool, or a coat you put on and take off, or a way to prove your biases are correct. It has to be baked in at all levels.

What the Leafs did when Brendan Shanahan hired Brad Treliving, when Brad Treliving hired Craig Berube was choose to lose. It didn't have to happen right away, and it didn't! But it was inevitable, and it's high time the richest team in the NHL quit hiring substandard people.

PPP Runs on Your Support

If you enjoy reading PPP Leafs, and want to see it continue, please consider becomming a paid subscriber. We want to keep all our content open to all users, but to be a sustainable site, we need more support from paid members.