Unless you've been living under a rock, you're aware that the Arizona Coyotes recently hired John Chayka as their new general manager. Chayka is a 26-year-old Western graduate who co-founded the hockey analytics company Stathletes, and was originally hired as an Assistant GM by the Coyotes last offseason. The hiring has been touted as another example of the changing of the guard in the NHL -- a sign that analytics experts are becoming more and more attractive as front office candidates.
To say this is out of the ordinary would be an understatement. Not only is Chayka the youngest GM in NHL history, but at the time of his original hiring in 2015, he was probably among the youngest NHL front office employees in any capacity. In an industry where incompetent workers hang around simply by virtue of experience, the hiring and promotion of Chayka is unique. I suppose a similar situation occurred with Toronto and Kyle Dubas, but even Dubas had a more 'traditional' path to a NHL front office after first plying his trade as a GM in the OHL (Dubas is also a few years Chayka's elder). Chayka had a playing career cut short by injury, but had no formal management experience for a hockey team that I'm aware of.
As with any non-traditional hire, there are skeptics. We're used to the old guard bemoaning the idea of hiring someone so green for such a pivotal position. With Chayka's background and work in hockey analytics largely being private, we're also seeing some skeptics from the fancystats community, likely due to the reputation of other private entities who have tried to break into the hockey analytics community, such as SAP and the Department of Hockey Analytics. It's easy to dismiss the first group as out-of-touch old timers and the second group as jealous nerds, but in my opinion, there's a kernel of truth inside the arguments of both groups of skeptics.
First, experience does matter. Chayka is very green, and there's no getting around that. There will be a learning curve, which he himself has acknowledged. Being a GM in the NHL is not easy, to be sure. That said, I think the skeptics are overselling this factor. Arizona isn't going to tell him to fly solo. They have experienced hockey minds around him, including coach Dave Tippett, who also received a promotion (to Executive Vice President of Hockey Operations). Chayka isn't going to be the sole decision-maker (at least not initially). There will be others involved in his role, and to me, that mitigates the risk of hiring someone so inexperienced.
I also think there's logical reasons to be unconvinced by his hockey analytics background. Stathletes' methods and the information they track is mostly unknown, and understandably so. As a private company, it's in their interest to keep any competitive advantage they have under wraps. We can take some educated guesses as to what they do, which by extension, reflects upon Chayka's outlook for hockey analytics. From their website, they claim:
Every facet of each game is observed and recorded through our intensive video analysis process, yielding over 100x the statistical resolution of existing tracking methods
A 2014 article from the Globe and Mail also sheds some light on the sort of work done by the firm:
With information gleaned from more than 30,000 data points during a game, Stathletes offers hockey’s decision makers rich insight that can help them ensure an optimal mix of talent as well as guide them in training their players.
Obviously, this is vague, but it seems as though they basically track everything possible through intensive video review. This doesn't seem dissimilar from what Sportlogiq claims, though Sportlogiq automates some of their data collection. The data that the above sources hint at Stathletes collecting is what's commonly referred to as microstats -- tiny elements of the game that are broken down, coded to specific situations, tracked, and then analyzed. It covers a wide swath of the game, and is somewhat polarizing in and of itself (refer to the end of the article). Frankly, this doesn't narrow it down much. We know they collect data, and lots of it. How much of it is useful? How much of it relates to possession, or winning? Are they reliable or random? The answer to each of those is "we don't know."
It's easy to roll your eyes at the idea that ideas need to be 'vetted' by hobbyists on Twitter, but peer-review is a thing for a reason and some of the best hockey minds in the world are currently working in the public sphere. They're more than capable of assessing whether someone is legit, or full of it. We don't know what bucket Stathletes falls into.
Now, that doesn't mean that they're automatically garbage. I want to emphasize that a lot, because I definitely don't want this article seen as me bashing Stathletes for not being public with their methods and data. It just means that as fans, we don't know their (and by extension, Chayka's) work the way we do someone like Eric Tulsky's. We know Stathletes had/has NHL teams as clients (Vancouver and Arizona are mentioned by name in this article by James Mirtle, which also indicates there were several more lining up). Obviously, that's a good sign, though NHL teams certainly aren't infallible with regards to these decisions.
But ultimately, there's a lot we don't know. When it comes down to it, as outsiders, it's very hard to assess Chayka's analytics background for that reason. As with most personnel decisions, we'll have to evaluate it as more information comes out (and even then, we won't know what decisions/ideas are his, and what aren't).
Those who have met Chayka seem to be pretty unequivocal in their praise for him (that Mirtle article contains some quotes that are quite fawning). And though I've never met him, he seems incredibly impressive. How can you not be impressed by someone who broke into the old boys club of the NHL at this young an age? It's remarkable, and he deserves admiration for that alone. As time passes, we'll see if he's able to prove Arizona right. I'm definitely going to be rooting for him.
Aside on Microstats
In researching for this piece, I reached out to the excellent Domenic Galamini (@MimicoHero), one of the pre-eminent public hockey analysts, for his thoughts on microstats and their usefulness. The piece then took a different direction, but I thought his answer was insightful and poignant and (with his permission) I wanted to share it. In my opinion, he rather eloquently summarizes why the tracking of microstats is so polarizing by describing their benefits, and the potential pratfalls of collecting them. You should definitely give him a follow on Twitter, and contribute to his GoFundMe if you can. His HERO and WARRIOR charts are among the most used data visualizations in hockey today, and are an invaluable resource.
Q:
Do you think there's useful data in micro-events, and if so, do you think the cost of mining that data in terms of person-hours is worth it?A:
I think micro-stats can be useful when trying to unearth the "why" regarding the results we’re observing at a macro-level like Corsi, scoring chances or expected goals. They can help determine who is driving play or anchoring play on a particular line/pairing, where the strong points/soft spots are in a team’s system/an individual’s game and so forth.
That being said, for micro-stat tracking to be worth the time and effort (and trust me, it can be incredibly tedious and draining), you need to understand their limitations. Even though they are events involving individual skaters, micro-events are still going to be influenced by contextual factors and in no way can they be used to replace measured results. Secondly, you need to have precise definitions of what you are tracking or else you’re opening the door to a world of bias and subjectivity.
Finally, and most importantly, what you track needs to be related to results at a macro-level. This can be difficult since you typically don’t have a dataset to work with when experimenting with micro-stat tracking. This means that you’re probably going to have to figure out what works from a theoretical standpoint first and then do some trial and error on the fly.