Monday, January 26, 2009

Continuing the Dialogue About Pomeroy Rankings

A couple weeks ago, I briefly touched on some of my concerns about using the Pomeroy rankings as a predictive tool. One game this weekend reminded me that I wanted to explore the topic a little further.

Before I do that, let me say that I love the Pomeroy Rankings. Like DVOA in football and third-order records (among many many other metrics) in baseball, they give us a predictive tool that a huge majority of the "squares" ignore. And for the most part I think it's useful.

But like all tools, it does have a couple problems, and it helps sports bettors if we understand them. There's one in particular that's always bothered me about Pomeroy, and that is its failure to account for teams "taking their foot off the gas" in blowouts. This is particularly important in college basketball because blowouts happen all the time, and because certain teams/styles of play are prone to playing in far more or far fewer blowouts than other teams.

To explain what I'm talking about, let's look at this weekend's Duke-Maryland "game." As you probably saw, Duke kicked the ever-loving shit out of Maryland, 85-44. As you might expect, it's given Duke a substantial cushion atop the Pomeroy rankings, and rightfully so. But I think the system undervalues Duke's performance on Saturday? Why? Well, Duke was winning 60-20 five minutes into the second half. Yes, preposterous. From that point on (most of the second half), Duke outscored Maryland by only one. Maryland poured in 24 more points over 15 minutes, which is the equivalent of a 64-point game on the offensive end. But is it fair to say that "Duke" was playing "Maryland" at that point, for purposes of predictive data? I don't think so. It's pretty clear that the game was over. However, the possessions that took place over those last 15 minutes will be weighed equally with the rest of those teams' respective seasons in the Pomeroy data.

In most cases, anomalies like this are few and far between, and when compiling data over the course of a season, they tend to equalize. But that's not the case in college basketball, where elite teams that play at a fast pace play in a huge number of blowouts and may have played as much as 10% of their seasons thus far with their foot off the gas. I wonder why Pomeroy doesn't simply discount data past a certain point margin/time remaining point in the game. Say, the last 12 minutes if a team is winning by 40 or more, the last 8 minutes if it's 30 or more, and the last 4 minutes if it's 20 or more. What's the harm in that?

I've noticed this concern not because I see fast-paced teams being undervalued, but because I see slow-paced teams being overvalued. To take this to an extreme, imagine, as a player or coach, the difference in your attitude in if you have a 90-70 lead with five minutes to play as compared to a 45-35 lead. Because slowdown, half-court, limited possession teams don't play in blowouts due to their style, they don't have as many "wasted" possessions. Makes sense, right?

This year I'm keeping an eye on Georgetown in this respect. Georgetown is #12 in the Pomeroy rankings despite their 6 losses, mostly due to their ridicuously difficult schedule. But are they overvalued? They've only attempted 915 shots 100 less than anyone else in not-particularly uptempo Big East, so we know they're a slow tempo team (and also one that doesn't do well on the offensive boards). And if you scan their games to date, you'll find a lot of wins with scores in the 50s and 60s and a few wins against cupcakes in which they only scored in the 70s. It's safe to assume that they played hard to the final whistle in most of those games, whereas equivalent performances from an efficiency standpoint by higher tempo teams would have resulted in a lot of garbage time.

Moral of the story: Use Pomeroy. Love Pomeroy. It's as valuable a predictive tool as you'll find out there. But be aware that it may have some limitations. When a snail's-pace team like Washington State finds itself matched up against a team like North Carolina in the tournament (as happened in 2008), be aware of this issue before you play the standard fade of the public sentiment based on Pomeroy's love of the Cougars. And for God's sake, Grover, stop making Bo Ryan's Wisconsin teams your upset Final Four pick in your brackets just because Pomeroy keeps telling you they are a top ten team.

I'm nothing more than an interested amateur when it comes to all these fancypants numbers, so comments/critiques are welcome and encouraged in the comments.

4 comments:

James Sherrill said...

Good points. You forgot to mention the flipside, though, were it is say Gonzaga -25 against Pepperdine. When Pomeroy has it at 27 you know Gonzaga's garbage time is factored in from other blowouts earlier in the season.

Like all stats, you have to realize they have shortcomings and what they are.

xlssports said...

I think your point is well stated. This idea could likely be tested and I'd expect your hypothesis to pass.
With regards to the Pomeroy stats, I see the constraint as a lack of data. DVOA became possible since Aaron Schatz (et al.) were willing to gather, clean, and analyze data down to a play-by-play level. With that granularity, DVOA became capable of omitting the tail end of blowout games.

I'm sure NCAAB analysis will continue to grow along a similar path and I look forward to whomever brings us this data, be it the expanding BasketballProspectus or others. In the meantime I'll wonder what rule of thumb should be applied when sluggish Pomeroy powerhouses play against their peers.

I am building a set of data that might help in sketching out a short term answer, but my immediate reaction is a lean toward "pass"

Anonymous said...

love the discussion! do you think it is easy for a fast team to get a slow team to play fast. that would then give them the advantage that would not be seen in the ratings

Grover said...

Thanks for the comments. The aim of my piece was really to point out a problem rather than propose a solution (as I told Hambone: I'm a lawyer, not an engineer). Really, it's just something we should be aware of when using Pomeroy to put their card and their brackets together- which everyone should still do.

That said, I'm not sure why he doesn't just eliminate results after a certain point in certain games. That would address the concerns I expressed here, and I don't think it would be too difficult to do. Perhaps I'm wrong though: Pomeroy is certainly smarter than I am, and I'm sure he has his reasons.

I did a decent review of his methodology and didn't see anything about this concern, other than an offhand remark about him not having a "cap," and recognizing the problem with that, and his continuing search to address it. Is this problem what he's referring to? It wasn't entirely clear from the wording.

Or does he actually eliminate late-game blowout scenarios somehow? It's possible that he does and I just didn't see it. I've been known to make mistakes.