Rating the Raters

Welcome back, Dr Blackheart. Here is the good doctor's latest guest column -- a rating of the raters, he calls it.

***************************************************************
As my post-season NCAA hoops withdrawal kicks in, I decided to look for a back edition of Sports Illustrated in my private library. No, not the “Swimsuit Issue” but the November 22nd “College Basketball” preview. It was such a crazy season of “The Unexpected” that I wanted to look back and see how well SI did in predicting the 2010-11 season. I remember this edition arrived in the mail just before I hopped on a plane to Kansas City for the CBE where the unranked Warriors were to face SI’s #1, #3 or #14 ranked teams in a pre-Thanksgiving tournament.

MU played well against the favored competition back then, but I was left with a lot of uncertainty around their experience, roles and identity. As it wound up, this uncertainty spread to all of college basketball and lasted down to the last half of the championship game. Reflecting, I figured that it was worth a look-see to determine how well all the mainstream experts did predicting this crazy season. After all, this is what we pay them to do—to help us average fans separate our irrational loyalties from the facts. So I dug up their preseason to final rankings for a postmortem face-off (Note: I used each poll’s final rankings to compare to preseason--whether the poll’s last ranking was at the end of the regular season or post-NCAA’s.)

So, how well did these experts test out? The answer: Their test scores needed to be curved and they all seemingly copied off of each other. Here are some quick snippets of their performances:

• No pundit had the eventual national champion in their Top 25’s. Sports Illustrated didn’t even have UCONN in their preseason “Field of 68”.
• All graded out around 57% on the pre to post test overall…including the pre-post Top 25’s.
• Each had a C/D grade for the Elite 8’s being ranked in the Top 25 preseason…and a C- for predicting closely for the Round of 32.
• Virginia Tech, purported to be the last team out of the Big Dance on Selection Sunday, was the one team the experts had in their Top 25’s who did not make the Big Dance.
• Most of the Top 10 teams in each poll’s final ranking were at least listed in their preseason Top 25’s.
• The coaches (USA Today/ESPN) were no better or worse than the media.

So who won out? Basically, they were all subterranean on their “expert” prognostications. To try to get to some conclusion, however, I created an index based on the average score of all the raters for each element and in total. So, a 1.00=average of the five raters. This provides a relative score for each versus the other to determine if any one stands out. Well, that didn’t help much as they all appear to be guilty of “group think” as their picks were almost the same. Sports Illustrated did score a bit better—and I mean a ‘lil bit—so I will have to crown them the “The King of the Misfits”-- although they only were able to get 45 out of the “Field of 68” right preseason—or 66%.

So what’s the take away from all of this? Archie the Bartender is probably a more reliable source of expertise. Or, even better, strike up a conversation with your local librarian.