New RPI Info -- Read and Weep

Since there seems to be a bunch of "waiting, bored, for the season to begin" angst out there, I thought I'd give you all something to be mad about. The beginning of this is background, but keep reading because it gets good (or really bad) at the end.

Some of you know that I have an obsession with the NCAA's RPI as applied to Division I women's soccer. As part of my obsession, I've developed a parallel RPI system to provide information about how the RPI is working since the NCAA only publishes its RPI rankings a couple of times during the season and, in particular, does not publish the RPI based on data through the last of the regular season (including conference tournament) games. It is too bad the NCAA does not publish this, since this is the RPI the Women's Soccer Committee uses in selecting at large teams for NCAA tournament participation; in seeding teams; and in placing teams in the tournament bracket.

One of the difficulties I've had in running a parallel system is that after the NCAA computes a basic RPI, which I can match, it has a system of adding bonuses and subtracting penalties for good and bad wins and ties. The bonus/penalty system involves 24 variables and it's very hard to figure out.

I have spent a good part of this summer trying to home in on the correct bonus/penalty awards. I finally concluded that it was impossible for me to come up with awards that would result in my RPI rankings exactly matching the NCAA's. That caused me to wonder whether the NCAA and I were applying the RPI formulas to different data sets. When the NCAA publishes its end of the season RPI (meaning RPI that includes the results of the NCAA tournament games, which has no practical use at all), it publishes teams' win-loss-tie and home-away-neutral records too. To check for data errors, I compared my win-loss-tie and home-away-neutral records to the NCAA's. Low and behold, I was missing some data. But, low and behold too, the NCAA had some incorrect data (which they have agreed was incorrect). Since the NCAA isn't going to regenerate its RPI with its data corrected, I revised my data so it included the NCAA's incorrect data and then got back to work. I still don't have the exactly correct bonus/penalty amounts but am pretty confident in the bonuses because I now am able to match the NCAA's rankings for roughly the top 80 teams, which is plenty for NCAA tournament purposes.

While still working on the bonus/penalty problem, I decided to do something else. I decided to see how the NCAA's data errors might have affected last year's tournament, if at all.

The first thing I did along that line was to compare the regular season ending RPI with the Women's Soccer Committee's at large selections and seeds, to see how much weight the Committee gives to the RPI in its Championship tournament decision-making process. (The Committee is supposed to consider three primary factors: the RPI, head-to-head game results, and results against common opponents. It also can consider recommendations from regional advisory committees. In a stalemate, it can consider two secondary factors.) Here's what I learned:

1. The Committee's 34 at large selections for participation in the tournament came from the RPI's 36 top ranked teams! (The two that got bumped out of this group were Dayton and Arizona State.)

2. The Committee's 16 seeds were the RPI's 16 top ranked teams! The NCAA has four #1 seeds in the bracket, four #2 seeds, etc. The placement of teams withing these groups did not exactly match the RPI ranks. For example, USC got a #2 seed rather than a #3, most likely based on its head-to-head win over Portland. But overall, the groupings followed the RPI fairly closely with reasonable explanations for differences.

In other words, it appears pretty clear that what the NCAA does is follow the RPI very closely unless head-to-head results or results against common opponents justify a change.

With that knowledge, I then compared the RPI rankings based on the NCAA's erroneous data with what the rankings would have been based on the correct data. Here's where it gets fun.

The erroneous data involved two Sun Belt Conference tournament games, one involving Denver v. Florida Atlantic and the other involving Arkansas Little Rock v. Middle Tennessee. Think "Denver" as you read this. Both games were 0-0 ties followed by shootouts. Under the RPI formula, those games are treated as ties. However, the NCAA data lists them as wins by Denver and ALR/losses by Florida Atlantic and Middle Tennessee. What that means is that Denver and ALR ended up with higher RPIs than they should have had and Florida Atlantic and Middle Tennessee had lower RPIs.

Those of you who remember last year's season and are familiar with the RPI may guess where this is going. During the regular season, Denver played Stanford. Part of Stanford's RPI is based on how well Denver did during the season. Thus since Denver's win-loss-tie record, as recorded in the NCAA data base, was better than it should have been, Stanford' RPI was higher than it should have been.

So, when I computed the RPI using the NCAA's erroneous data, what were the RPI rankings of the top six teams?

1. North Carolina
2. Penn State
3. UCLA
4. Stanford
5. Portland
6. Texas A&M

But, when I corrected the data and recomputed the RPI, what were the rankings?

1. North Carolina
2. Penn State
3. UCLA
4. Portland
5. Stanford
6. Texas A&M

(The Stanford/Portland switch, by the way, was the only change among the RPI's top 50 teams, which more than covered all at large selections.)

Now, we don't know what the Committee would have done if it had had the correct RPI. In its seeding process last year, based on the RPI it had, the Committee gave North Carolina, Penn State, UCLA, and Stanford #1 seeds and gave Portland, USC (11 in the RPI, moved to a #2 seed probably because of its win over Portland), Texas A&M, and Purdue #2 seeds. (I actually think the Committee seeds the teams 1-16, based on bracket placement, and had both USC and Texas A&M seeded ahead of Portland.) But, if the Committee had had the correct RPI, any of Portland, USC, Stanford, or Texas A&M might have gotten the fourth #1 seed.

The benefit of a #1 seed is that notwithstanding the NCAA's travel rules, the four #1 seeds are guaranteed home field for tournament rounds 1 and 2 as well as for the round of 16 and the quarter-finals.

In other words, this was a very significant error by the NCAA that may have resulted in Portland getting shafted.

(I've provided all this information to the NCAA staff responsible for the RPI. Also to those on the U of P staff who might be in a position to complain about one more shafting of U of P soccer by the NCAA. I'll be publishing the information elsewhere over the coming weeks as I have time, but I thought it would give all of you something to "grrr" about.)

Typical. It might also explain a bit why the Stanford RPI didn't look right when we all saw it last year. I remember there being discussions about it.

I'm not so interested is the GRRR part as I am the transparency part. If it weren't for your diligence, we would never know about this kind of stuff.

It seems to me that if the RPI is going to be used as such an important criteria, there should be checks and balances to ensure that things will actually BE fair.

Having an open system seems the only way to do it. Keeping the RPI computing closed just seems like an invitation to perpetuation error and inviting unchecked corruption.

I'm sure this was an honest error, but what if it Hypothetically wasn't? how would it ever get corrected in time for the Tournament?

Great job again, UPSF

Thanks, Geez. I agree this looks like an understandable and good faith error. And I agree completely that having the process open is what's really critical at this point. That way, everyone will understand what the RPI is, how it works, and how errors in data gathering can impact decisions.

Unfortunately, one of the things I'm concluding is that the NCAA's lack of transparency, as reflected (1) in the secrecy about parts of the RPI formula, (2) in the unwillingness of the NCAA to make public the data to which it applies the RPI formula, (3) in the secrecy of the actual RPI numbers (as distinguished from the rankings derived from those numbers), and (4) in the secrecy of the RPI information provided to the women's Soccer Committee following the completion of the regular season games, which is the only RPI information that counts, is that the NCAA has the conscious intention of keeping the public from actually understanding what it is doing.

I think the NCAA has two fears. One is that if what it is doing is open, it will have to deal with every nut in the world who has a gripe about how his or her team got treated this year. The other is that the NCAA knows there are problems with what it is doing. As one example, I bet the NCAA does not want people to know about the potential for data errors to affect important decisions, since people knowing about that would lessen the credibility of NCAA tournaments' at large selection and seeding processes. As another example, I bet the NCAA does not want people to know that from a statistical perspective, the RPI has a pretty big standard error so that its ratings are very imprecise, but that the various sports committees nevertheless use the RPIs as though they are very precise (as I have learned is done by the Women's Soccer Committee). And most certainly, the NCAA does not want people to know that at least for sports that play only the number of games played in soccer, the RPI does not work as a tool to reasonably compare teams from one region of the US to teams from other regions, resulting in discrimination against teams from the stronger regions and for teams from the weaker regions.

In other words, the NCAA uses the RPI as "cover" to explain its decisions; and is afraid it will not be able to continue doing this if its process is fully public.

I am somewhat sympathetic with the NCAA's plight. In a sport like soccer, with the limited number of games, there is not going to be any clearly and indisputably credible basis for making tournament decisions. To deal with that problem, the NCAA apparently has decided a very flawed but secret basis is better than a public process with all its warts exposed. Like you, I don't like it -- I understand it, but I think it's wrong.

You make a compelling case, UPSF.

You've done a mountain of work and somehow managed to uncover what seems a terrible injustice, whether the committee was aware if it or not. The past is past. However, perhaps your tenacity will result in a more fair and transparent process going forward, if people you send the info to are inclined to act on it. The more pressure put on the NCAA, the more likely change will be in the air.

Hip, Hip Hooray!!! You're very determined with your work and beliefs, and you certainly are a very important piece of the puzzle that is the "PURPLE CULT" of Pilot Nation.

Yes, I'm a very PC person. PC T-shirts, anyone?

UPSoccerFanatic wrote:Yes, I'm a very PC person. PC T-shirts, anyone?

The cult of the Purple

Resistance is futile. You will be assimilated.

I just wish they'd do away with the RPI and just have conference playoffs where the winner moves on to the National playoffs. Only conference playoff champions participate in the playoffs. Conference playoffs are seeded based on conference standings. National seeding would be more difficult.

-SciFi

UPSoccerFanatic wrote:

Yes, I'm a very PC person. PC T-shirts, anyone?

That's actually a great idea, UPSF!

To the folks responsible for promoting this site: Can you think of a better/cheaper way to promote PN than to have a bunch of us walking around games, in ALL sports, wearing T-shirts that read, "PURPLE CULT" at the top with "of Pilot Nation" below it?

You might give it some thought. We can all buy our own T-shirts. You'd just need some "up-front/pro bono work" money/services to design and make X amount of shirts. There are probably people reading this right now with the skills and resources to help make this happen.

Who? Me?

SciFi wrote:I just wish they'd do away with the RPI and just have conference playoffs where the winner moves on to the National playoffs. Only conference playoff champions participate in the playoffs. Conference playoffs are seeded based on conference standings. National seeding would be more difficult.

-SciFi

Actually, this was the exact policy for many years on the basketball side: One--only one--from each conference. The conference tournaments really mattered for everyone back in the 60s and 70s because the big teams weren't just playing for seedings.

I think Pilot Nation shirts would be a great idea. And anyone heard about more soccer scarves this year?

PurplePrideTrumpet wrote:And anyone heard about more soccer scarves this year?

I'd ask about this over at the TalkTimbers board: www.talktimbers.com

-SciFi

PurplePrideTrumpet wrote:

SciFi wrote:I just wish they'd do away with the RPI and just have conference playoffs where the winner moves on to the National playoffs. Only conference playoff champions participate in the playoffs. Conference playoffs are seeded based on conference standings. National seeding would be more difficult.

-SciFi

Actually, this was the exact policy for many years on the basketball side: One--only one--from each conference. The conference tournaments really mattered for everyone back in the 60s and 70s because the big teams weren't just playing for seedings.

I think Pilot Nation shirts would be a great idea. And anyone heard about more soccer scarves this year?

Check the UP website Bookstore/apparell link and you will find "rugby scarves" since Im not aware of a rugby proggie at UP I'm thinking these would suit soccer fans nicely. I am getting one for sure for my collection (I have one for team USA and Newcastle in EPL). Lets all get em. :face:

Now that the Image for the scarves is up, not quite what I had hoped for... Just a regular single color scarf... Meh? I'll wait for something unofficial I guess.

I think he's talking about this link or This.

Whoops! My bad!

-SciFi

Now if your sig line was a scarf, I'd rock that to soccer matches