Fix College Soccer's Point/Ranking System?

I couldn't help but think about the point system the WCC uses for the standings and how it relates to college soccer. They are clearly following the 3-win, 1-draw, 0-loss model that most all leagues across the globe use. However, I think a big problem with this system as it relates to college soccer (and college soccer only) is that we play overtime to determine a winner and only concede draws after overtime. No league with the 3-1-0 system plays overtime. I know that college soccer prefers overtime to determine a winner and loser because it affects RPI, which is the preferred (albeit flawed) ranking metric.

While we have not lost in overtime in WCC play, it seems ridiculous to me now that our three overtime losses in non-conference are recorded as losses. We played a good game to a draw. By anyone else's measure, we earned a result from that (against UNLV, Cal, and Virginia). Then the rules forced us into a position to play for more, where we came up short. What do we get from that effort? Nothing.

There is a benefit to winning in overtime (3 points and a win) but absolutely no benefit to losing in overtime (0 points and a loss). I noticed that in hockey they award 1 point to an overtime loss, which at least rewards the team for playing a good game in regulation. I wish we had some way to include that in our point/ranking system. Maybe just do away with overtime altogether. That would seem to be the best solution.

Thoughts?

I agree with this point. Overtime is an abomination except when needed (i.e., elimination games in tournaments).

An alternative that borrows from a procedure used in youth tournaments: Record the match as a draw, then play extra time as a tie-breaker that is *only* used if the two teams are level in the table at the end of the season. That gives you the advantage of the tie-breaker, but only if it is needed. (The youth tournaments use kicks from the mark, but extra time is less bad than that.)

I don't think that the college soccer overtime rule has anything to do with the RPI. I've experimented with the RPI, to see if giving special treatment to overtime games would improve it -- such as treating an overtime loss as less than a full loss and vice versa -- and it doesn't improve the RPI. The RPI, and other mathematical rating systems, simply aren't precise enough for that to make a significant difference.

I've not seen any discussion from those in the college coaching community about eliminating overtimes, so it appears to be something they're comfortable with.

UPSoccerFanatic wrote:I've experimented with the RPI, to see if giving special treatment to overtime games would improve it -- such as treating an overtime loss as less than a full loss and vice versa -- and it doesn't improve the RPI.  The RPI, and other mathematical rating systems, simply aren't precise enough for that to make a significant difference.

I believe you, but I'm honestly surprised that using an NHL-style system (in which overtime losses count as a tie) wouldn't make a difference in the RPI for a team like this year's Pilots, which has three OT losses.

Would there really not be an RPI difference between a 6-5-0 record and a 6-2-3 record?

Stonehouse wrote:

UPSoccerFanatic wrote:I've experimented with the RPI, to see if giving special treatment to overtime games would improve it -- such as treating an overtime loss as less than a full loss and vice versa -- and it doesn't improve the RPI.  The RPI, and other mathematical rating systems, simply aren't precise enough for that to make a significant difference.

I believe you, but I'm honestly surprised that using an NHL-style system (in which overtime losses count as a tie) wouldn't make a difference in the RPI for a team like this year's Pilots, which has three OT losses.

Would there really not be an RPI difference between a 6-5-0 record and a 6-2-3 record?

Sorry, I wasn't clear. Yes, treating games that go to overtime differently would have an effect on individual teams' ratings and ranks.

But, it wouldn't make the RPI any more accurate. Here's what I mean by that:

It's possible to measure how well ratings, adjusted for home field advantage, correlate with actual game results. I've made this measurement for the Adjusted RPI and I've also done it for a system that treats overtime games as closer to ties than to wins/losses. My hypothesis was that this alternative system would provide ratings that correlate better with game results than the ARPI. My hypothesis was wrong, it didn't make a difference. What that means is that a system that treats an overtime game differently may move teams around, but it doesn't make the rating system, on average, more representative of all teams' actual results.

For Division 1 women, which is what I study, the current version of the Adjusted RPI produces ratings that, when adjusted for home field advantage, match game results 72.7% of the time, that are incorrect because the game is a tie 10.6% of the time, and that are incorrect because the better rated team loses 16.7% of the time. (That's based on 10 years' data, roughly 30,000 games.) Alternative rating systems, such as "technically correct" Elo-like systems or Kenneth Massey's excellent system, produce nearly identical correlation rates. My conclusion so far has been that this is close to the best a mathematical rating system can do.

What distinguishes rating systems thus isn't what the correlation rate is, rather it's how the "missed" results are distributed. For the current version of the ARPI that the women use, there is a pattern to the "missed" results. On average, the conferences with higher average ARPIs have better game results than the ratings say they should and the conferences with the lower average ARPIs have poorer results than the ratings say they should. The ARPI that the men use has some differences, but for the men I expect the same thing would be true.

On the other hand, there's an alternative version of the ARPI (that the NCAA doesn't use) that doesn't discriminate among conferences, and Massey's system doesn't discriminate. So they have the same "error" rate as the current version of the ARPI, but the errors appear to be more randomly distributed, which to my mind is fairer.

For the women, and I'm pretty confident it would be true also for the men, the Soccer Committee appears to use the RPI mainly to identify candidate groups of teams -- seed candidates and at large selection candidates (the "bubble" teams). Once it's identified the groups of candidates, it looks at detailed data on the teams. The detailed data include data in Team Sheets (a sheet for each team) that includes the result of every game the team played. In relation to overtime games, I think there is a shortcoming in the data the Committee gets, which is that the data does not identify overtime games. That, I think, is an important deficiency.

Wow! Thank you for that thorough explanation. If that is what your data tells you, then I guess I have no reason to worry. Do you publish this information anywhere? I would love to read into it more and have a better discussion.

I do want to know what you mean by "missed" results. Is that games that RPI predicts should score one way, but finishes another? Is RPI calculated as a regression then?

SouthCarolinaPilot wrote:Wow! Thank you for that thorough explanation. If that is what your data tells you, then I guess I have no reason to worry. Do you publish this information anywhere? I would love to read into it more and have a better discussion.

I do want to know what you mean by "missed" results. Is that games that RPI predicts should score one way, but finishes another? Is RPI calculated as a regression then?

Here's where you can find all the info you want on the RPI as used for women.  Most of what's there applies equally to men.  It also has information on how the Women's Soccer Committee's decision-making process works, which probably is similar to how it works for the men.  

https://sites.google.com/site/rpifordivisioniwomenssoccer/

Yes, by "missed results" I mean games that the RPI predicts should score one way, but it finishes another way. No, it's not calculated as a regression. At the linked website, if you go to the RPI Formula page, you'll see exactly how it's calculated for Division I women. The basic architecture is the same for all NCAA sports, including men's soccer.

I echo the thanks to UPSF from SouthCarolinaPilot. UPSF has been a tremendous source of good quality information for several years on all things related to NCAA division 1 women's soccer on this forum and we are truly blessed to have him as a member and contributor.

UPSoccerFanatic wrote:For the women, and I'm pretty confident it would be true also for the men, the Soccer Committee appears to use the RPI mainly to identify candidate groups of teams -- seed candidates and at large selection candidates (the "bubble" teams).  Once it's identified the groups of candidates, it looks at detailed data on the teams.  The detailed data include data in Team Sheets (a sheet for each team) that includes the result of every game the team played.  In relation to overtime games, I think there is a shortcoming in the data the Committee gets, which is that the data does not identify overtime games.  That, I think, is an important deficiency.

UPSF, thanks as always for your detail and insight.

I wonder what the argument *against* including the OT information is? Just simply that a win is a win and a loss is a loss and overtime ultimately doesn't matter?