Why are course ratings and slope not based on score data?

[larrybud]

On 11/12/2022 at 10:04 AM, jvincent said:

 

Shotlink, Arccos, and Shotscope are all based on shot data, not scores. The sample set is so large that any anomalies in the data are inconsequential.

 

I know with a small team of programmers I could have a statistical handicap system built in under a year.

 

The only challenge would be for courses that don't have mapping data.

 

Shotlink can be taken out of the discussion since that is PGA Tour level data and has no bearing on anything else.

Shot tracking app data is only as good as those who are recording the shots. There's now way to know whether the rounds which were recorded are accurate or not (e.g. missed tagged shots, incorrect pin locations, not playing by the rules, etc).

 

[larrybud]

On 11/13/2022 at 3:58 PM, jvincent said:

The scratch scoring average becomes the rating and the bogey average is used to calculate slope. Doing it this way would actually allow you to create different slopes for different index ranges, but that's probably more complicated than it needs to be.

The whole point of slope is to adjust FOR different index ranges, as it refers to a the "slope" of a line on a graph. The higher the slope value, the progressively more difficult it is for higher indexed players. It is, in effect, a multiplier.

 

The handicap system isn't perfect, but invalid course ratings has to be way down on the list it's problems.

 

You keep saying we have better data, but we really don't. Shot tracking apps are subject to human judgement and failures just like course ratings are. It is NOT objective data, so the dataset as a whole really isn't that good, and there's no good way to clean up the garbage data.

 

Now, I do think (and maybe they do this already) that the USGA could run analysis on a course by course basis to detect whether the differentials posted are out of whack compared to the differentials of the same players played at other courses. If the same set of players consistently have an average differential 2-3 lower or higher at Course "A" vs all other courses, then it might trigger a review of the rating at Course A to see what's up. 

[jvincent]

25 minutes ago, larrybud said:

The whole point of slope is to adjust FOR different index ranges, as it refers to a the "slope" of a line on a graph. The higher the slope value, the progressively more difficult it is for higher indexed players. It is, in effect, a multiplier.

 

The handicap system isn't perfect, but invalid course ratings has to be way down on the list it's problems.

 

You keep saying we have better data, but we really don't. Shot tracking apps are subject to human judgement and failures just like course ratings are. It is NOT objective data, so the dataset as a whole really isn't that good, and there's no good way to clean up the garbage data.

 

Now, I do think (and maybe they do this already) that the USGA could run analysis on a course by course basis to detect whether the differentials posted are out of whack compared to the differentials of the same players played at other courses. If the same set of players consistently have an average differential 2-3 lower or higher at Course "A" vs all other courses, then it might trigger a review of the rating at Course A to see what's up. 

 

There are countless discussions on this forum about how a single slope isn't good enough for the range of handicaps it has to cover. A range based system could, in theory, improve that with different slopes for different index ranges.

 

I would argue that there is very little judgement in the tracking systems. There are certainly errors / uncertainties in the data but I'm betting that the amount of data available smooths that out.

 

Mark Broadie has published at least one paper where he assesses relative course difficulty based on PGA Tour strokes gained data. IIRC the context was that the OWGR didn't properly account for course difficulty.

 

[larrybud]

30 minutes ago, jvincent said:

There are countless discussions on this forum about how a single slope isn't good enough for the range of handicaps it has to cover. A range based system could, in theory, improve that with different slopes for different index ranges.

 

I would argue that there is very little judgement in the tracking systems. There are certainly errors / uncertainties in the data but I'm betting that the amount of data available smooths that out.

 

Mark Broadie has published at least one paper where he assesses relative course difficulty based on PGA Tour strokes gained data. IIRC the context was that the OWGR didn't properly account for course difficulty.

 

 

I'm not sure you really understand what "slope" is, in context of the handicapping system.

 

You can't smooth out data when most of it is garbage. I'll give you an example of a common problem with all of these systems (which I'm a user of): 30 foot putt, player hits it a foot to the hole. The player doesn't record their conceded (gimme) putt. They adjust their shot tracking after the round and they add a shot. By default, the system places a shot at the 15 foot mark, between the location of the first putt and the center of the green. Player then hits "save" without making any adjustments to where the 2nd putt took place and the hole location.  You've then completely skewed a number of aspects with just 1 data point: The proximity to the hole of the first putt, the distance made for the 2nd putt, and the hole location. 

 

I *know* stuff like this happens all the time when looking at other player's shot tracking. I don't know about Arrcos, but with GameGolf and Shotscope, you can see other player's rounds. You'll see repeated holes where a player two putts from X distance, with the 2nd putt exactly half the distance from the hole. I'm a data freak, and my putting SG stats vs non-pga tour players caught my eye. Something didn't quite look right, so I started looking at other people's rounds. If they are missing such critical data as putts, I'm sure they are screwing up their other data.

 

PGA Tour data is completely different. They play the same courses generally year after year, they have shotlink which measures every shot down to the foot, and scores played under the rules of golf, with essentially the same set of players from tournament to tournament. But that has zero to do with the 15,000 other courses which are rated. AMs are swiping in 4 footers, not playing stroke and distance, rolling the ball, etc. Two sets of data with completely different quality.

[Celeras]

Forgive me if I didn't see it already mentioned, I only scanned the conversation. 

 

This already exists and it is called GHAP: https://ghap.golf/

 

Quote

The Course Difficulty System was developed to identify the real difficulty of every Golf Course from every Teebox. It leverages more than 30 million real golf rounds to perform this calculations.

The purpose is to be able to calculate the real Score Value of a Score. Since an 85 at your Local Muni is not the same as an 85 at a Major Championship venue.

“While other Handicap Systems rely on people to assess the difficulty of a golf course we rely on real data from real golfers.”

Relevant Definitions:

• Tee Statistical Par: Represents the real par of a golf course’s teebox for a Scratch golfer, based on real data as shown in GHAP’s Course Difficulty Database.

• Tee Hdcp Adjustment %: It is a percentage used to adjust the Tee Statistical Par to the different Handicap Levels.

 

[Halebopp]

9 hours ago, larrybud said:

Now, I do think (and maybe they do this already) that the USGA could run analysis on a course by course basis to detect whether the differentials posted are out of whack compared to the differentials of the same players played at other courses. If the same set of players consistently have an average differential 2-3 lower or higher at Course "A" vs all other courses, then it might trigger a review of the rating at Course A to see what's up. 

 

The PCC already provides similar data. I've said this before that the one interesting thing the PCC does provide is the data on courses with an abnormal amount of adjustments. You'd just need someone to process the data and start searching for possible patterns in course types, design etc. to figure out if there are issues within the rating system itself. At my home club we had PCC adjustments on about 7% and 30% of the days during the first season under WHS.

[klebs01]

7 hours ago, Celeras said:

Forgive me if I didn't see it already mentioned, I only scanned the conversation. 

 

This already exists and it is called GHAP: https://ghap.golf/

 

 

how did they determine what a scratch golfer was?

[North Butte]

Looks like this GHAP did all of jvincent's hard graft for him. He can just sign up for one of those handicaps and his problems are solved. 

 

Their "30 million rounds" presumably means they are mining data from Arccos and/or similar shot trackers. That would make this in effect the handicap system for golfers who live in one of the shot tracker ecosystems but don't like the ersatz "handicap" stats in their tracking app. 

Edited November 18, 2022 by North Butte

[ChaosTheory]

Lot of belligerence here, kinda disappointing.  Surprised this is such a sacred cow, not to be questioned or considered for improvement.  Something like this will come someday.  The present system is a holdover from times before easy mass number crunching was available.  
 

Okay carry on.

[ChaosTheory]

1 hour ago, North Butte said:

 

Edited November 18, 2022 by Snowman9000

[jvincent]

9 hours ago, Celeras said:

Forgive me if I didn't see it already mentioned, I only scanned the conversation. 

 

This already exists and it is called GHAP: https://ghap.golf/

 

 

 

Sort of. My quick read of GHAP is that it is score based.

 

What I am proposing is based on shot data, not score data.

[North Butte]

Any handicapping system has to be "score based" by definition. Scoring is the thing being handicapped. Not shot-making. 

[DaveLeeNC]

38 minutes ago, North Butte said:

Any handicapping system has to be "score based" by definition. Scoring is the thing being handicapped. Not shot-making. 

If you can generate usefully accurate scoring statistics from something that is shot-making based, that would (in principle) work. It is a relatively common analytical technique. I am not sure that I would "choose to go there" if I was trying to improve the current system, but (in principle) it could work. 

 

dave

[North Butte]

5 minutes ago, DaveLeeNC said:

If you can generate usefully accurate scoring statistics from something that is shot-making based, that would (in principle) work. It is a relatively common analytical technique. I am not sure that I would "choose to go there" if I was trying to improve the current system, but (in principle) it could work. 

 

dave

It may indeed be possible for something along those lines to work in principle if you were trying to prove a point. What point it would prove, other than being an interesting academic exercise, is beyond my comprehension.

 

But even then, it would require better data than just convenience samples from a commercial shot tracking app's database. 

[ChaosTheory]

https://www.ibm.com/cloud/learn/monte-carlo-simulation

[klebs01]

6 hours ago, Snowman9000 said:

Lot of belligerence here, kinda disappointing.  Surprised this is such a sacred cow, not to be questioned or considered for improvement.  Something like this will come someday.  The present system is a holdover from times before easy mass number crunching was available.  
 

Okay carry on.

Its not a sacred cow at all. He is solutioning an undefined problem. First start with a problem statement and convince us why it’s a problem that should be addressed. 

[jvincent]

30 minutes ago, klebs01 said:

Its not a sacred cow at all. He is solutioning an undefined problem. First start with a problem statement and convince us why it’s a problem that should be addressed. 

 

Easy.

 

The current system is a holdover from before there was sufficient data analytics. It relies on subjective evaluations of course obstacles as part of the calculation.

 

Additionally, it has an arbitrary constant value as a part of the calculation which was presumably reverse engineered to arrive at what was arbitrarily determined to be the correct number for (presumably) a sample set of golf courses.

 

Given the amount of data available to us now it is possible to derive a course rating based on that shot data.

[ThinkingPlus]

7 minutes ago, jvincent said:

 

Easy.

 

The current system is a holdover from before there was sufficient data analytics. It relies on subjective evaluations of course obstacles as part of the calculation.

 

Additionally, it has an arbitrary constant value as a part of the calculation which was presumably reverse engineered to arrive at what was arbitrarily determined to be the correct number for (presumably) a sample set of golf courses.

 

Given the amount of data available to us now it is possible to derive a course rating based on that shot data.

I presume the data driven approach you are suggesting bootstraps off the current system because I don't really see an apriori, non-arbitrary way to do it otherwise.

[North Butte]

  

Just now, klebs01 said:

I don’t see a problem to address in there. There are some opinions expressed on the current state, but no problem that requires action. 

 

The argument boils down to "Anything not invented in the last few years is obsolete". He is saying that ANY system not based on Arccos or some other shot tracking gadget is by definition obsolete and therefore wrong.

Edited November 18, 2022 by North Butte

[klebs01]

21 minutes ago, jvincent said:

 

Easy.

 

The current system is a holdover from before there was sufficient data analytics. It relies on subjective evaluations of course obstacles as part of the calculation.

 

Additionally, it has an arbitrary constant value as a part of the calculation which was presumably reverse engineered to arrive at what was arbitrarily determined to be the correct number for (presumably) a sample set of golf courses.

 

Given the amount of data available to us now it is possible to derive a course rating based on that shot data.

I don’t see a problem to address in there. There are some opinions expressed on the current state, but no problem that requires action. 

[North Butte]

45 minutes ago, jvincent said:

 

Easy.

 

The current system is a holdover from before there was sufficient data analytics. It relies on subjective evaluations of course obstacles as part of the calculation.

 

Additionally, it has an arbitrary constant value as a part of the calculation which was presumably reverse engineered to arrive at what was arbitrarily determined to be the correct number for (presumably) a sample set of golf courses.

 

Given the amount of data available to us now it is possible to derive a course rating based on that shot data.

Can't you see that you still have not defined any problem with the current system beyond the fact that you would do it differently if you were doing it? 

 

So go invent your own system like those GHAP folks did and sell subscriptions to your superior handicapping service. 

 

Edited November 18, 2022 by North Butte

[jvincent]

34 minutes ago, ThinkingPlus said:

I presume the data driven approach you are suggesting bootstraps off the current system because I don't really see an apriori, non-arbitrary way to do it otherwise.

 

If you look at the current system the most measurable criteria are average driving distance and then the ability to reach a 470 yards hole in two. Then it gets very fuzzy with a description of scoring ability.

 

Bootstrapping off the current system would be one way to filter data, but there is probably another way.

[jvincent]

28 minutes ago, North Butte said:

  

 

The argument boils down to "Anything not invented in the last few years is obsolete". He is saying that ANY system not based on Arccos or some other shot tracking gadget is by definition obsolete and therefore wrong.

I thought you were out of the discussion?

 

I never said wrong. I said that I believed a data driven system would be better. And note that I've never said it would be perfect either.

[North Butte]

2 minutes ago, jvincent said:

I thought you were out of the discussion?

Yes, that's right. My mistake. Won't happen again.

[jvincent]

22 minutes ago, North Butte said:

You're proposing to replace two "arbitrary" numbers for each course (The CR and Slope) with your own pair of "arbitrary" numbers. The only difference is one was arrived at by successive refinement over a period of decades while yours will be arrived at by someone poring over a bunch of Arccos data and making "arbitrary" decisions about how to convert that into two numbers for each course. 

 

There has to be a conversion from all those "shot based" data into expected scoring at some point. And that is not automatic, it requires a statistical model and that requires a person to choose how to construct the model.

 

 

I don't know why you don't understand the premise of a statistical scoring model

 

For a "scratch" golfer, every shot distance has an expected distribution. Have a computer play each hole a very large number of times using the shot distribution, and generate an average score.

 

The only arbitrary part of that is the definition of a scratch golfer.

[ThinkingPlus]

1 minute ago, jvincent said:

 

If you look at the current system the most measurable criteria are average driving distance and then the ability to reach a 470 yards hole in two. Then it gets very fuzzy with a description of scoring ability.

 

Bootstrapping off the current system would be one way to filter data, but there is probably another way.

I don't see any other choice than to use the current handicap indices to provide bins to qualify the shot data as associated with scratch players and bogey players.  Given that association you could use the shot data to get dispersion for binned distances all the way to the hole.  At that point you can Monte Carlo a course with your synthetic scratch and bogey players to produce most likely outcome scores for each. 

 

Course rating and slope can be directly computed at that point. Rinse and repeat for each tee and gender. Once done for all courses, you would be gathering new shot data from that point on and would eventually end up with a new set of scratch and bogey players. Obviously the intersection of the set of old scratch players with the set of new scratch players should be non-null, but there will be some differences (likewise for bogey players).  You will also then get new dispersion data and rinse repeat again and again over years.

 

Presumably there is convergence.  Certainly the members of the scratch set and the bogey set will change over time.  One would hope that the dispersion converges and you would get a stable definition of scratch golfer and bogey golfer as well as stable ratings and slope.  Who knows if it would truly be a "better" system.

[jvincent]

7 minutes ago, ThinkingPlus said:

I don't see any other choice than to use the current handicap indices to provide bins to qualify the shot data as associated with scratch players and bogey players.  Given that association you could use the shot data to get dispersion for binned distances all the way to the hole.  At that point you can Monte Carlo a course with your synthetic scratch and bogey players to produce most likely outcome scores for each. 

 

Course rating and slope can be directly computed at that point. Rinse and repeat for each tee and gender. Once done for all courses, you would be gathering new shot data from that point on and would eventually end up with a new set of scratch and bogey players. Obviously the intersection of the set of old scratch players with the set of new scratch players should be non-null, but there will be some differences (likewise for bogey players).  You will also then get new dispersion data and rinse repeat again and again over years.

 

Presumably there is convergence.  Certainly the members of the scratch set and the bogey set will change over time.  One would hope that the dispersion converges and you would get a stable definition of scratch golfer and bogey golfer as well as stable ratings and slope.  Who knows if it would truly be a "better" system.

 

Yep, exactly how I think it would work.

 

One thought on avoiding the boot strapping issue would be to filter shot data that averages getting up and down for par from within a prescribed distance. Since the "scratch" golfer is supposed to shoot par I think that would be a reasonable approach.

[ThinkingPlus]

8 minutes ago, jvincent said:

 

Yep, exactly how I think it would work.

 

One thought on avoiding the boot strapping issue would be to filter shot data that averages getting up and down for par from within a prescribed distance. Since the "scratch" golfer is supposed to shoot par I think that would be a reasonable approach.

One issue in this data driven scenario would be the almost non-existent data for women.

[DaveLeeNC]

Probably just because I have geeky tendencies, I like the idea behind this approach (different than saying it should be put in place right now or even seriously investigated right now). But I wonder if this might better be used as a tool for the rating committee rather than as a methodology. When the rating committee gets done with the course tour/measurements, they could use simulations based on this technique to fine tuneand or check the hole by hole scratch/bogey ratings that they will come up with. 

 

dave

 

ps. Lots of words about 'no demonstrated problem to solve' therefore we should not be discussing this. That seems to be a standard rarely enforced around here (and I don't think it should be enforced anyway). 

[betarhoalphadelta]

1 hour ago, North Butte said:

 

The argument boils down to "Anything not invented in the last few years is obsolete". He is saying that ANY system not based on Arccos or some other shot tracking gadget is by definition obsolete and therefore wrong.

 

Essentially you're building a model. And you know what they say? 

 

All models are wrong. Some of them are useful. 

 

I don't think @jvincent is saying that the current model isn't useful. I think he's suggesting that a data-driven analytics model would be more useful.