Just when you thought it was safe to go back to your homes… another theory emerges!
True Score Theory, aka Classical Test Theory, is the idea that a performance is the sum of true ability and random error.
Random error in a sporting context can be thought of as injury, weather change, the bounce of a ball… much of which can be interpreted as luck.
If you like sports modelling, grab yourself a copy of Michael J. Mauboussin’s book “The Success Equation: Untangling Skill and Luck in Business, Sports and Investing”.
The book draws on the theory, essentially saying that skill and luck both contribute to outcomes and that differing activities tend towards their own position on a luck-skill continuum. As an example, he raises roulette as essentially an activity governed by luck, whereas chess is predictably holding its place down at the skill end of the spectrum.
So where do AFL and NRL sit, and how do you go about assessing that?
Mauboussin draws on one aspect of the theory which postulates that:
Variance in observed results = Variance attributable to skill + Variance attributable to luck
If a sport was all skill, on a ladder you would get a predictable sliding scale of results where the most skilful team wins every encounter, the next all but one, all the way down to the team who win nothing for the season. If a sport is all luck, you end up with many teams winning around half their games, and a handful by sheer luck doing better or worse than the rest.
Variance is a calculated measure of the difference from the mean across a series of results (it’s the standard deviation squared). So from the above, variance should be higher where skill is playing a larger role.
Let’s look at the just-completed 2016 ladders for both AFL and NRL (variance is a simple function in Excel):
VarObs AFL = 24.1
VarObs NRL = 19.8
This would indicate that skill played a larger role in the results of the AFL season than NRL.
But what is the proportion of skill compared to luck?
Mauboussin uses the normal approximation to the binomial distribution to get a value for the variance we would expect if game outcomes were nothing but luck (the right-hand side of the equation), dead simple as it is the number of games teams play x probability of a win (0.5) x probability of a loss (0.5). That gives us:
VarLuck AFL = 22 x 0.5 x 0.5 = 5.5
VarLuck NRL = 24 x 0.5 x 0.5 = 6
Now we have two pieces of the puzzle, so VarSkill can then be calculated by the difference:
VarSkill = VarObs – VarLuck
VarSkill AFL = 24.1 – 5.5 = 18.6
VarSkill NRL = 19.8 – 6 = 13.8
Not sure about you, but I find the numbers pretty damn scary as it indicates that luck forms 5.5/24.1 or 23% of AFL performance, and 6/19.8 or 30% of NRL.
This compares somewhat with what Mauboussin found in US NFL – across 5 seasons luck figured at an average of 38%.
Here is the local equivalent:
|Average last 5 seasons||25%||49%|
Check out that NRL figure – basically 50:50 luck and skill.
Activities that are weighted towards luck tend to exhibit more likely reversion to the mean, i.e. teams can often follow a flashy or poor performance with something more expected of them. I’ll leave you to think about whether NRL seems to exhibit that more than AFL, as I don’t watch AFL.
OK, you are about to throw all analytics out and bet on whims… but here are some comments:
- Where there is only a minor difference in relative skill level across teams, luck will of course have a larger role in the determination of outcomes
- This is only a theory
- Regardless, rating is a good exercise and odds should reflect uncertainty
- Although this provides context, I don’t see where this necessarily helps you predict luck in any one game
- It raises the issue of the cost/benefit of collection of ever more data around games given the background of uncertainty, i.e. what data is significant enough to matter, and what is fluff?
Food for thought… seeking comment!
For a good account and the maths see here.
And if you want to skip the book, check out this talk from Mauboussin.