A few months ago, Doug Drinen wrote a series of fascinating articles on his blog based on 10,000 simulations of the NFL season. One of the surprising findings of his study was that the strongest team in the NFL is likely to win the Superbowl only about 24% of the time, and even the worst team in the NFL could win the Superbowl by random chance.
His study inspired me to conduct a simulation study to determine the effectiveness of the NFL tiebreaking procedures for identifying the better of the two tied teams. Isnt that what a tiebreaker should do? What I mean by better is the team with the higher power ranking, as used in my software. So I basically simulated the 2005 season 100,000 times, using the end-of-season power rankings. For the purposes of this study only, I am treating the power ranking as a true indicator (as opposed to approximate indicator) of each teams relative strength. For each simulation, I noted whether each tiebreaker decision favored the team with the better or worse power rating. For simplicity, I only considered two team tiebreakers. My software breaks all ties (even between 3rd and 4th place teams in a division) so I only considered tiebreakers between teams with win-loss-tie percentages of 0.600 and better to emphasize the most important tiebreaking decisions. Listed below are the fraction of time each tiebreaker favored the better team:
Head to head: 0.48
Division: 0.505
Common Opponent: 0.485
Conference: 0.51
Strength of Victory: 0.45
Strength of Schedule: 0.47
These numbers are accurate to the number of significant figures shown, as they remain consistent with repeated simulations with different random seeds. Two points jump out: 1) For the most part tiebreakers are no better than a coin flip for determining which of two teams is better. 2) Most of the tiebreakers seem to favor the weaker teams by small, but statistically significant amounts. This could be an artifact of my 2005 power ratings and the 2004 finishes (which determines the 2005 schedule). I havent yet found an obvious explanation. Ill do this analysis again at the end of 2006 and see if the trends are real.
If the NFL tiebreakers are not effective in identifying which team is intrinsically stronger, then what do they do? Obviously, they emphasize certain games as being more important that other games. Each teams NFL schedule breaks down as follows:
6 division games (home-and-away against the three division foes)
4 conference common-division games (division vs. division within the conference)
4 non-conference games (division vs. division across conferences)
2 additional conference games (determined by division place finish from previous season)
Everyone knows that the division games are most important, followed by the conference games, followed by the non-conference games, right? Not so fast. Those non-conference games turn out to be pretty important in determining the division championship because every team in your division plays them, and the record against common opponents tiebreaker is considered before the record within the conference tiebreaker. To determine exactly how important each type of game is in 1) winning the division and 2) making the playoffs, I did another simulation study. Here, I set the power rankings of each team equal and eliminated home field advantage. This made every game a 50-50 chance for either team to win. I then pre-set the outcome of one division game (e.g., Chiefs beat the Broncos) and then simulated the whole NFL season 50,000 times leaving the outcome of every other game to chance. I recorded how much of an advantage winning the division game made. I then repeated the exercise with each of the other three types of games described above. Below, Ive reported how much winning each type of game helps in terms of (winning the division) and [making the playoffs], normalized to a baseline of winning a non-conference game:
Division game: (1.23) [1.07]
Conference Common-Opponent game (1.00) [1.03]
Additional Conference game (0.98) [1.03]
Non-Conference game (1.00) [1.00]
This analysis indicates that the division games are most important both for winning the division and making the playoffs. After those, there is very little difference in the importance of the remaining games. In fact, if winning the division is your primary goal, the non-conference games are slightly more important than the two games that are determined by your placing from the pervious year.
A couple caveats should be mentioned. By making all the teams equal strength, we are going to get more ties than a typical NFL season, so the differences of the various types of games are exaggerated here. So in reality, the small differences between any non-division games are actually even smaller. Secondly, this study ignores the correlation of a teams strength from one year to another. For example in 2006 the Chiefs play Jacksonville and Miami because all three teams finished second in their division in 2005. Lets say all the teams in the AFC finish in exactly the same place as in 2006. All of those second place teams are competing for wild card places and those head-to-head wins could come in handy. However, even though there is some correlation in a teams strength from year-to-year, there is usually enough shake up in the standings that this shouldnt introduce too much error.
One think’s for sure, there are not many other places you can get this kind of information.