Beating the NFL Draft: Part Three

Editor’s note: This is the final piece in a three part research paper. Check out part one and part two before you proceed.

DISCUSSION OF FINDINGS

Obviously, while the outcome is not entirely unexpected given what I believed coming into the project, a model that beats the draft by a wide margin is an audacious enough result that it caused me to go back to both professors to review my approach.  In addition to those reviews and the subsequent vetting of this paper, I believe there are several reasons to expect these results will hold up to a more rigorous review:

• Everything reported here has been cross-validated.

• In addition to walking both professors through the process and presenting it to the entire department, I also went back to the original data and worked through it from scratch again to document the process and replicate the original results.

• Calling what was in my head a hypothesis would be too strong, but I had a clear image of the relationships in the data and the model is a good step forward in terms of representing that mental image. The formal model is backed by intuitively simple ideas.

• The break points defining WR types and variables segments weren’t arrived at via trial and error or optimization during this analysis. The final models are the first formal models built using the segmented records and segmented variables, and many of the break points were similar to those used naively for the last few years.

With all that said, if these results are correct they may represent only a partial realization of the total lift available over draft position:

[am4show have=’g1;’ guest_error=’sub_message’ user_error=’sub_message’ ]

• Sample sizes are too small to fully replicate the naïve models, but I believe those models are better than the one presented in this paper. Fortunately, the available sample will continue to grow each year as more players enter the league.

• Using the best two of a player’s first five years in the response variable may overstate the value of draft position as a predictor. The best measure of a player may be what he does after his rookie contract, and with the passage of time we’ll be able to shift the response variable to the best three of the first six years, best four of eight, etc. to capture more data from the “second contract” period.

• Since the break points defining WR types and variable segments weren’t arrived at via trial and error or optimization, there could be some marginal value to tweaking those breakpoints or segmenting the variables differently.

 

SUBSEQUENT WORK, IMPROVED RESULTS & LIMITATIONS

Despite having finished the practicum and made a final presentation, work to improve them continued afterward.  Those findings are presented here.

“Physical” and “Developmental” Latent Variables:  In addition to exploring neural networks, I wondered what might happen if the variables were combined to create two latent variables (variables that are not directly measured, but inferred from other variables that are directly measured)  – physical and developmental:

Physical Variables – Height, BMI, Size, Speed, Explosion, Agility

Developmental Variables – logDraft, Age, Volume, Skill

Grouping the variables this way, running separate stepwise regression for each vs the response variable, and using the output of those regressions as inputs in a new model improved the overall results significantly.  Taking the best available result for each WR type using different permutations of this approach further improved the total lift vs draft position:

(Note that the raw totals reported here are not directly comparable to the raw totals in the table above due to a change in the response variable that shifted the total error downward across the board.)

Again, the raw numbers can’t be compared across the two tables due to an improvement in the way the response variable is calculated, but the overall lift here compared to draft position alone is now approximately 160%.

It’s possible that some of the lift is due to measuring only the receiving component of a WRs value (and not blocking or special teams value), but similar improvement shows up even looking at only early round picks (who are presumably being drafted primarily for receiving skill).

One interesting item in these results is that when the data is organized in this way it’s possible to beat draft position without using draft position as in input.

A second interesting finding is the relative value of draft position vs the metrics.  If the error reduction provided by the draft only (9.1) and the metrics only (17.0) were summed it would result in a total error reduction of 26.1.  Given that the actual error reduction for draft plus metrics is almost as large (23.5) it implies that there is very little overlap between the two – that draft position has historically measured something valuable, but has made relatively little use of the quantitative data available.

 

LIMITATIONS OF THE RESEARCH

There are a couple caveats in terms of actually using these models.  First, they were built using data from 1999-2011, so don’t reflect what’s happened in the last four years.  Given the explosion in big data and analytics it seems likely that draft position increasingly reflects these findings and that it will become increasingly difficult to beat draft position going forward.  And in fact in 2013 and 2014 the average “development” score (the output of the developmental variables component of the model) for drafted WRs has gone up by nearly 50%.

On the other hand, it seems likely that looking at additional or improved inputs in a new way might improve these models even further.

The second caveat is  that there’s a large portion of the original dataset for which the data is incomplete.  In some cases missing data can be estimated or may not be relevant, but in many cases missing values mean that a prospect can only be modeled less formally.

Finally, these models were built as an approximation of a naïve model developed via intuition and years of playing with the data informally, and there are instances when the output is wrong (where predicted values for a player don’t match the NFL career outcome) for reasons I believe are explainable.  In particular I believe that the sample size limitations preventing the data from being subset into 10 or 12 WR types result in higher average errors than might be realized with a larger data set.

Regardless of the limitations, these results demonstrate the importance of quantitative measurements and that data from a player’s NCAA career and the NFL combine can be used to improve on more subjective methods and that draft position, at least through 2011, has not realized the full value of that data.

[/am4show]