Before the 2017 season, I was thinking quite a lot about predictions. This included how to create a model and try and build an image of what will happen, what we can predict (and what we can’t), and how we can measure our accuracy.
The last point is super interesting to me. How do we really measure how good we are? Rankings? Stats? Standard deviation? They’ve all got strengths and weaknesses, so I tried to come up with something that would be both measurable and visceral – something that would give our predictions a score but also just show how right or wrong predictions are in an easy-to-grasp manner.
The method I ended up going with was simple. I wrote up 100 statements about the 2017 season and simply asked a selection of analysts to predict whether they’d be true or false. The statements were calibrated to try and make them all roughly a 50/50 chance.
This is an account of how accurate we were. It’s also really useful to see how things we thought were obvious turned out to be untrue. And vice versa.
Just over half the analysts agreed the Cardinals would not have an effective lineman. Calais Campbell was exceptional for years but with him gone no one really stands out on that unit. Chandler Jones led the league in sacks and Haason Reddick had an extremely difficult rookie season. He played two positions and honestly didn’t look good in either. 78% of respondents thought he would play a lot but he failed to manage the target.
More than half the respondents saw Vic Beasley wouldn’t have a good season. Frankly, I’m surprised so many people thought he would hit that target. No one here thought Duke Riley would come in and take a starting job, although it was a widespread belief at the time. Unsurprisingly no defensive back managed five picks. That’s a rare number. IDP players routinely overestimate interception numbers.
Just over a fifth of respondent s thought the Ravens would have a top pass rusher. Terrell Suggs turned back time to hit 11 in the end. C.J. Mosley was a bit of a gamble in 2017. In 2016 he had an awful season (with Zach Orr piling up numbers) and we didn’t quite know what to expect. Eric Weddle was a popular pick after a very good 2016 but he shared stats with Tony Jefferson in 2017 and failed to hit the (fairly low) target.
Reggie Ragland was traded away by the Bills just before the season and failed to play a single snap for them. Woe to those respondents who thought he’d be a full-time starter in Buffalo.