Most projection sites only publish the wins. We ran the full backtest, including the experiments that made accuracy worse, and the positions where we still trail the market.
This is not a methodology paper — that is coming separately. This is the scoreboard: what the harness measures, the per-position numbers from a fresh run, what we tried that failed, and where the model is still weak.
What we measure
Every model change has to clear a regression guard before it ships. The guard is a real historical backtest, not a hand-picked highlight reel.
Setup (projection backtest):
- Project season 2025 using only history from 2023 and 2024 — the same information available before the 2025 season.
- Full-population nflverse regular-season fixtures (not a curated top-N board).
- Score projected season points against actual season points under reference PPR + tackle-heavy IDP scoring — the product's core league shape.
- Players need usable prior history and at least 8 actual games in the target season. Season-ending injuries measure availability risk (a Monte Carlo problem), not whether we ranked healthy seasons correctly.
- Primary ranking metric: Spearman rank correlation (ρ) between predicted and actual points — do we order players correctly?
- Secondary: MAE per game — absolute error on a per-game basis so injury-shortened seasons do not dominate the error table.
Setup (market benchmark):
- Project season 2024 from 2022 and 2023.
- Join period-correct Fantasy Football Calculator preseason PPR ADP (real drafts from that offseason).
- Score our projection ranks and market ADP ranks against the same actuals on the identical player sample — only players who have our projection, an FFC ADP match, and 8+ games.
- Market data is evaluation-only. We do not ship, sell, or redistribute ADP; the harness only reports same-sample rank correlation.
Spearman of 1.0 is perfect order. 0.0 is random. Negative means the ranking points the wrong way. Anything clearing ~0.7 on a full position population is strong signal; below ~0.5 is "useful but leaky."
Per-position results (2025 projected from 2023–2024)
Fresh harness run on this codebase (ProjectionBacktestTest, target=2025):
| Pos | Players | Spearman ρ | MAE/game |
|---|---|---|---|
| RB | 77 | 0.840 | 2.61 |
| WR | 130 | 0.787 | 2.41 |
| TE | 76 | 0.781 | 2.08 |
| S | 15 | 0.750 | 2.01 |
| LB | 173 | 0.705 | 2.30 |
| CB | 119 | 0.628 | 1.83 |
| DE | 86 | 0.560 | 1.59 |
| DB | 24 | 0.529 | 2.47 |
| QB | 31 | 0.508 | 3.82 |
| DT | 83 | 0.462 | 1.29 |
| K | 28 | 0.265 | 0.12 |
Weighted (player-count weights, same formula the CI guards use):
- Offense (QB/RB/WR/TE): ρ ≈ 0.771 on 314 players
- IDP (DT/DE/LB/DB/CB/S): ρ ≈ 0.614 on 500 players
Skill-position offense is the strong suite. Linebackers clear 0.70 on a huge sample. Interior DL and generic DB pool are harder — role volatility and scheme matter more than last year's tackle total. Kickers are noise with a low MAE because the point scale is tiny; we score them so the table is honest, not because anyone should draft off ρ = 0.265.
What actually improved accuracy
A few signals graduated only after the harness measured lift (not vibes):
- Expected games from availability history — stop assuming every player plays 17.
- Position age curves — year-over-year production decays and peaks differently by position.
- Usage-informed receiving — targets-implied receptions blended toward position catch rates (especially helped TE rank correlation on the market-matched sample).
Each shipped behind a feature flag, cleared an ablation, then raised the CI floor. Signals without lift do not ship — which is the next section.
Negative results (what we tried that failed)
Snaps-implied tackling degraded IDP accuracy
Hypothesis: defensive snaps × positional tackles-per-snap should regress noisy tackle volume toward role-implied volume — the same shape that helped receiving.
Result: monotone degradation at every blend weight on both validation seasons. REG-only re-measure (SnapSignalSweepTest):
| off | w=0.3 | w=0.5 | w=0.7 | |
|---|---|---|---|---|
| IDP ρ target 2025 (n=500) | 0.605 | 0.592 | 0.579 | 0.558 |
| IDP ρ target 2024 (n=491) | 0.573 | 0.567 | 0.561 | 0.547 |
Interpretation: tackle efficiency per snap is a stable skill, not luck that should shrink toward the mean the way catch rate does. Forcing that regression pulls good tacklers toward mediocrity. Not shipped. Snap fixtures stay in the repo for cheaper retries (per-game share while active, trend, floor indicators).
Injury-listing shade was a null result
Hypothesis: shade expected games by Questionable (or all-listing) rate beyond what games-played history already captures.
Result: accuracy moved by roughly ±0.001–0.003, inconsistent in sign across both validation seasons; the all-listings variant mildly hurt. Not shipped.
Interpretation: expected-games from games-played history already absorbed most of the season-total injury signal. Variants still worth retrying: injury-type severity, multi-season chronicity, and week-grain availability as a Monte Carlo distribution input rather than a rank input.
Ours vs the market (identical sample)
Market comparison is the honest gap check. Target 2024, FFC preseason PPR ADP, identical sample only (MarketBenchmarkTest):
| Pos | Players | Ours ρ | Market ρ |
|---|---|---|---|
| K | 13 | 0.669 | -0.081 |
| QB | 22 | 0.285 | 0.420 |
| RB | 48 | 0.610 | 0.689 |
| WR | 55 | 0.444 | 0.483 |
| TE | 18 | 0.201 | 0.598 |
Weighted: ours 0.463 vs market 0.504.
We trail. The market's edge is news, depth-chart roles, and camp buzz we do not ingest as ranking inputs. TE is the loudest gap on this sample — opportunity and role signals are the highest-leverage work queue. RB is our closest competitive line. Kickers flip the script (we beat ADP noise), which is entertaining and mostly irrelevant for drafting.
We will not invent competitor projection leaderboards we do not run. This is the only external benchmark the harness defends: same players, same actuals, rank correlation only.
Known weak spots
QB remains soft (ρ 0.508 on the full 2025 backtest; 0.285 on the smaller market-matched 2024 sample). Small sample, rushing volatility, and scheme swings all hurt. This is a standing improvement target, not a footnote.
Rookies and thin priors. The backtest requires usable history, so pure rookies without a college/prior path do not dominate these tables — and that is exactly where real draft boards get hardest. Draft-capital and college-production priors are the honest next front, not another spin on last year's rate tables.
Interior DL / secondary variance. DT and mixed DB pools still leave rank signal on the table; scheme and role beat raw volume.
We still trail the market on offense ADP. Closing that gap without laundering market rankings into the product is the point of the learning loop: measure, ablate, ship only lift.
None of this is hidden in a "future work" appendix. The CI floors sit under these numbers so a clever-looking change that tanks rank correlation fails the build.
Why this matters for your league
These tables are under reference PPR+IDP scoring. Your league's tackle weights, flex slots, and IDP starter counts change who is valuable — that is the whole product. Projection accuracy is the substrate; league-aware ranking is how those rates become a board that matches your rules instead of consensus.
If you want the methodology deep-dive (how rates, age curves, and VORP are built), that is a separate post. This one is the report card.
LeagueFrenzy is live now. Paste your league URL and see your board free — early launch Season Passes are discounted while we keep tightening the model against these same harnesses.