NCAA Outcome Predictor

a self-retraining basketball pipeline that caught itself cheating

CHARTER VII

The number went down. That is the correct result.

Most student machine-learning projects train once on a fixed dataset and stop there. This one fetches real NCAA game data from ESPN across three seasons, about 2,900 games, with no API key needed. It then trains six models in competition, keeps whichever scores best, serves it behind a web dashboard, and runs the whole cycle again on its own every six hours. It has been improving itself since the day it was deployed.

The interesting part is not the accuracy. An early version scored an AUC of 0.9666, which is the kind of number that should make you suspicious rather than pleased. It was cheating. One of the features held what a team shot during the game, so the model had learned that teams who shoot well win, and it was reading the result off the box score while calling it a forecast. That is data leakage, and it is the most common way a model looks brilliant and is worthless.

The fix was to make every feature something you could actually know before tip-off. Each field now holds a rolling average of that team’s last ten games going into that night, so a shooting percentage of 0.47 means "this team has averaged 47% over its last ten", not "this team shot 47% in the game I am pretending to predict". Games where either side has no history yet are dropped from training entirely. The AUC fell from 0.9666 to about 0.74. The lower number is the honest one.

Because that mistake is easy to make twice, the pipeline now checks for it before any model sees the data. Four validations run first: leakage detection flags any feature correlating above 0.70 with the outcome, near-zero variance features are caught, class balance has to sit between a 40% and 70% home win rate, and a sample-to-feature ratio check catches the conditions that cause overfitting. Tree depth is capped automatically from the size of the dataset, so a small dataset cannot be overfit just because a config file said it could.

The rest is built the way a pipeline in a real job would be, rather than as a notebook. Structured logging, a versioned model registry that can roll back, a REST API, deduplication, adaptive regularisation, and a background scheduler that retrains without anyone asking it to. One command runs the whole thing.

  • Python 62.4%
  • HTML 37.6%

Built from

  • Python
  • scikit-learn
  • Flask
  • pandas
  • ESPN API

What it is not

  • Not a betting tool, and not advice. An AUC of about 0.74 means it is better than a coin toss and a long way from certain.
  • The public demo runs on a mix of synthetic and real NCAA data, so the metrics shown there do not represent real prediction performance. The honest figure is the one quoted here.
  • It predicts from form going into a game. It knows nothing about injuries, travel, or anything else that never reaches a box score.

Screenshots

NCAA Outcome Predictor: matchup prediction
Matchup prediction

Roads out

Related articles

  • Coding war crimes: learning from deliberately terrible code

    Most advice about writing good code arrives as a rule you are told to follow. This is the opposite: write the terrible version on purpose, measure exactly what it costs, and let the rule be the conclusion instead of the premise.

    4 min read · 2026-08-28

Neighbouring holdings

↑ back to the Afon Empire → read the chronicles ↑ return to the map