Are the forecasts right?
A probability is only worth something if it is honest: of the teams we gave a 70% chance, about 70% should get there. This page checks that against everything we published in finished editions, day by day, versus what actually happened.
That is 7 finished editions and 60,734 forecasts: every chance published for a team in an objective, at every update, while the outcome was still open. When the published chance was between 10% and 20%, the outcome happened 13.6% of the time. When the published chance was between 50% and 60%, the outcome happened 59.1% of the time. When the published chance was between 80% and 90%, the outcome happened 89.7% of the time. In the 50–60%, 60–70%, 70–80%, and 80–90% ranges, the outcome happened more often than forecast: there the model was conservative. In the 40–50% ranges, it happened less often than forecast: there the model overshot. The overall Brier score is 0.086 (0 would be getting everything right with certainty). A naive forecast giving everyone the same average chance would score 0.140: the model's error is 38% lower.
Forecast vs. outcome, by range
The bar shows how often it happened; the tick, the average forecast. In a calibrated forecast the bar ends at the tick.
| Forecast range | Forecasts | Average forecast | Happened | Comparison |
|---|---|---|---|---|
| 0–10% | 36,241 | 3.2% | 3.8% | |
| 10–20% | 9,044 | 14.4% | 13.6% | |
| 20–30% | 4,454 | 24.7% | 26.4% | |
| 30–40% | 3,070 | 34.9% | 33.8% | |
| 40–50% | 2,124 | 44.7% | 39.9% | |
| 50–60% | 1,658 | 54.5% | 59.1% | |
| 60–70% | 1,001 | 64.8% | 71.6% | |
| 70–80% | 869 | 74.9% | 83.7% | |
| 80–90% | 786 | 85.0% | 89.7% | |
| 90–100% | 1,487 | 96.6% | 97.6% |
By competition
Brier score of each division (lower is better). A more balanced league has more genuine uncertainty and therefore a higher Brier — it doesn't mean a worse model.
| Division | Editions | Forecasts | Brier |
|---|---|---|---|
| La LigaSpanish League | 2 | 22,383 | 0.058 |
| Brazilian Série ABrazilian Championship | 2 | 16,806 | 0.132 |
| Premier LeagueEnglish Football League | 2 | 16,201 | 0.072 |
| World CupFIFA | 1 | 5,344 | 0.105 |
How we count
Every chance published in the daily updates of each finished edition counts, for every team and objective (title, berths, relegation, cup rounds). Chances already decided — mathematically secured or eliminated, or pinned at 0% or 100% — are left out, because they don't test the model. The outcome is where the team finished the edition.
What the Brier score is
It is the average of the squared difference between the forecast chance and the result (1 if it happened, 0 if not). Giving 90% to something that happened costs 0.01; giving 90% to something that didn't costs 0.81. Lower is better.
Limitations of this count
A team's forecasts on consecutive days are very similar, so the number of forecasts overstates the amount of independent evidence. There are still few finished editions — the count gets firmer with every season that ends. And good calibration in aggregate doesn't guarantee each call: a 10% chance happens, once in ten.