Totals are easier to forecast than single items
The M5 Competition asked teams to forecast 42,840 Walmart unit sales series 28 days ahead, from total sales down to single products in single stores (organiser's page). The organisers note that most of the series are intermittent, with sporadic sales including zeros. The official scores show how much accuracy falls as the data gets more granular.
| Level | Series | Winner (YJ_STU) | Best benchmark (ES_bu) | Seasonal naive |
|---|---|---|---|---|
| Total sales | 1 | 0.199 | 0.426 | 0.560 |
| By state | 3 | 0.310 | 0.514 | 0.673 |
| By category | 3 | 0.277 | 0.478 | 0.623 |
| By department | 7 | 0.365 | 0.557 | 0.708 |
| By store | 10 | 0.400 | 0.580 | 0.718 |
| By state and category | 9 | 0.390 | 0.577 | 0.760 |
| By state and department | 21 | 0.474 | 0.654 | 0.829 |
| By store and category | 30 | 0.480 | 0.643 | 0.801 |
| By store and department | 70 | 0.573 | 0.728 | 0.888 |
| By product | 3,049 | 0.966 | 1.012 | 1.223 |
| By product and state | 9,147 | 0.929 | 0.969 | 1.205 |
| By product and store | 30,490 | 0.884 | 0.915 | 1.176 |
| Average of all levels | 42,840 | 0.520 | 0.671 | 0.847 |
Source: "Scores and Ranks.xlsx" in the organisers' M5-methods repository, checked 13 November 2026. The score is WRMSSE, a weighted root mean squared scaled error: each series' errors are scaled by the errors of a naive forecast on its own history (RMSSE in Forecasting: Principles and Practice, 5.8). Lower is better.
At the total level the winner's error was 53% lower than the benchmark's (0.199 against 0.426). At the product-and-store level the gap was 3% (0.884 against 0.915). If a predictive assistant forecasts your company's total demand, expect the first kind of accuracy; if it forecasts one item in one shop, expect the second.
Some frequencies gain little over a naive forecast
The M4 Competition scored 100,000 series at six frequencies. The winning method beat the Naive2 benchmark at every frequency except daily, where it was 4.6% worse on OWA.
| Frequency | Winner (Smyl) sMAPE | Naive2 sMAPE | Comb sMAPE | Winner's OWA (Naive2 = 1) |
|---|---|---|---|---|
| Yearly | 13.176 | 16.342 | 14.848 | 0.778 |
| Quarterly | 9.679 | 11.012 | 10.175 | 0.847 |
| Monthly | 12.126 | 14.427 | 13.434 | 0.836 |
| Weekly | 7.817 | 9.161 | 8.944 | 0.851 |
| Daily | 3.170 | 3.045 | 2.980 | 1.046 |
| Hourly | 9.328 | 18.383 | 22.053 | 0.440 |
Source: "Evaluation and Ranks.xlsx" in the organisers' M4-methods repository, checked 13 November 2026. On daily series every method had a low sMAPE, and the plain Comb benchmark (an average of three exponential smoothing methods) was the most accurate of the three on that measure. On hourly series the winner's OWA was 0.440, less than half of Naive2's. Before paying for a complex model, check whether your data behaves like the daily series here: the forecast score calculator compares any forecast with a naive one.
Questions about future events
Forecasting a one-off event, such as an election result or a policy decision, has no long history to extend. ForecastBench scores forecasts of such questions with the Brier score, where 0 is perfect and always saying 50% scores 0.25.
| Forecaster, ForecastBench paper (200 questions, 2024) | Brier score |
|---|---|
| Superforecaster median | 0.096 |
| General public median | 0.121 |
| Best LLM (Claude 3.5 Sonnet, with the crowd forecast on market questions) | 0.122 |
| Best LLM without the crowd forecast | 0.136 |
| Always forecasting 50% | 0.250 |
Source: Karger et al., ForecastBench, table 2, checked 13 November 2026. Lower is better. Superforecasters were significantly more accurate than both the public and the best model (p < 0.001).
The live leaderboard has moved since. It reports a Brier Index, which turns the difficulty-adjusted Brier score into a percentage where 100% is perfect and 50% is always forecasting 50%.
| Forecaster, ForecastBench leaderboards (updated 11 November 2026) | Overall Brier Index | 95% confidence interval |
|---|---|---|
| Top tournament entry (Anonymous 24, model-4) | 69.5% | 68.1 to 71.1 |
| Superforecaster median | 68.2% | 66.8 to 69.8 |
| General public median | 63.0% | 61.4 to 64.6 |
| Best model without tools (OpenAI o3, baseline leaderboard) | 61.2% | 59.9 to 62.3 |
| Always forecasting 50% | 50.0% | 50.0 to 50.0 |
Source: ForecastBench leaderboards, tournament and baseline tables, checked 13 November 2026. Tournament entries may use tools, news and ensembles; baseline models run without them. The baseline leaderboard scores the superforecaster median at 67.2% and the public median at 62.1% on its own question set. The leaderboard's own test does not find the superforecasters significantly better than the top entries, and the intervals overlap, so the fair reading is a tie for the best scaffolded systems and a clear gap for models used out of the box.
Signs a forecast will be unreliable
- It is about one item in one place. Accuracy fell sharply from totals to single products in M5.
- A naive forecast does as well. If repeating last period's value scores the same, as on M4's daily series, the model adds nothing.
- The pattern has just changed. Model-based intervals assume "the historical patterns that have been modelled will continue into the forecast period" (FPP3, 9.8).
- It was scored on its own training data. A model that fits the training data well will not necessarily forecast well (FPP3, 5.8).
- It is a one-off event and the assistant has no tools. Out-of-the-box models scored below the public median on ForecastBench's baseline leaderboard.
Sources
Checked 13 November 2026:
- M5 Competition organisers, M5-methods repository (official scores) and competition page, University of Nicosia; results paper: Makridakis, Spiliotis and Assimakopoulos (2022), M5 accuracy competition: Results, findings, and conclusions, International Journal of Forecasting 38(4), 1346-1364
- M4 Competition organisers, M4-methods repository (official scores)
- Karger, E. et al., ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities, arXiv, revised February 2025; Forecasting Research Institute, ForecastBench leaderboards
- Hyndman, R. J. and Athanasopoulos, G., Forecasting: Principles and Practice, 3rd ed.: 5.8, 9.8
Next step
Score your own forecasts against a naive forecast with the forecast score calculator, then read what the uncertainty range means.
Frequently asked questions
- Are forecasts more accurate for totals or for single items?
- For totals. In the M5 Competition the winning entry's WRMSSE was 0.199 for total sales across all stores and 0.884 for single products in single stores, and its lead over the best exponential smoothing benchmark shrank from 53% to 3% between those two levels.
- Do AI assistants forecast future events as well as superforecasters?
- On ForecastBench's tournament leaderboard (updated 11 November 2026) the top AI entry scores a Brier Index of 69.5% and the superforecaster median 68.2%, with overlapping confidence intervals. Models without tools score lower: the best on the baseline leaderboard is 61.2%.