Paste your own numbers. Rows must be in time order for the naive comparison.
Runs entirely in your browser. Nothing you paste is sent anywhere or stored. Commas, tabs or spaces between numbers all work; header lines are skipped.
The formulas
In every formula, e = actual − forecast, the convention in Forecasting: Principles and Practice, section 5.8.
| Metric | Formula | What it tells you | Source |
|---|---|---|---|
| MAE (mean absolute error) | mean(|e|) | Average miss in the data's own units. A method that minimises MAE forecasts the median. | FPP3 5.8 |
| RMSE (root mean squared error) | √mean(e²) | Like MAE but squares each miss first. A method that minimises RMSE forecasts the mean. | FPP3 5.8 |
| MAPE (mean absolute percentage error) | mean(|100e / actual|) | Average miss as a percentage of the actual, so it compares across units. | FPP3 5.8 |
| Mean error (bias) | mean(e) | Positive: forecasts ran low on average. Negative: they ran high. | Same error definition, FPP3 5.8 |
| Relative MAE against naive | MAE(yours) / MAE(naive) | Below 1 beats a forecast that repeats the previous actual, scored on rows 2 onward. | Naive method, FPP3 5.2 |
| Winkler score | (u − l) + (2/α)·miss | Interval width, plus a penalty when the actual lands outside; α is 1 − coverage, so 0.2 for 80%. | Winkler (1972), FPP3 5.9 |
| Brier score | mean((p − o)²) | p is the forecast probability, o is 1 or 0. 0 is perfect; always saying 50% scores 0.25. | Brier (1950); Karger et al. (2024) |
| Base-rate reference | r(1 − r) | The Brier score from always forecasting the observed rate r. Beat it, or the forecasts add nothing. | Follows from the Brier formula |
| Skill score | (S_ref − S) / S_ref | Share of improvement over a reference forecast; 0 means no better. | FPP3 5.9 |
| CRPS (not computed here) | Average quantile score over all probabilities | Scores a whole forecast distribution, in the data's units. | FPP3 5.9 |
The Winkler miss is l − actual when the actual falls below the lower limit l, and actual − u when it lands above the upper limit u. Check the calculator against the book: the row 741.84, 758.88, 744.54, 773.22 at 80% coverage gives a Winkler score of 55.68, the worked example in section 5.9.
When each metric misleads
| Metric | Misleads when | Source |
|---|---|---|
| MAE, RMSE | You compare series in different units: both are on the scale of the data. | FPP3 5.8 |
| RMSE | One or two large misses dominate, because each error is squared before averaging. | Follows from the formula |
| MAPE | Any actual is 0 (undefined) or near 0 (extreme values); the data has no meaningful zero, such as temperature in °C or °F; negative errors get a heavier penalty than positive ones. | FPP3 5.8 |
| sMAPE | Hyndman and Koehler (2006) recommend it not be used: it is unstable near zero and can be negative. | FPP3 5.8 |
| Brier score | Events are rare. If an event happens 5% of the time, always forecasting 5% scores 0.0475 with no skill at all, so compare with the base-rate reference. | Computed: 0.05 × 0.95 |
| Interval coverage | You have few forecasts. With 10 intervals at 80%, 8 inside is expected, but anything from 6 to 10 is within 2 standard deviations (√(10 × 0.8 × 0.2) = 1.26). | Binomial distribution |
| Any metric on fitted values | You score the data the model was fitted on. A model that fits the training data well will not necessarily forecast well. | FPP3 5.8 |
Sources
Checked 13 November 2026:
- Hyndman, R. J. and Athanasopoulos, G., Forecasting: Principles and Practice, 3rd ed., OTexts: 5.2 Some simple forecasting methods, 5.8 Evaluating point forecast accuracy, 5.9 Evaluating distributional forecast accuracy
- Brier, G. W. (1950), Verification of forecasts expressed in terms of probability, Monthly Weather Review 78(1)
- Winkler, R. L. (1972), A decision-theoretic approach to interval estimation, Journal of the American Statistical Association 67(337), 187-191
- Hyndman, R. J. and Koehler, A. B. (2006), Another look at measures of forecast accuracy, International Journal of Forecasting 22(4), 679-688
- Karger, E. et al., ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities, arXiv, revised February 2025
Next step
Score last month's forecasts here, then make it a routine with the forecast review loop.
Frequently asked questions
- What is a good Brier score?
- Lower is better. 0 is perfect and always forecasting 50% scores 0.25. Compare your score with the score you would get by always forecasting the observed base rate, p(1 - p), because rare events make low scores easy.
- Why does the calculator say MAPE is undefined?
- MAPE divides each error by the actual value, so it is infinite or undefined when any actual is 0, as Hyndman and Athanasopoulos note in section 5.8 of Forecasting Principles and Practice. Use MAE or the relative MAE instead.
- What does a relative MAE below 1 mean?
- Your forecasts had a smaller average error than a naive forecast that repeats the previous actual, measured over the same periods. Above 1 means the naive forecast did better.