Precogs.chat

How should you review forecasts from a predictive assistant?

By Joel Maldonado · Last reviewed 13 November 2026

Log every forecast with its interval when you make it, score it against the outcome once it is known, and compare it with the four simple benchmarks in Hyndman and Athanasopoulos's Forecasting: Principles and Practice: the mean, naive, seasonal naive and drift methods. Which benchmark is hardest to beat depends on the data: in the book's beer production example the seasonal naive method had the lowest error (MASE 0.94), and in its Google stock example the naive method did (MAE 40.38). Retire any forecast that does not beat the best benchmark on your own outcomes.

The loop in four steps

Step When What to record or compute
1. Log When the forecast is made Date made, date it is for, point forecast, interval and its coverage (80% or 95%), who or what made it (assistant, model, version), one line of reasoning
2. Score When the outcome is known Outcome; error = outcome − forecast; inside or outside the interval; for yes-or-no questions, the Brier score
3. Compare Every review The same scores for the benchmarks below, on the same periods
4. Retire After enough outcomes to judge Drop or rework any forecast that does not beat the best benchmark

The forecast score calculator does steps 2 and 3 for a pasted log: MAE, RMSE, MAPE, bias, interval coverage, the Winkler score, the Brier score and a comparison with the naive forecast.

The benchmarks to beat

Hyndman and Athanasopoulos use four simple methods as benchmarks throughout Forecasting: Principles and Practice, calling them "extremely simple and surprisingly effective" (section 5.2, checked 13 November 2026).

Benchmark Forecast for every future period The book's note
Mean The average of all past values Every future forecast is the same number
Naive The last observed value Works "remarkably well for many economic and financial time series"
Seasonal naive The last observed value from the same season, such as the same month last year Useful for highly seasonal data
Drift The last value plus the average change per period so far, times the number of periods ahead The same as drawing a line from the first to the last observation and extending it

How the benchmarks did in the book's examples

The same four methods win in different places. The book scores them on two real series, forecasting periods the methods had not seen.

Example Method RMSE MAE MAPE MASE
Australian beer production, quarterly, 2008 to 2010 Mean 38.45 34.83 8.28 2.44
Naive 62.69 57.40 14.18 4.01
Seasonal naive 14.31 13.40 3.17 0.94
Drift 64.90 58.88 14.58 4.12
Google closing stock price, January 2016 Mean 118.03 116.95 16.24 16.41
Naive 43.43 40.38 5.67 5.67
Drift 53.07 49.82 6.99 6.99

Source: Forecasting: Principles and Practice, section 5.8, checked 13 November 2026. Lower is better; the best method in each example is in bold. For the seasonal beer series, repeating last year's quarter beat every other benchmark on all four measures. For the stock price, repeating yesterday's close did. A predictive assistant forecasting either series has to beat that number to be worth reading.

Score genuine forecasts only

Score forecasts made before the outcome was known. The book is direct about it: the accuracy of forecasts "can only be determined by considering how well a model performs on new data that were not used when fitting the model" (section 5.8). A back-test that lets the model see the period it is predicting will look better than live use. If you need to test before going live, use time series cross-validation: forecast each period using only the data before it, then average the errors (section 5.10).

How many outcomes before you judge

Interval calibration needs many outcomes: after 10 forecasts at 80%, anything from 6 to 10 inside is within 2 standard deviations of the expected 8, so even a perfect log looks noisy. The uncertainty guide explains the arithmetic. For yes-or-no forecasts, compare the Brier score with the score from always forecasting the base rate, which the calculator reports.

Sources

Checked 13 November 2026:

Next step

Paste last quarter's forecasts and outcomes into the forecast score calculator and read the relative MAE first: below 1 means the forecasts beat the naive one.

Frequently asked questions

What is a naive forecast?
A forecast that repeats the last observed value. Hyndman and Athanasopoulos note it works remarkably well for many economic and financial time series, which makes it the first benchmark any model should beat.
Which benchmark should a forecast beat?
The best of the mean, naive, seasonal naive and drift methods on your own data. For seasonal data that is often seasonal naive; in Forecasting Principles and Practice's beer example it had a MASE of 0.94 against 2.44 to 4.12 for the other three.