A forecast looks excellent but may just repeat yesterday's value — how do you catch that?
A daily demand forecast reports MAPE 4.1% on the test window, and the chart looks near-perfect.
Constraints:
daily.csvhasdateandunits, one row per day for three years.preds.csvhasdateandyhatfor the last 90 days.- Some SKUs sell 0-2 units a day.
import pandas as pd
y = pd.read_csv("daily.csv", parse_dates=["date"]).set_index("date")["units"]
p = pd.read_csv("preds.csv", parse_dates=["date"]).set_index("date")["yhat"]
a = y.loc[p.index]
mape = ((a - p).abs() / a).mean() * 100
print(f"MAPE {mape:.1f}%") # 4.1%Show that this forecast is a lagged copy, and fix the evaluation.
Cross-correlate forecast against actuals — a peak at lag one means it copies yesterday. Score it against naive and seasonal-naive baselines with MASE, because on a near random walk MAPE looks small; beating seasonal naive is the proof.
- ✗Judging a forecast without ever fitting a naive baseline to compare it against
- ✗Trusting a small MAPE on a slow-moving series that a lag-one copy also achieves
- ✗Reporting MAPE on a series with near-zero actuals, where it explodes
- →Why does MAPE blow up as actuals approach zero, and what replaces it?
- →Which baseline do you pick for a series with strong weekly seasonality?
The metric proves nothing: on a slow-moving series a copy of yesterday's value also scores a tiny MAPE. Worse, for SKUs selling 0-2 units the denominator a hits zero and MAPE diverges.
The diagnostic is a cross-correlation of forecast against actuals — a lagged copy peaks at lag 1 rather than lag 0. The evaluation is fixed by scoring against the naive and seasonal-naive baselines with MASE.
import numpy as np
import pandas as pd
a = y.loc[p.index]
lags = {k: a.corr(p.shift(-k)) for k in range(-3, 4)}
print(max(lags, key=lags.get)) # 1 → the forecast repeats yesterday's actual
naive = a.shift(1) # naive
snaive = a.shift(7) # seasonal naive, weekly cycle
scale = (a.diff(7).abs()).mean() # MASE denominator
def mase(pred):
return (a - pred).abs().mean() / scale
print(mase(p), mase(naive), mase(snaive))If mase(p) is not below mase(snaive), the model learned nothing beyond a seasonal repeat and must not ship.
