Test predictions outside the fitting calculation

Try

A model fits last month’s demand closely. The intended decision concerns next month. Evaluating on the same observations used to choose the equation and parameters answers a different question from evaluating future prediction. Write down the prediction time and which information would actually be available then.

Calibration chooses parameters or model structure. Validation assesses specified claims against evidence not used for those choices. In a small teaching exercise, reused observations remain useful practice, but they are no longer untouched evaluation evidence.

Predict

Three prediction errors are −2,4,−2. Predict their mean signed error and mean absolute error. Can a zero signed mean justify calling every prediction accurate?

Experiment

The signed mean is zero, while the mean absolute error is 8/3 and mean squared error is 8.

Error statistic	Calculation	What it emphasizes
Signed mean	(−2+4−2)/3=0	Average direction
Mean absolute error	(2+4+2)/3=8/3	Typical magnitude
Mean squared error	(4+16+4)/3=8	Stronger penalty for large errors

Select the metric according to the decision loss before examining which model wins. If underprediction is much more costly than overprediction, a symmetric error statistic may not represent the operational objective.

Understand

For forecasting, a temporal split usually preserves the information order: earlier observations train the model, a later development period supports tuning and a final later period evaluates the frozen choice. Randomly mixing time points can leak future information or conceal drift. Preprocessing, feature selection and threshold tuning also belong inside the development process; they must not use the final evaluation outcomes.

Transporting a model to another setting requires more than matching variable names. Check measurement definitions, population composition, operating range and causal mechanisms. A cooling coefficient estimated for one object need not transfer to another material or geometry. A queue model fitted at low load may fail near capacity, where neglected dependencies become important.

Inspect performance in relevant subgroups or regimes, but remember that small subgroup samples have large uncertainty. A favorable overall average can hide a serious failure where the decision matters most. Conversely, selecting the worst-looking subgroup after many searches requires care before treating it as a confirmed effect.

When validation fails, preserve the failure. Diagnose using development evidence, revise the model and evaluate again on new data if available. If the same final set guides revision, label it as exposed and avoid presenting the improved score as independent confirmation. This record is essential for an honest investigation, including one whose final conclusion is that the model is not ready for use.

Apply

Write a validation plan with a target population, chronological or otherwise justified split, metric, tolerance and uncertainty statement. Include one out-of-range scenario and explain whether it is a test of intended use or an exploratory stress test. Record what would cause you to reject the model for the original decision.

Continue

Model comparison puts alternatives under the same evaluation contract. A richer equation should earn its complexity through relevant evidence, not merely through a lower training error.