Prefer a model for a reason that can be checked

Try

Two models predict a bounded synthetic dataset. One is a constant baseline; the other has several adjustable coefficients. The richer model fits the training observations better. Before choosing it, ask whether the improvement survives on unused cases and whether it matters for the intended decision.

Use the same eligible cases, output units and error definition for both models. Comparing one model on easy cases with another on difficult cases does not isolate model quality.

Predict

If one model family contains another as a special case, can its minimum training least-squares error be larger? Predict the answer by considering the feasible set of fits. Then explain why that ordering need not hold for held-out error.

Experiment

Consider this synthetic comparison, with scores defined before inspection.

Model	Parameters	Training squared-error sum	Held-out squared-error sum
Baseline	1	12	10
Richer	4	3	14

The richer model wins on training error and loses on held-out error. If the decision criterion is held-out squared error under the same cases, the baseline is preferred here. This table does not prove that simpler models always win; it demonstrates why fitting and evaluation evidence differ.

Understand

Nested model families guarantee that the best training loss cannot increase when parameters are added, because every old fit remains available. Extra flexibility can also fit noise or incidental features. Prediction depends on the data-generating setting and estimation variability, so training inclusion alone gives no generalization guarantee.

Complexity penalties require a coherent scale. An arbitrary score S=error+λ times parameter count changes if output units rescale the squared-error term but λ remains fixed. Information criteria have specific likelihood-based derivations and assumptions; a convenient teaching penalty should not be mislabeled AIC or BIC. Explain what the chosen criterion measures and why its weighting is appropriate.

Mechanistic and predictive models can serve different purposes. A flexible predictor may interpolate observed conditions well while lacking justified intervention behavior. A mechanistic model may support an intervention analysis only if its structural assumptions are credible. Calling an equation mechanistic does not validate those assumptions, and a favorable predictive score does not establish causality.

Include computational and operational considerations when relevant: fitting cost, latency, required observations, interpretability of parameters and failure modes. These should be measured or explicitly modeled, not added as vague praise for a preferred method. A small accuracy gain may not alter the action selected under the loss table.

If several alternatives were tried, preserve that selection history. The best score among many tuned candidates may be optimistic, especially when the same evaluation data guide repeated choices. A final frozen comparison or a clearly labeled reused practice set keeps the evidence boundary visible.

Apply

Compare one baseline and one richer model for your investigation. Report training and evaluation losses, parameter count, residual patterns, costs and the decision consequence. State one regime where each fails. Select according to the original contract and explain whether the evidence supports a definitive choice or only a provisional preference.

Continue

An informative next experiment can be more valuable than adding another model. Choose observations that distinguish competing mechanisms or parameter directions rather than merely accumulating more of the same data.