Fit parameters and inspect what the fit misses

Try

A synthetic experiment records output y at input x. You propose y=bx with no intercept and observe (1,3) and (2,4). Neither point alone should determine the slope when both are noisy. Least squares chooses the slope minimizing the sum of squared discrepancies, but its result depends on the model form and loss.

Write the objective before calculating: S(b)=(3−b)²+(4−2b)². The decision to omit an intercept is a substantive assumption, not merely a shortcut in the algebra.

Predict

The individual slopes are 3 and 2. Predict whether the least-squares slope is their simple average. The larger input has greater leverage in a through-origin fit, because changing b changes its prediction more strongly.

Experiment

Differentiating gives S′=−2(3−b)−4(4−2b)=10b−22, so b=11/5. The second derivative is 10, proving the unique minimum.

x	Observed y	Fitted bx	Residual y−bx
1	3	11/5	4/5
2	4	22/5	−2/5

The squared-error sum is 4/5. The residuals do not sum to zero, but the input-weighted residual sum does: 1·(4/5)+2·(−2/5)=0. A zero residual sum is a property of an intercept-containing ordinary least-squares fit, not every fit.

Understand

For y≈Xθ, differentiating ∥y−Xθ∥² yields the normal equations XᵀXθ=Xᵀy. Full column rank makes the minimizer unique. If columns are dependent, different parameters can yield the same fitted vector. Numerically, directly forming normal equations can worsen conditioning; QR or singular-value methods are often preferable. The derivation identifies the optimum condition, not a mandate for one implementation.

Ordinary least squares coincides with maximum likelihood for independent normal errors with common variance and the stated mean model. Normality is not needed to define or compute the minimizer. Unequal variances suggest weights; correlated errors require a covariance-aware loss if the probabilistic assumptions justify it. Robust losses change the influence of large residuals and answer a different optimization question.

Residuals should be inspected against input, time, fitted value and relevant groups. Curvature may indicate a missing nonlinear term, but it can also reflect sensor calibration or an omitted variable. A residual pattern suggests hypotheses; it does not identify a unique cause. Test a revised model on observations not used to select that revision.

Parameter bounds can also matter. If a physical rate must be nonnegative, an unconstrained negative estimate signals either noise, misspecification or an inappropriate unconstrained fit. Report the constraint and whether the optimum lies on its boundary rather than silently replacing the estimate after fitting.

Apply

Fit one simple model to a small explicit dataset and show the objective, derivative or solver condition, estimate and residual table. Compare a baseline and identify one plausible misspecification. State which observations were used for fitting and which remain available for validation. Optimizer convergence is evidence about the numerical objective, not about the model’s truth.

Continue

Sensitivity analysis next asks how predictions change when inputs or parameters move. A precise fitted value can still support a fragile decision if the output is highly sensitive in a poorly measured direction.