Fit an inconsistent system without hiding the error

Try

The inconsistent equations c=1,c=2,c=6 cannot all hold. Choosing c=3 minimizes the sum of squared residuals: the residuals are −2,−1,3 and their sum is zero. Least squares asks for a best approximation, not an exact solution to an impossible system.

Predict

Predict where the fitted vector must lie. For Ax≈b it belongs to col A. Least squares minimizes ||b−Ax||². At an optimum the residual is orthogonal to every column, giving Aᵀ(b−Ax)=0, or AᵀAx=Aᵀb, the normal equations.

Experiment

Fit y=a+bt to observations (0,1),(1,2),(2,2). Write A with rows [1,0],[1,1],[1,2]. Compute AᵀA=[[3,3],[3,5]] and Aᵀb=(5,6), then solve for a=7/6,b=1/2. List fitted values and residuals before inspecting a plot.

Understand

Worked verification: fitted values are 7/6,5/3,13/6 and residuals −1/6,1/3,−1/6. Their sum is zero and their t-weighted sum is zero, so the residual is orthogonal to both columns. Its squared norm is 1/6. The projection theorem therefore certifies optimality among every fitted line, not merely those sampled in a search.

The fitted vector is always unique as the orthogonal projection onto a finite-dimensional column space. Coefficients are unique only with independent columns. Indeed xᵀAᵀAx=||Ax||²>0 for nonzero x precisely when the kernel is zero. Redundant columns allow coefficients to vary along the kernel without changing predictions.

Invalid inference: a small training residual proves a causal relationship or good future prediction. It measures only fit to these observations under this squared-error criterion.

Apply

Supply the design matrix, coefficients, residuals and at least one independent optimality check. State units and model assumptions. A fitted line is different from joining data points, and extrapolation beyond observed inputs needs additional justification.

Transfer: one prediction, many coefficient vectors

Redundant features create a family of equally good coefficients, even when the fitted observations are uniquely determined. Let the two columns of A both be (1,1), and let the observed vector be b=(1,3). For coefficients (u,v), the prediction is (u+v,u+v). Put s=u+v. The squared error is

$$(1-s)^2+(3-s)^2=2(s-2)^2+2.$$

Its minimum is 2, attained exactly when s=2. Thus every vector (t,2−t) minimizes error. The fitted vector is always (2,2), and the residual b−Ax is (−1,1). Its dot product with either column vanishes, independently checking the normal equations. A solver that returns only (2,0) has found one minimizer, not proved unique coefficients.

General proof of the entire family. If x₀ is a least-squares minimizer, every x₀+z with z∈ker A has the same prediction. Conversely, for any minimizer x, orthogonality gives

$$\|b-Ax\|^2=\|b-Ax_0\|^2+\|A(x-x_0)\|^2.$$

The cross term vanishes because b−Ax₀ is orthogonal to im A. Equality of the two minimum errors forces A(x−x₀)=0. Hence the solution set is exactly x₀+ker A. This argument proves uniqueness of the fitted vector without assuming invertibility of AᵀA.

Predict, then inspect the certificate

Changed case Duplicated columnsMinimum coefficient norm

A has two identical columns (1,1), b=(1,3). Which coefficients minimize squared error?

Your prediction Choose a predictionOnly (1,1)Every (t,2−t)Every (t,t) Check reasoning Reset prediction

Two authored cases are practice, not a proof of the general theorem. Explain which hypothesis changed. Responses are temporary and do not change saved progress.

A second decision, not a second fit. To select the smallest Euclidean coefficient norm, minimize t²+(2−t)²=2(t−1)²+2. This selects (1,1), the unique minimizer orthogonal to ker A=span((1,−1)). In general decompose x₀ orthogonally into kernel and kernel-perpendicular parts and remove the kernel part. This changes no fitted observation. Feature units matter: rescaling one feature changes the coefficient norm criterion, even though the possible predictions are unchanged.

Changed task. Replace b by (2,6). Describe all least-squares coefficients, the fitted vector, the residual and the minimum-norm choice. Explain why none of these calculations justifies extrapolation.

Feedback

The optimal sum is 4, giving coefficients (t,4−t), fitted vector (4,4), residual (−2,2), and squared error 8. The minimum-norm choice is (2,2). Orthogonality proves optimal fit only to these two observations under squared loss; it supplies neither causal identification nor a guarantee on a new population.

Continue

Depth: normal equations are useful for derivation, but forming AᵀA can worsen numerical conditioning. QR solves the same projection problem more stably in many cases. Weighted least squares, statistical uncertainty, regularization and singular-value methods extend this foundation in later courses.

Change a data case in the least-squares investigation and compare residuals before returning to the derivation.