Fit residuals without unnecessarily squaring conditioning

Try

Fit a straight line to three points whose measurements are not exactly collinear. The equations Ax=b may be inconsistent, but the squared residual ‖Ax−b‖² still has a meaningful minimum. For the simpler constant fit to values 2,4,9, differentiation gives c=5, with residuals −3,−1,4. Their sum is zero, expressing orthogonality to the single all-ones design column.

Predict

Predict whether a tiny residual guarantees accurate coefficients. Nearly dependent columns can produce very different coefficients with nearly identical fitted outputs. The approximation problem may be well determined in output space but sensitive in coefficient space. Adding more digits to a normal-equation solve does not remove this data geometry. Inspect rank and singular values before interpreting individual coefficients as reliable measurements.

Experiment

Use the constant example as an exact projection table.

Observation	Fitted value	Residual	Squared residual
2	5	−3	9
4	5	−1	1
9	5	4	16

The sum is 26. Perturb the fitted constant by d and expand the squared error; the cross term vanishes and the increase is 3d². This proves the minimum directly.

Understand

For a full-column-rank m-by-n matrix A with m≥n, take thin QR: A=QR, Q*Q=I and R invertible upper triangular. Split b into QQ*b and its orthogonal complement. Then ‖Ax−b‖²=‖Rx−Q*b‖²+‖(I−QQ*)b‖². The second term is unavoidable, and the first is minimized uniquely by Rx=Q*b. Back substitution solves the triangular system. The same minimizer satisfies A*(Ax−b)=0, the normal equations. These exact descriptions agree algebraically; their floating-point implementations can behave differently.

Apply

Forming A*A squares the spectral condition number because its eigenvalues are the squared singular values of A. QR avoids this explicit squaring. Householder reflectors preserve Euclidean norms and can eliminate column tails with stable sign choices; modified Gram–Schmidt can be useful but may lose orthogonality for difficult inputs. A real reflector H=I−2vvᵀ/(vᵀv) is orthogonal because the rank-one normalized projector is idempotent. Choose v from x and a signed coordinate vector to avoid subtracting nearly equal leading magnitudes. If rank is deficient, the output projection remains unique but coefficient vectors are not; SVD exposes the null space and supplies a minimum-norm solution.

For a full-rank design, orthogonality of the residual is a necessary and sufficient minimum condition: if A*r=0, replacing x by x+d changes the squared residual by ‖Ad‖² because the cross term vanishes. Full column rank makes that increase positive for d≠0, proving uniqueness. This gives a second proof independent of the QR factorization and a useful implementation check. In floating arithmetic, scale the orthogonality residual before interpreting its magnitude.

Continue

An implementation report should show residual norm, an orthogonality check A*r, rank evidence and coefficient sensitivity. Data weighting changes the inner product and should be stated. A small residual on training observations is not evidence of predictive quality on new data. Written QR design is not a substitute for implemented tests with independent references and nearly dependent columns. Next, quadrature applies approximation to integrals, where exactness on a polynomial class and a smoothness-based error bound provide complementary evidence.

Review calculus and linear algebra when a prerequisite needs practice. Continue to Integrate local approximations with an error model.