Solve least squares while exposing lost directions

Try

Let A=diag(2,0) and b=(8,3). The equation Ax=b has no exact solution because the second output coordinate is always zero. Nevertheless the best attainable output is (8,0), produced by every input (4,t). The smallest input length occurs at t=0. This separates three questions: whether the equation is solvable, which output is nearest, and which input among its preimages has minimum norm.

Predict

Predict why replacing the zero singular value by an infinite reciprocal is meaningless. The missing output coordinate is not merely hard to recover; it never depends on the input. A pseudoinverse instead sets the inverse action on that direction to zero. For a tiny positive singular value the situation is different: the inverse exists but can amplify noise greatly. Exact rank deficiency and unstable near-rank deficiency must not be conflated.

Experiment

In singular coordinates, work one component at a time.

Singular value	Output coefficient	Minimum-norm input coefficient	Residual coefficient
2	8	4	0
0	3	0	−3

Change the zero to 0.01 and recompute the exact input: the second coefficient becomes 300. Compare its length with the truncated solution before deciding which reconstruction is useful for noisy measurements.

Understand

If A=UΣV*, define A⁺=VΣ⁺U*, where Σ⁺ transposes the rectangular shape and replaces each positive singular value by its reciprocal while leaving zeros unchanged. The four Moore–Penrose identities follow by multiplying the diagonal factors: AA⁺A=A, A⁺AA⁺=A⁺, and both AA⁺ and A⁺A are self-adjoint. Their products are orthogonal projections onto im A and (ker A)⊥. Hence AA⁺b is the closest attainable output to b. Every least-squares solution is A⁺b+z with z∈ker A. Because A⁺b is orthogonal to that kernel, the squared input norm is ‖A⁺b‖²+‖z‖², proving unique minimum norm at z=0.

Apply

For approximation, retain only the first k singular terms: A_k=Σⱼ≤k σⱼuⱼvⱼ*. Its operator-norm error is σₖ₊₁, and its Frobenius squared error is the sum of the omitted σⱼ². The operator-norm optimality has a short proof. Any rank-at-most-k matrix B kills a nonzero vector in the span of v₁,…,vₖ₊₁ by rank-nullity. Normalize that vector. Then ‖(A−B)x‖=‖Ax‖≥σₖ₊₁, so no B improves on the truncated error. The best approximation need not be unique, especially at repeated singular values. The tail formula alone does not prove Frobenius optimality; the projection argument below supplies the missing lower bound.

For completeness, Frobenius optimality follows from a projection argument. Let W=im B* be the input-side row space of B and P its orthogonal projection, so B(I−P)=0 and dim W≤k. Orthogonality of the input decomposition gives ‖A−B‖²_F≥‖A(I−P)‖²_F=‖A‖²_F−‖AP‖²_F. In a right singular basis, ‖AP‖²_F=Σσⱼ²‖Pvⱼ‖². These weights lie between zero and one and sum to dim W≤k, so the sum is at most the sum of the first k squared singular values. Therefore every B has at least the omitted squared-singular-value error, attained by A_k.

Continue

These constructions reuse one orthonormal geometry for inversion and compression. Their interpretation depends on the task: minimum input norm is a criterion, not evidence that the true input had that norm. Truncation can suppress noise and erase genuine signal simultaneously. Next, condition numbers will quantify worst-case sensitivity and distinguish a mathematical problem’s instability from errors introduced by a chosen numerical algorithm.

Review Linear Algebra I when a prerequisite needs practice. Continue to Separate sensitive problems from inaccurate algorithms.