Separate sensitive problems from inaccurate algorithms

Try

For A=diag(1,0.02), the equation Ax=(1,0) has solution x=(1,0). Change the right-hand side by δb=(0,0.02). The new solution is (1,1): a two-percent relative change in data caused a one-hundred-percent relative change in solution. The calculation is exact; no rounding error is required. This is sensitivity of the mathematical problem, not proof that the solver malfunctioned.

Predict

Predict whether every perturbation is amplified by the same factor. A change δb=(0.02,0) produces δx=(0.02,0), with no amplification. The vulnerable direction is the left singular vector corresponding to the smallest singular value. A condition number describes a worst-case relative bound, not an expected error or a guarantee that all data are equally dangerous. The original solution direction also matters for relative sensitivity.

Experiment

Compare these two directions using the same perturbation size.

Perturbation	Solution change	Perturbation length	Solution-change length
(0.02,0)	(0.02,0)	0.02	0.02
(0,0.02)	(0,1)	0.02	1

Replace 0.02 in A by 0.002 and predict the new worst amplification. Explain why setting it exactly to zero changes the problem from sensitive inversion to noninvertibility.

Understand

For invertible A, define κ₂(A)=‖A‖₂‖A⁻¹‖₂=σmax/σmin. If Ax=b and A(x+δx)=b+δb with A fixed, then δx=A⁻¹δb. Combining ‖δx‖≤‖A⁻¹‖‖δb‖ with ‖b‖≤‖A‖‖x‖ gives the relative bound ‖δx‖/‖x‖≤κ₂(A)‖δb‖/‖b‖ for b≠0. This theorem assumes the matrix itself is unchanged. If A also changes by E and ‖A⁻¹E‖<1, the inverse of I+A⁻¹E is bounded by a convergent geometric series, producing a denominator 1−‖A⁻¹E‖ in perturbation bounds. Omitting that denominator can falsely promise stability near singularity.

Apply

A residual measures how well a computed vector solves the equation: r=b−A x̂. The actual error is A⁻¹r, so a small residual gives a useful forward-error bound only in combination with conditioning. For full-column-rank least squares, A*A has eigenvalues σⱼ² and therefore κ₂(A*A)=κ₂(A)². Forming normal equations can make numerical sensitivity much worse than working with QR or SVD. This does not mean QR changes the underlying problem’s condition number; it avoids adding a particular algebraic source of numerical difficulty. Backward stability means a computed result solves a nearby problem, while forward accuracy also depends on how that nearby problem’s solution changes.

For a rectangular map with full column rank, the ratio of largest to smallest positive singular values measures sensitivity on the identifiable input space. If the map has a nontrivial kernel, no full-space inverse condition number is finite; a restricted ratio must be explicitly labeled as restricted. Otherwise a rank-deficient map could appear “well-conditioned” after silently discarding exactly the directions that cannot be recovered.

Continue

The unit ends with a choice of decomposition guided by the question. Similarity and Jordan form describe exact operator structure; orthogonal spectral decompositions reveal self-adjoint energy; SVD handles general maps, rank and least squares. None alone supplies a noise model or a universally correct tolerance. Further numerical linear algebra includes stable QR/SVD algorithms, pivoting, iterative methods and verified error estimates; those are not established merely by the existence proofs in this unit. Independent proof review and cumulative assessment remain separate evidence.

Review Linear Algebra I when a prerequisite needs practice. Return to the operators unit to select a targeted retry.