Try
Keep the four worked observations (13,21), (11,23), (9,17), (7,19). Their mean is μ=(10,20), and their first loading is v=(1,1)/√2. Suppose storage permits one score per observation instead of two centered coordinates. What can we reconstruct, and what information did we discard?
For the first observation, the centered row is (3,1) and its score is 2√2. Multiplying score by loading gives (2,2). This is still a centered reconstruction. Add the mean to obtain the original-unit approximation (12,22). Its residual is (1,−1), with squared length 2. Reporting (2,2) as an approximation to (13,21) would confuse coordinate systems.
Return to centering and scores for the live data table. This lesson proves what that table’s reconstruction error measures. It does not infer a response variable or fit a supervised regression.
Predict
The covariance eigenvalues are 32/3 and 8/3. Predict whether the discarded eigenvalue 8/3 equals the total squared reconstruction error. There are four observations and covariance used divisor 3. A variance is a normalized sum, so the total discarded sum is 3(8/3)=8.
Predict also whether keeping both component scores gives exact reconstruction. Two orthonormal coordinates represent the whole plane; keeping both is a change of basis, not compression. Perfect reconstruction in that case does not establish that either coordinate can be removed harmlessly.
Experiment
Use the worked-data preset in the PCA workbench, with covariance mode. Before inspecting its restored table, calculate all four reconstructed rows and residuals. Then change one observation and explain why the mean and all reconstructed rows may change, even though only one row was edited.
| Original | Restored rank-one reconstruction | Residual | Squared residual |
|---|
| (13,21) | (12,22) | (1,−1) | 2 |
|---|
| (11,23) | (12,22) | (−1,1) | 2 |
|---|
| (9,17) | (8,18) | (1,−1) | 2 |
|---|
| (7,19) | (8,18) | (−1,1) | 2 |
|---|
Total centered squared length is 40. The retained component accounts for 32 and residuals for 8, so the explained-variance fraction is 32/40=4/5. This is a fraction of training variation in the chosen coordinates. It is not prediction accuracy, the probability a component is correct, or the fraction of a scientific phenomenon explained causally.
The workbench draws perpendicular projections in its analyzed coordinate system. In correlation mode, that is the standardized system, while its reconstruction table restores original units. Keep those two displays distinct when comparing losses.
Understand
A direct error proof
Write a centered row as a column x for this calculation. For a unit vector v, its projection is xhat=(vᵀx)v. The residual x−xhat is perpendicular to v because vᵀ(x−xhat)=vᵀx−vᵀx=0. The Pythagorean identity gives
||x−xhat||² = ||x||² − (vᵀx)².
Sum over the n rows of centered matrix X. The squared Frobenius norm, written ||X||F², means the sum of squares of all entries. Thus
||X−Xvvᵀ||F² = ||X||F² − ||Xv||².
The first term is fixed by the data. Minimizing residual loss is therefore exactly maximizing score variation. The preceding lesson proved that a top eigenvector of C=XᵀX/(n−1) does this. Since trace(C), the sum of its diagonal entries, equals both total sample variance and Σ λ_j, the minimum one-component loss is
(n−1)(λ₂+⋯+λ_d).
This establishes a training least-squares optimum under a specific Euclidean metric. It makes no probabilistic assumptions about the rows.
Why the fitted affine line can pass through the mean
An arbitrary affine line can be written a+span(v). Its squared residual sum is Σ||(I−vvᵀ)(z_i−a)||² for original observations z_i. Write z_i−a=(z_i−μ)+(μ−a). The cross terms sum to zero because Σ(z_i−μ)=0. The loss is the centered projection loss plus n||(I−vvᵀ)(μ−a)||². The latter term is nonnegative and vanishes when the line passes through μ. Thus using the mean is part of the affine optimum, not an aesthetic choice. Moving a along the same line does not change that line.
More than one retained component
For an integer k with 1≤k≤d and k orthonormal loadings arranged as columns of V, projection is XVVᵀ. The same orthogonal decomposition gives loss ||X||F²−||XV||F². In a covariance eigenbasis q_j, put w_j=||Vᵀq_j||². Orthogonal projection implies 0≤w_j≤1, and summing squared coordinates of the k unit columns gives Σw_j=k. Therefore
Σ λ_j w_j ≤ λ₁+⋯+λ_k.
For clarity, subtract the left side from the right: the deficit is Σ over j≤k of λ_j(1−w_j) minus Σ over j>k of λ_j w_j. Bound the first weights below by λ_k and the second above by λ_k; the remaining difference is zero because Σ over j≤k of (1−w_j)=Σ over j>k of w_j. Choosing the first k eigenvectors attains the bound. The optimal loss is (n−1)Σ over j>k of λ_j. For k=d it is zero. The SVD and low-rank approximation lesson supplies the related general matrix theorem; the argument here connects it to centered statistical data.
Freeze preprocessing for validation
A learned transform includes its training mean and, if used, training standard deviations. Evaluate new observations using those stored quantities and loadings. Re-centering or refitting on the validation observations changes the transform being evaluated. A later revised model may be useful, but must be disclosed and evaluated on another appropriate record.
Apply
Train covariance PCA on (−2,0), (0,0), (2,0). Freeze the transform. Then inspect new observations (0,3), (0,−3). Compute the training fraction and validation loss; compare with a colleague who chooses a new axis after seeing the new observations.
Check training and new-data evidence separately
The training mean is zero and covariance is diag(4,0). The x-axis captures all nonzero training variation, so fraction=1 and training loss=0. The frozen projection sends both new points to zero. Their squared residuals are 9 and 9, giving validation loss 18. Fitting the new points alone instead chooses the y-axis and yields zero error on those same points. That is not the validation performance of the original transform. Two synthetic observations do not estimate a reliable population performance distribution; they expose the logical distinction between fit and evaluation.
Continue
In scaling and degenerate directions, decide which metric the reconstruction should optimize and what to report when the data do not identify a unique direction. Retain both a direct residual calculation and an eigenvalue-based check. Agreement is numerical evidence for your calculation, not verification of the collection process or the scientific interpretation.
