Try
A sensor exports the same physical quantity in a new unit, doubling every number in its first column. No new observations were collected. Should a statistical summary choose a different direction?
There is no answer until we choose the geometry. Covariance PCA minimizes squared error in the numerical units supplied. Doubling a column multiplies its contribution to that squared error by four. Correlation PCA first divides each centered column by its own sample standard deviation, putting both columns on a common variability scale. That chooses a different question; standardization is not an automatic correction that makes every analysis scientifically appropriate.
Use the worked observations from centering and scores. Their original sample covariance is [[20/3,4],[4,20/3]], with first line of slope 1 and explained fraction 4/5.
Predict
Before calculation, predict which original-coordinate direction will receive more weight after doubling variable 1. The new covariance is [[80/3,8],[8,20/3]], not twice the original covariance. Its eigenvalues are
50/3 + 2√41, 50/3 − 2√41.
The first explained fraction is 1/2+3√41/50≈0.884187454246. A leading direction is proportional to ((5+√41)/4,1), closer to the first coordinate axis. Normalize this vector before using it for dot-product scores.
Now predict the standardized correlation matrix. The sample SDs are 2√(20/3) and √(20/3), giving [[1,3/5],[3/5,1]]. Its eigenvalues are 8/5 and 2/5, so its first fraction stays 4/5. Unchanged correlation describes invariance to this positive unit rescaling, not invariance of raw squared distances.
Experiment
Open the shared PCA lab. Reset, then use “Worked data: double variable 1.” This replaces the input with (26,21),(22,23),(18,17),(14,19). Compare covariance and correlation modes. The preset is a fixed dataset, not a command that doubles arbitrary learner edits. Results update as settings change.
In covariance mode the plotted coordinates are centered original units; in correlation mode they are standardized. The displayed original-unit reconstructed rows have had the scaling and mean restored. Therefore the numeric residual sum of squares in correlation mode belongs to standardized coordinates. It cannot be compared directly with raw-coordinate squared loss as though both used the same unit.
Next choose “Equal-variance directions,” which supplies the four points (1,0),(−1,0),(0,1),(0,−1). Predict why the code can draw a line even when the mathematics selects no unique one. Finally try “Constant data.” Explain the difference between a meaningful zero reconstruction error and an undefined explained fraction. Presets retain the selected scaling mode, so a constant-data correlation refusal is expected; switch to covariance to inspect the zero matrix.
| Data mechanism | Covariance conclusion | Correlation conclusion |
|---|
| One constant column; another varies | Rank 1; principal line defined; fraction 1 | Undefined division by zero SD |
|---|
| Every row identical | Zero matrix; loss 0; fraction undefined; no preferred axis | Both SDs zero |
|---|
| Nonzero scalar covariance | Every axis optimal; fraction 1/2 in two dimensions | Check SDs and correlation matrix separately |
|---|
| Distinct positive eigenvalues | Principal line unique; loading sign arbitrary | May identify a different original-unit line |
|---|
Understand
The metric behind standardization
Write a centered observation as a column x, and let D be the diagonal matrix of its two positive training sample SDs. The standardized vector is z=D⁻¹x, with covariance D⁻¹CD⁻¹. If v is its unit first loading, zhat=vvᵀz is the standardized orthogonal projection. Restore original centered coordinates by xhat=Dzhat, then restore the training mean.
The minimized residual is
||z−zhat||² = (x−xhat)ᵀD⁻²(x−xhat).
This is a weighted original-coordinate squared error. In general x−xhat is not Euclidean-perpendicular to the original-coordinate axis Dv. For example D=diag(1,2), v=(1,1)/√2 and standardized residual (1,−1) give restored residual (1,−2); its dot product with Dv=(1,2)/√2 is −3/√2. Orthogonality holds in standardized coordinates, which is exactly where the workbench draws correlation-mode projection segments.
For the doubled-first-variable worked case, the standardized first loading is (1,1)/√2. Its restored line is proportional to (2,1), different from the covariance-PCA line ((5+√41)/4,1). Both calculations can be correct because their objectives differ. A predictor’s scientific importance is not automatically its variance, and equally weighting standardized columns is a modeling choice.
Constant columns and zero total variation
Sample variance is a sum of nonnegative squared deviations. It is zero exactly when every value in that column equals its mean. Dividing by its SD is then undefined. A tool must refuse correlation PCA or explicitly offer a separately defined variable-removal procedure; replacing the SD by 1 silently changes the method.
If all centered rows are zero, C=0, every unit vector is an eigenvector and all reconstructions equal the mean. Loss is zero, but λ₁/(λ₁+λ₂)=0/0 has no value. “100% explained” would hide that there was no variation to explain.
If instead at least one centered row is nonzero and all lie on span(u), write them as a_i u with u nonzero. Then C=(Σa_i²/(n−1))uuᵀ has rank 1. The principal line is span(u), its sign is arbitrary, and its training fraction is 1. This proves only a property of the given rows, not a noise-free population or a causal latent variable.
Repeated and nearly repeated eigenvalues
For the equal-axis example, C=(2/3)I. Every unit v has vᵀCv=2/3, so all one-dimensional projections are optimal. Each loses total squared length 2 and captures fraction 1/2. Choosing the x-axis or y-axis gives different reconstructed points with exactly the same objective value.
A numerical program may flag eigenvalues as tied within a tolerance. That warning indicates unreliable axis identification at the working precision; it is not proof of exact equality. Even in exact mathematics, replacing cI by diag(c+ε,c) or diag(c,c+ε) for arbitrarily small ε>0 selects perpendicular leading axes. Thus tiny perturbations can change a selected axis substantially near a tie. Compare subspaces, projectors, eigenvalue gaps and reconstruction objectives instead of relying on a printed arrow’s sign or apparent stability.
Apply
Use rows (2,0),(2,2),(2,4). Determine every covariance-mode output you can justify, then decide whether correlation mode is defined. Explain why this case differs from the constant-data preset.
Check the distinction
The mean is (2,2); centered rows are (0,−2),(0,0),(0,2). C=diag(0,4). The first line is vertical, first scores can be −2,0,2, and reconstruction restores every original row. Positive total variation gives fraction 1 and loss 0. Correlation mode is undefined because the first SD is zero. With all rows identical, both variances and total variation would be zero, so no preferred first line or defined explained fraction would remain.
Continue
Complete the mixed PCA assessment using a written centering/scaling contract, independent reconstruction checks and a frozen validation transformation. Return to SVD and low-rank approximation for the general operator viewpoint, or regression diagnostics to distinguish explanatory data reduction from an outcome model.
These lessons implement small-sample PCA mechanisms and their boundaries. They do not supply sparse/high-dimensional PCA theory, model-based uncertainty for estimated eigenvectors or a completed real-data investigation. Those claims require their own assumptions and evidence; a high training fraction is not a substitute.
