Center data and construct principal-component scores

Try

Four synthetic observations of two quantitative variables are (13,21), (11,23), (9,17) and (7,19). These values were constructed for exact arithmetic; they are not a random sample or a licensed observational dataset. We want a new pair of perpendicular coordinate axes that describes their spread. We do not want a causal explanation or a predicted outcome variable.

Draw the four points and mark their average (10,20). A direction through the origin would be dominated by how far this average lies from (0,0). A direction through the average instead describes variation around a typical observation. Centering means subtracting the average of each column from every row. Our centered points are (3,1), (1,3), (−1,−3), (−3,−1).

A loading is a unit direction vector. A score is an observation’s centered coordinate along that direction, calculated by a dot product. Scores are new coordinates of existing observations, not new observations. Review complex and real inner products if the dot-product interpretation needs a refresher; everything here is real.

Predict

Predict which line captures more centered spread: the horizontal line, the line of slope 1, or the line of slope −1. Calculate the score of (3,1) along v₁=(1,1)/√2: it is 4/√2=2√2. Along v₂=(1,−1)/√2 it is 2/√2=√2.

Now negate v₁. Its score changes sign too. Predict whether score times loading changes: (−2√2)(−v₁)=2√2 v₁. A program that prints the opposite sign has not necessarily found a different component. A geometric line has no preferred arrow direction.

Experiment

Live experiment: data, directions and reconstruction

Predict what doubling the first measurement will change. Compare covariance and correlation PCA, then explain the difference. These synthetic observations and controls are temporary; save your reasoning in the lesson response.

Data pairs: variable 1, variable 213,21 11,23 9,17 7,19 PCA scalingCovariance: original measurement unitsCorrelation: sample-standardized variables

Preset buttons replace the input with fixed examples; they do not transform your edited dataset. The scaling choice stays selected until Reset.

Worked dataWorked data: double variable 1Equal-variance directionsConstant dataReset PCA experiment

4 observations. Mean: (10, 20). Sample standard deviations: (2.58199, 2.58199). Covariance denominator: n−1 = 3. Displayed values are rounded; calculations use floating-point arithmetic. Tiny nonzero residuals may reflect roundoff.

Analyzed covariance matrix (rows)

(6.66667, 4); (4, 6.66667)

First and second loading directions

(0.707107, 0.707107); (-0.707107, 0.707107)

Variances along these directions

(10.6667, 2.66667)

First-component explained variation

80%

Rank-one residual sum of squares

8 in centered original coordinates

The plot shows centered original-unit coordinates. Circles are observations; squares are rank-one reconstructions. Dashed segments show discarded components. Both axes share the range −3.45 to 3.45 and equal screen scales. The range changes with the data; compare the numerical errors across presets.

Score 1 is the dot product of each analyzed row with loading 1. Keeping only this score discards score 2. Reconstruction restores the original means and, in correlation mode, sample standard deviations. Reversing a loading and its score together leaves reconstruction unchanged.

Input → centered → analyzed → scores → rank-one reconstruction

Row	Original pair	Centered pair	Analyzed pair	Scores (1, 2)	Reconstruction in original units
1	(13, 21)	(3, 1)	(3, 1)	(2.82843, -1.41421)	(12, 22)
2	(11, 23)	(1, 3)	(1, 3)	(2.82843, 1.41421)	(12, 22)
3	(9, 17)	(-1, -3)	(-1, -3)	(-2.82843, -1.41421)	(8, 18)
4	(7, 19)	(-3, -1)	(-3, -1)	(-2.82843, 1.41421)	(8, 18)

PCA summarizes these observations. It does not establish causation, validate a sampling model, or guarantee useful prediction. Correlation scaling changes the error metric; high explained variation alone does not decide which scaling answers your question.

The lab starts with the four worked observations and covariance mode. Results update when you edit valid data; there is no separate calculate or reveal step. Before each edit, write a prediction. Compare the mean, centered rows, covariance, two loading vectors and scores with the hand calculation below.

The plot uses centered analyzed coordinates: centered original units in covariance mode, standardized units in correlation mode. The reconstruction table returns to original units and restores the mean. Do not read a standardized axis label as an original measurement. Preset buttons replace the whole input with fixed examples; “Worked data: double variable 1” does not transform an arbitrary edited sample. Reset restores the original worked data and covariance mode.

First add 10 to every first coordinate by editing all four rows. The mean changes but centered scores and covariance do not. Then reset and compare the two modes. Their first lines happen to agree for these equal-variance columns; the third lesson supplies a unit change where that coincidence fails. Invalid data should produce an explanation, not a silently retained previous fit. The bounded tool accepts 2–100 finite pairs with coordinates between −1,000,000 and 1,000,000; numerical refusal is not a proof that PCA is mathematically impossible.

The lab can choose the negative of our second loading; its entire second score column then changes sign. Check the products or projectors instead of demanding matching printed signs.

Understand

Let n≥2 be the number of observations and d the number of variables. Let X be the n-by-d matrix of centered rows. The sample covariance is C=XᵀX/(n−1). Its diagonal entries are sample variances; its off-diagonal entries are sample covariances. Our sum of centered outer products is [[20,12],[12,20]], so

C = [[20/3,4],[4,20/3]].

We deliberately use the sample divisor n−1. The archived MIT 18.650 empirical-covariance convention uses divisor n. With the same centered matrix, the latter covariance is (n−1)/n times ours. Therefore our eigenvalues are n/(n−1) times its eigenvalues, while eigenspaces, consistent scores and positive-total-variance fractions agree. Always state the convention; the word “covariance” alone does not choose a divisor.

Here Cv₁=(32/3)v₁ and Cv₂=(8/3)v₂. Thus the first score variance is 32/3 and the second is 8/3. Verify directly: the first scores have squared sum 32, divided by 3. The two score columns are uncorrelated in this sample because their crossproduct is zero. Uncorrelated coordinates need not be independent random variables.

Why an eigenvector maximizes sample variance

For any unit vector v, the score vector is Xv. Its mean is zero because every column of X sums to zero. Its sample variance is

||Xv||²/(n−1) = vᵀCv.

The squared Euclidean norm sums squared coordinates. Since uᵀCu=||Xu||²/(n−1)≥0, C is symmetric positive semidefinite. By the self-adjoint spectral theorem and Rayleigh quotient, it has an orthonormal eigenbasis q₁,…,q_d with nonnegative eigenvalues λ₁≥…≥λ_d. Write v=Σ a_j q_j; unit length means Σ a_j²=1. Then vᵀCv=Σ λ_j a_j²≤λ₁. Equality holds exactly when v belongs to the top eigenspace. This proves the first principal-component rule without inferring it from a picture. The second direction maximizes variance among directions perpendicular to the first; the same expansion proves this successive rule.

The theorem describes the supplied matrix. It does not establish representative sampling, normality, causal factors or stable population directions. Exact ties and near ties will matter later.

Apply

Use the changed observations (0,1), (4,1), (2,7). Find the mean, centered covariance and both score columns. Decide whether zero off-diagonal covariance can be concluded from a drawing alone, then verify it with centered crossproducts.

Check the changed calculation

The mean is (2,3). Centered rows are (−2,−2), (2,−2), (0,4). Their crossproducts give C=diag(4,12), using divisor 2. Choose first loading (0,1) and second (1,0). Score rows are (−2,−2), (−2,2), (4,0). Each column sums to zero; variances are 12 and 4. A sign-reversed loading with its matching reversed scores is equally correct. No estimate of a population direction’s uncertainty follows just from this calculation.

Continue

Keep the centered matrix and scores in your work record. In reconstruction and variance, use them to rebuild approximate observations and prove exactly how much training variation is lost. Public feedback supports correction; independent understanding requires a changed case and a review of the argument, not merely matching one plot.