Derive regression inference from projections

Try

Write a fixed-design linear model Y=Xβ+ε, where Y has n entries and X is an n-by-p known matrix of full column rank. The p entries of β are unknown coefficients. Least squares minimizes the squared residual length. Differentiating that quadratic gives XᵀXβ̂=XᵀY, hence β̂=(XᵀX)⁻¹XᵀY. Full rank is what makes the inverse exist. This is first an algebraic fit; distributional claims arrive only after assumptions about ε are supplied.

Predict

Predict which conclusions require normality. If Eε=0 and Cov(ε)=σ²I, β̂ is unbiased with covariance σ²(XᵀX)⁻¹, regardless of whether errors are normal. Exact t and F distributions require more. Under normal errors, orthogonal projections split fitted and residual variation into independent pieces. Increasing the number of fitted directions reduces residual degrees of freedom, so a lower residual sum of squares alone is not sufficient evidence that an expanded model is useful.

Experiment

Compare two nested fixed-design models on the same observations. The restricted column space is contained in the full column space. Use this synthetic table to compute the extra-direction F statistic.

Quantity	Value
Number of observations	12
Restricted coefficients	2
Full coefficients	4
Restricted residual sum of squares	32
Full residual sum of squares	20

The added rank is two and the full residual degrees of freedom are eight. The numerator mean square is(32−20)/2=6; the denominator is 20/8=5/2. Their ratio is 12/5. Identify the null hypothesis as the claim that the true mean lies in the restricted space, not merely that the observed full fit improves by a small amount.

Understand

The hat matrix H=X(XᵀX)⁻¹Xᵀ is the orthogonal projection onto the column space of X. Fitted values are HY and residuals are(I−H)Y. Under the stated linear mean model, the residual mean is zero. If ε∼Nₙ(0,σ²I), the residual sum of squares divided by σ² is a chi-square quadratic form of rank n−p and is independent of the fitted projection, hence of β̂. Thus s²=SSE/(n−p) is unbiased for σ² and supplies exact t pivots for individual coefficients when n>p.

For nested models, Hfull−Hrestricted is itself an orthogonal projection of rank q, the number of added independent directions. It is orthogonal to I−Hfull. Under the restricted null, the corresponding scaled quadratic forms are independent central chi-squares of degrees q and n−pfull. Dividing each by its degrees of freedom and taking their ratio gives the exact F law. This is the mechanism of the extra-sum-of-squares test and of many ANOVA decompositions. Non-nested model spaces do not generally yield this projection difference or this F proof.

Apply

For a one-way fixed-effects ANOVA with g nonempty groups, the full mean space contains a separate mean per group and has rank g. The equal-means null has rank one. The extra-direction test therefore uses numerator degrees g−1 and denominator degrees n−g under independent normal errors with common variance. This connects the familiar group comparison to the same linear-algebra argument, instead of introducing an unrelated memorized formula.

Now diagnose a causal overstatement. Rejecting a coefficient restriction shows incompatibility with that restriction under the regression and sampling assumptions. It does not show that intervening on a predictor changes the response. Confounding, selection and reverse causation require a design or identification argument beyond projection algebra. A strong association and a small p-value do not eliminate those possibilities.

Continue

State whether the design is fixed or inference is conditional on observed predictors, whether errors are independent normal with common variance, and whether the tested spaces are nested. Report the effect and uncertainty as well as a test result. Continue to Regression diagnostics to inspect leverage, residual covariance and heteroskedasticity. These checks investigate assumptions; they cannot certify them merely because a finite plot looks reassuring.