Lesson 1 of 3

Fit coefficients and audit residuals

Construct a numeric design and check a rank-aware least-squares fit against its defining equations.

Loading your saved responses…

Try

Eight synthetic records describe a response y in minutes using two dimensionless predictors x and z. Each row represents a separate hypothetical unit; repeated predictor values are allowed. These numbers were engineered for transparent arithmetic, not collected as independent Gaussian observations. We will first calculate a fit, then state separately what a probability model would allow us to infer.

Our proposed mean equation is β₀+β₁x+β₂z. The intercept β₀ is the prediction when both predictors are zero. The design matrix X has one row (1,x,z) per observation; that first column of ones is part of the model, not a response measurement.

Row	x	z	y	Fitted y	Residual y−fitted
1	−1	−1	8.5	9	−0.5
2	−1	−1	9.5	9	0.5
3	−1	1	6.5	7	−0.5
4	−1	1	7.5	7	0.5
5	1	−1	11.5	13	−1.5
6	1	−1	14.5	13	1.5
7	1	1	9.5	11	−1.5
8	1	1	12.5	11	1.5

Cover the last two columns and try the equation 10+2x−z. Compute all eight predictions. A residual is a signed vertical response discrepancy for a particular row; it is not an unobserved population error that we know exactly.

Predict

Predict whether omitting the intercept changes the problem. Then predict what happens if a fourth column w=2x is added. More columns need not mean more identifiable coefficients: increasing βₓ by 2a while decreasing βw by a leaves βₓx+βww unchanged.

Least squares chooses coefficients minimizing the sum of squared residuals, abbreviated SSE. A plot that looks plausible is not the certificate. We can check that every predictor column is perpendicular to the residual vector: Xᵀr=0. The column of ones makes this include Σrᵢ=0.

Experiment

Live experiment: fit, diagnose and predict

The worked data are synthetic: two dimensionless predictors and a response in minutes. An intercept is included automatically. Predict the coefficient signs and residual pattern before comparing the calculations. Controls are temporary; record evidence in your lesson response.

Data rows: predictors, then response

Use one row per observation and the same 1–3 predictors in each row. Do not add an intercept column. Preset buttons replace the data with fixed examples; they preserve any separately frozen model.

8 observations, 3 fitted coefficients including the intercept. Coefficients in order (intercept, predictors): (10, 2, -1). Calculations use floating-point arithmetic; values below are rounded.

Residual sum of squares (SSE): 10
Residual degrees of freedom: 5
Estimated error variance s²: 2
Scaled QR conditioning diagnostic: 1

Ordinary least squares with an automatic intercept. The covariance formula requires a correctly specified linear conditional mean, uncorrelated homoskedastic errors and positive finite variance. Exact reference 95% t intervals additionally require independent Gaussian errors. A new-response interval additionally assumes an independent new error with that same variance. Fitting does not verify these assumptions. Floating-point calculations and t quantiles are numerical approximations, not certified enclosures.

Residuals are observed minus fitted. Standardized residuals use s sqrt(1 − leverage); they are internally standardized, not leave-one-out studentized residuals.
Condition screening uses the infinity-norm product ||R||∞ ||R⁻¹||∞ of the centered, column-scaled, pivoted QR triangular factor. It is not the condition number of the original design or evidence that statistical assumptions hold.

Estimated coefficient covariance under the stated error model
Coefficient	Intercept	Predictor 1	Predictor 2
Intercept	0.25	1.387779e-17	-1.387779e-17
Predictor 1	1.387779e-17	0.25	-1.387779e-17
Predictor 2	-1.387779e-17	-1.387779e-17	0.25

Residual plot horizontal axis

Inspect residuals against fitted values and each predictor before interpreting intervals. Zero is the horizontal reference. The vertical axis uses response units; the horizontal axis uses fitted response units; horizontal range 6.4 to 13.6, vertical range −1.8 to 1.8. Ranges adapt to the data, so compare numeric errors across presets. A pattern suggests an investigation; its absence does not establish the assumptions.

Original data and row-level calculations
Row	Predictors	Observed	Fitted	Residual	Leverage	Internally standardized residual
1	(-1, -1)	8.5	9	-0.5	0.375	-0.4472136
2	(-1, -1)	9.5	9	0.5	0.375	0.4472136
3	(-1, 1)	6.5	7	-0.5	0.375	-0.4472136
4	(-1, 1)	7.5	7	0.5	0.375	0.4472136
5	(1, -1)	11.5	13	-1.5	0.375	-1.341641
6	(1, -1)	14.5	13	1.5	0.375	1.341641
7	(1, 1)	9.5	11	-1.5	0.375	-1.341641
8	(1, 1)	12.5	11	1.5	0.375	1.341641

Internally standardized residuals divide by s√(1−hᵢᵢ). They are unavailable when the estimated scale is zero/unresolved or leverage is numerically one. They are not independent standard normal observations, and a small residual does not imply low influence.

Prediction row: predictor values only

Current fitted model: two different uncertainty targets

Estimated response: 11.5. Mean-response standard error: 0.6123724. New-response standard error: 1.541104.

Model-conditional 95% mean-response interval: (9.925847, 13.07415). Model-conditional 95% new-response prediction interval: (7.538467, 15.46153). Student-t critical value: 2.570582.

The prediction interval adds new-observation noise. These calculations do not check independence, constant variance, Gaussian errors, representative sampling or correctness of the mean model. Extrapolation and model selection add uncertainties not represented by these formulas.

Keep a model fixed while checking different responses

Freeze copies the current fitted model. Later data edits update the current fit but keep this copy separate. The prediction row may change without refitting. Enter validation responses observed at that same predictor row. This public synthetic demonstration is not an untouched evaluation set.

Use Worked regression data. The input rows contain predictors followed by the response; the workbench supplies the intercept automatically. Here each row is x,z,y, not 1,x,z,y. Results update from valid data. Compare the coefficient, fitted-value and residual tables with the exact calculation below.

Choose Collinear predictors to inspect a rank failure. Do not interpret it as evidence that all fitting is impossible: a unique fitted response vector can exist while several coefficient vectors produce it. Restore the worked data before changing the Prediction row: predictor values only to 0.5,−0.5. This does not add a training row or refit the coefficients. The next lesson distinguishes the two displayed interval targets.

The experiment is bounded, temporary numerical evidence. Preserve your reasoning in a lesson response or your own document; fitting does not award statistical proficiency. Floating-point rank checks use a numerical tolerance, so a refusal can indicate an unsupported numerical case rather than a theorem of exact rank deficiency. Read the displayed error instead of reusing stale results.

Understand

Exact fit before inference

For this 8-by-3 design, direct column products give XᵀX=8I₃ and Xᵀy=(80,16,−8)ᵀ. Solving the three diagonal equations gives β̂=(10,2,−1). The fitted response vector is (9,9,7,7,13,13,11,11). Its residual vector has four entries of magnitude 1/2 and four of magnitude 3/2, so SSE=4(1/4)+4(9/4)=10.

The normal equations XᵀXβ̂=Xᵀy are a mathematical characterization. They do not require a program to form an explicit inverse. Here a thin QR certificate is Q=X/√8 and R=√8 I₃. In general, rank-aware QR or SVD is preferable to blindly inverting XᵀX; in the full-rank 2-norm setting, forming XᵀX squares the condition number. This unusually orthogonal example does not demonstrate every numerical difficulty.

For a general real n-by-p full-column-rank X, write X=QR with QᵀQ=Iₚ and nonsingular triangular R. Solve Rβ̂=Qᵀy. The fitted vector is ŷ=QQᵀy, and r=(I−QQᵀ)y satisfies Xᵀr=0. For any alternative β̂+d,

‖y−X(β̂+d)‖² = ‖r‖² + ‖Xd‖².

The cross term vanishes by orthogonality. Full column rank implies Xd=0 only for d=0, proving uniqueness. This is the orthogonal projection mechanism applied to a numeric design; least squares and QR develops the computational viewpoint.

Rank and residual dimension

The fitted vector is unique even if the columns are dependent: orthogonal projection onto their column space is unique. Coefficients then differ by vectors in the kernel of X. Adding w=2x to the worked design produces the family β₀=10, βz=−1, βₓ+2βw=2. It is not legitimate to report four individually identified coefficients with an ordinary inverse covariance matrix.

If the design rank is p, its orthogonal residual space has dimension n−p. In the worked example this is 8−3=5. Calling this a residual dimension is pure linear algebra. Calling s²=SSE/(n−p)=2 an unbiased estimate of a common error variance needs an error model. Duplicate records do not create independent information merely because the larger matrix has more rows.

A separately assumed statistical model

The fixed-design model is y=Xβ+ε: X is treated as known, and the error vector has mean zero and covariance σ²I. These covariance assumptions yield Cov(β̂)=σ²(XᵀX)⁻¹ and E(SSE)=(n−p)σ². The covariance formula follows by multiplying the error covariance on both sides of β̂−β=(XᵀX)⁻¹Xᵀε; the SSE expectation follows from the residual projection’s trace n−p. Consequently s² is unbiased under these assumptions. For our data the estimated coefficient covariance is 2(8I₃)⁻¹=I₃/4.

Exact small-sample t statements additionally require independent Gaussian errors with σ²>0, and n>p. A successful computation, zero residual sum or attractive plot verifies none of those collection/model assumptions. The Gaussian projection lesson supplies the normal and chi-square independence theorem used next.

Apply

For x=(−1,0,1) and y=(1,2,3), fit both an intercept-and-slope model and a slope-only model. Before calculating, predict which residual-sum claim survives. Then identify what would be lost by describing both as “the regression.”

Check the model distinction

The intercept model gives ŷ=2+x and all residuals zero. Through the origin the slope is Σxy/Σx²=1, so fitted values are (−1,0,1), residuals (2,2,2), and SSE=12. The residual vector is perpendicular to x but not to the absent intercept column. These models impose different mean spaces. Exact agreement in a tiny constructed example does not establish zero population noise.

Continue

In mean-response and prediction intervals, keep the fitted equation fixed and ask which uncertainty target the interval should describe. Retain the design, its rank and the residual calculations: they are the evidence needed to interpret the next formulas.

Apply · explain · revise

Practice and reasoning

Drafts save locally as you type. Reveal worked solutions when useful; the site remembers that exposure. Automatic matches check the requested answer, while written arguments require review against the rubric.