Compare estimators through expected loss

Try

Suppose X has mean θ and variance one. Two proposed estimates of θ are T=X and S=X/2. Which is better? Before answering, specify the loss. Under squared error, the first has expected loss one. The second has variance 1/4 and bias−θ/2, giving expected loss(1+θ²)/4. Shrinkage helps when θ²<3 and hurts when θ²>3. There is no parameter-free winner over the entire real line. Compute the comparison at θ=0 and θ=3 before considering the general formula.

Predict

Predict whether removing bias must reduce total error. A constant estimate equal to zero has no sampling variance at all, but its squared error is θ². Conversely, an unbiased estimate can fluctuate widely. Bias is an expectation over repeated samples, not the error of one realized estimate. An estimator can be unbiased and produce a poor estimate on a particular sample. These distinctions matter whenever a visual comparison shows just one simulated run.

Experiment

The live experiment below uses a separate binomial model. Change the sample size, true probability and shrinkage strength. It enumerates every possible success total, so the displayed risk is an exact finite expectation up to floating-point arithmetic, not a Monte Carlo estimate. Predict where shrinkage will help before changing the true probability. The worked normal-location comparison following it is a different model.

Live experiment: compare risk, not just bias

Assume X counts successes in n independent Bernoulli trials with one fixed probability p. Compare the maximum-likelihood estimate X/n with the estimate (X+a)/(n+2a), which moves estimates toward 1/2. Predict where this tradeoff lowers mean squared error and where it raises it.

Sample size n 1251020True probability p 00.050.20.50.80.951Added count a 00.512Reset experiment

At p=0.5, the added-count estimate has expectation 0.5, bias 0 and variance 0.0255102. Its mean squared error is variance + bias² = 0.0255102. Direct enumeration gives 0.0255102.

The MLE is unbiased in this binomial model and has mean squared error 0.05. Here the reduction in variance outweighs the squared bias.

The formulas are bias=a(1−2p)/(n+2a), variance=np(1−p)/(n+2a)², and MLE risk=p(1−p)/n. They follow from E[X]=np and Var(X)=np(1−p). At p=0 or 1 the data are deterministic; added counts can still create error.

Inspect every count and its contribution to risk

Count k	P(X=k)	Estimate	Error	Probability × error²
0	0.03125	0.14285714	-0.35714286	0.00398597
1	0.15625	0.28571429	-0.21428571	0.00717474
2	0.3125	0.42857143	-0.07142857	0.00159439
3	0.3125	0.57142857	0.07142857	0.00159439
4	0.15625	0.71428571	0.21428571	0.00717474
5	0.03125	0.85714286	0.35714286	0.00398597

For a>0, the same estimate is the posterior mean under a Beta(a,a) prior and squared-error loss. That is a separate Bayesian interpretation: the risk above still averages over data at each fixed true p, not over the prior. No single choice is declared best for every probability or loss function.

Compare the center with a boundary while holding n and a fixed. Explain why “unbiased” does not mean “smallest mean squared error,” and why the boundary result prevents an unrestricted superiority claim. All arithmetic is bounded and uses browser floating point; displayed probabilities are rounded. Settings are temporary.

Use the exact risk table rather than Monte Carlo randomness. Fill the missing reasoning for each row and identify the crossing value between the first two estimators. All entries use the same loss and same sampling variance.

Estimator	Bias	Variance	Squared-error risk
X	0	1	1
X/2	−θ/2	1/4	(1+θ²)/4
0	−θ	0	θ²

Then restrict the parameter space to [−1/2,1/2]. On this smaller space the constant zero estimate has risk at most 1/4, strictly below the risk of X. Changing the permitted population parameters changes a uniform comparison, even though the estimators themselves are unchanged.

Understand

For a statistic T with a finite second moment, write T−θ=(T−ET)+(ET−θ). Squaring and taking expectations gives Var(T)+(ET−θ)² because the cross term contains E(T−ET)=0. This is the bias–variance decomposition of mean squared error, or MSE. A risk function is the expected loss as a function of the true parameter; squared error is one possible loss, not a law of nature.

An estimator dominates another under a specified loss if its risk is no larger at every parameter and smaller at at least one. A smaller risk at one selected parameter does not establish domination. Likewise, comparing procedures under different losses does not create a meaningful ordering until the decision goal is specified. We will later use conditional expectation to reduce squared risk and loss functions to derive Bayesian actions.

Apply

Derive the bias of the variance estimate that divides by n. For iid observations with mean μ and variance σ², expand the identity Σ(Xᵢ−X̄)²=Σ(Xᵢ−μ)²−n(X̄−μ)². The expectations on the right are nσ² and σ². Thus dividing the left side by n has expectation(n−1)σ²/n; dividing by n−1 gives an unbiased variance estimator when n≥2. This correction does not by itself prove the corrected estimator minimizes MSE. To make that claim, its variance must also be compared.

As an error diagnosis, explain why replacing every population expectation in a risk formula by one observed squared error does not estimate the entire risk function reliably. A repeated-sampling criterion requires more information than one favorable outcome.

Continue

Keep the parameter space, loss and comparison class beside every optimality claim. Unbiased estimation theory deliberately restricts the comparison class; decision theory permits other tradeoffs. Neither should be presented as a universal rule. Continue to Consistency and moments to study what changes as sample size grows. Finite-sample bias, finite-sample risk and limiting consistency are related but distinct properties that need separate arguments.