Improve an estimator by conditioning

Try

To estimate a Bernoulli success probability p from n iid trials, one learner reports just the first outcome U=X₁. This is unbiased but discards most data. Another learner is told the total S and averages the possible first outcomes among all records with that total. Exchangeability gives E[X₁|S]=S/n. The resulting estimator is the sample proportion. Conditioning has converted an inefficient unbiased estimate into a more stable one without changing its expectation.

Predict

Predict when the improvement is strict. For n>1 and 0<p<1, the first outcome still varies among records with many possible totals, so conditional variance remains positive on events of positive probability. At p=0 or p=1 all observations are fixed, and both estimators already have zero variance. “Improves variance” means never increases it, with strict reduction under an additional nondegeneracy condition. It does not mean every single observed estimate is closer to the true parameter.

Experiment

Use n=3 and condition on S=2. All compatible arrangements are equally likely. Calculate the conditional mean and conditional variance of X₁ directly.

Compatible record	X₁	Conditional weight
110	1	1/3
101	1	1/3
011	0	1/3

The mean is 2/3 and the variance is 2/9. Across the unconditioned experiment, Var(X₁)=p(1−p), whereas Var(S/3)=p(1−p)/3. The variance reduction is 2p(1−p)/3. Explain why the conditional variance 2/9 for this particular total is not the same object as its expectation across all totals.

Understand

Let U have a finite second moment and let V=E[U|T]. Write U−g(θ)=(U−V)+(V−g(θ)). The cross term has expectation zero because E[U−V|T]=0 and V−g(θ) is a function of T. Therefore E[(U−g(θ))²]=E[(V−g(θ))²]+E[Var(U|T)]. This proves squared-risk reduction even before unbiasedness is assumed. If U is unbiased for g(θ), the tower property gives EV=EU=g(θ), so variance also cannot increase.

Why require a sufficient statistic? Conditional expectation under a fixed parameter can depend on that unknown parameter, leaving an unusable “estimator.” Sufficiency supplies a parameter-free conditional law, allowing a common function V(T) to be computed. The variance identity holds for any conditioning variable, but its implementation as a statistic across the model family needs this extra point. In the Bernoulli example, the conditional average S/n is visibly independent of p.

Apply

For n≥2 Bernoulli trials, start from U=X₁X₂, an unbiased estimate of p². Given S=s, count ordered pairs of distinct successful positions: there are s(s−1) out of n(n−1) ordered pairs. Thus the Rao–Blackwell estimate is S(S−1)/(n(n−1)). Derive this by counting rather than guessing the square of the sample proportion. The naive square S²/n² has a different expectation because it includes diagonal terms Xᵢ²=Xᵢ.

The same conditional-averaging idea extends to convex losses through conditional Jensen, assuming the relevant expectations exist. Strictness then depends on the convexity and conditional distribution. Do not generalize the squared-loss equality condition to arbitrary losses without checking their structure.

Continue

Write the proof in three parts: the improved rule is a parameter-free statistic, it preserves the target expectation, and it reduces the chosen risk. Omitting the first part can hide an unknown parameter inside a formally attractive formula. Continue to Completeness and UMVU to determine when all unbiased functions of a sufficient statistic must coincide. That uniqueness step will turn improvement into an optimality theorem within a precisely defined class.