Prove weak and strong averaging results

Try

Averages of independent observations should stabilize, but there are two different questions. Does the probability of a large error tend to zero at each sample size? Or does almost every entire observation sequence eventually stay close forever? The weak and strong laws answer these questions separately. Start with finite variance, then prove the iid strong law assuming only an integrable first moment.

Predict

Predict whether applying Chebyshev at every integer n and summing will prove the strong law. Its bound is proportional to 1/n, whose series diverges. This does not disprove the strong law; it shows the attempted proof is too weak. A maximal inequality and dyadic sample sizes produce a summable bound and control the gaps.

Experiment

For centered iid variables with variance one and tolerance ε=1/2, compare the estimates.

Dyadic endpoint	Bound for maximum crossing before endpoint
2⁴	1/4
2⁵	1/8
2⁶	1/16

The event is max over k≤2^m of abs (S_k)>ε2^m, not merely a deviation at the endpoint. Add the remaining geometric tail to estimate the probability of any late dyadic failure. Explain why an endpoint-only estimate would not automatically control intermediate partial sums.

Understand

Let X_i be independent identically distributed real random variables with finite mean μ and variance σ². Set S_n=Σ_(i=1)^n(X_i−μ). Independence gives E(S_n²)=nσ². Chebyshev therefore yields P(abs (S_n/n)>ε)≤σ²/(nε²), proving convergence in probability: the weak law.

Apply this with N=2^m and λ=ε2^m. The probability bound is σ²/(ε²2^m), summable in m. The first Borel–Cantelli lemma says that almost surely, for all sufficiently large m, every k≤2^m satisfies abs (S_k)≤ε2^m. For 2^(m−1)< n≤2^m, divide by n to get abs (S_n)/n≤2ε. Intersect the probability-one sets for ε=1,1/2,1/3 and so on. This proves S_n/n→0 almost surely, the strong law.

The maximal estimate supplies exactly the control needed between the selected dyadic times. Independence was used to eliminate covariance in the weak law and to eliminate the stopped cross terms in the maximal inequality. The theorem is not justified for arbitrary dependent data merely because each observation has the same distribution.

Why truncation is needed

A Pareto observation with density (3/2)x^(−5/2) for x≥1 has mean3 but infinite second moment. The variance calculation above cannot be applied to it. For cutoff b≥1, zero truncation Y=X1(X≤b) has expectation3(1−b^(−1/2)) and second moment3(√b−1). Both are finite at every fixed cutoff; the missing mean3/√b tends to zero. This suggests growing truncations, but a proof must also control the random centered fluctuations and the observations discarded over an infinite sequence.

A small tail probability can carry substantial expectation

Predict the missing mean before changing the cutoff. This is the exact analytic Pareto family with density αx^(−α−1) for x≥1, not a simulated sample or an empirical guarantee.

Tail exponent α11.53 Cutoff b1241664256Reset truncation

Full mean: 3.00000. Variance: infinite. Tail probability P(X>b) = 0.125000.

The bar uses a fixed probability scale0–1: navy is X≤b, pale remainder is X>b. Tiny positive tails may be smaller than a pixel; the number above remains authoritative.

Zero truncation Y = X1(X≤b)

E[Y] = 1.50000

Clipping C = min(X,b)

E[C] = 2.00000

Omitted mean E[X1(X>b)]

1.50000

Second moment of the zero truncation

E[Y²] = 3.00000

Clipping adds bP(X>b) = 0.500000 to the zero-truncated mean. Neither operation conditions on being below b. The theorem uses a growing cutoff n on observation n, not a fixed cutoff for every observation.

For iid observations, a union/integral bound for any truncation event X_n>n after integer b is 1.00000. This controls truncation events, not all sample-mean errors, and is not an observed frequency.

Repair the prediction

At α=1.5,b=4, only1/8 of the probability lies above the cutoff, but it carries1.5 of the full mean3. Zero truncation gives1.5; clipping gives2. At α=1 every finite cutoff has finite moments, yet the original mean is infinite. Finite truncated calculations cannot supply the missing integrability assumption.

Only18 bounded analytic settings; no animation or saved state. Use the written tasks to retain a prediction, derivation and revised conclusion. This display does not establish almost-sure convergence from a finite experiment.

Step1: discarded observations occur only finitely often

Assume now that X_n are iid real random variables and E|X_1| is finite. Set Y_n=X_n1(|X_n|≤n). Each Y_n is bounded and the Y_n remain independent, though they are not identically distributed. For u≥0, the number of positive integers n with n<u is at most u. Tonelli therefore gives

$$\sum_{n=1}^{\infty}\mathbb P(|X_n|>n)=\mathbb E\sum_{n=1}^{\infty}\mathbf1_{\{n<|X_1|\}}\leq\mathbb E|X_1|<\infty.$$

The first Borel–Cantelli lemma implies X_n=Y_n eventually, almost surely. It requires summability, not independence of these events. On each such sample path the finite sum of all differences, divided by n, tends to zero. The time after which equality holds is random; no deterministic universal stopping time is asserted.

Step2: enough weighted variance is summable

For u>0, let m=max(1,ceil(u)). The estimate Σ from n=m to infinity of n⁻²≤m⁻²+1/m≤2/m shows u²Σ_(n≥u)n⁻²≤2u when u≥1. When0≤u≤1, the same bound follows from Σn⁻²≤2 and u²≤u. Thus, using nonnegative interchange and identical distribution,

$$\sum_{n=1}^{\infty}\frac{\operatorname{Var}(Y_n)}{n^2}\leq\mathbb E\left[|X_1|^2\sum_{n\geq |X_1|}\frac1{n^2}\right]\leq2\mathbb E|X_1|.$$

This does not require E[X_1²] to exist finitely. The order of summation lets the cutoff supply the missing decay. Center and weight: Z_n=(Y_n−EY_n)/n. These independent variables have mean zero and summable variances.

Step3: prove convergence of the centered weighted series

Here is the precise consequence of the already proved maximal inequality. Choose increasing integers mj so that the variance tail Σ(n≥m_j)Var(Z_n)≤2^(−3j). Apply the maximal inequality to each finite tail beginning at m_j with threshold2^(−j). Its crossing probability is at most2^(−j), uniformly in the final index. Taking increasing unions of those finite events gives the same bound for the supremum over all finite tail endpoints.

The bounds are summable in j. Borel–Cantelli says that for almost every path, eventually every finite partial tail starting at m_j has magnitude at most2^(−j). Subtract two such tails: any sum from p to q, with q≥p≥m_j, has magnitude at most2·2^(−j). These partial sums are Cauchy in the real numbers. Consequently ΣZ_n converges almost surely. This establishes series convergence, not absolute convergence.

Step4: turn the weighted series into sample averages

For any real sequence ak such that s_n=Σ(k=1)^n a_k/k converges, finite summation by parts yields

$$\frac1n\sum_{k=1}^{n}a_k=s_n-\frac1n\sum_{k=1}^{n-1}s_k\longrightarrow0.$$

The two terms have the same limit by the elementary Cesàro lemma: split the average into a fixed finite prefix and a tail uniformly close to the limiting value. This is the required form of Kronecker’s lemma, proved here rather than invoked. Apply it pathwise to a_k=Y_k−EY_k. Their averages converge to zero almost surely.

Finally, dominated convergence gives EY_n=E[X_1 1(|X_1|≤n)]→EX_1=μ because |X_1| is integrable. Cesàro averaging gives n⁻¹ΣEY_k→μ. Combine the centered average, the deterministic expectation average and the finitely many discarded differences. This proves n⁻¹ΣX_k→μ almost surely under the finite-first-moment iid assumption. Almost-sure convergence also implies convergence in probability: apply dominated convergence to the indicators of a fixed error event, which eventually vanish on the probability-one convergence set.

The MIT18.175 Spring2014 Lecture9 notes give the primary general-strong-law/truncation reference. This lesson completes centered-series control using its existing maximal inequality and the explicit summation-by-parts argument above.

Limits of the claim

Replacing every X_n by the same integrable nonconstant X destroys independence: the average stays X instead of tending to its expectation. Finite numerical truncations also cannot prove integrability. For the Pareto density x⁻² on x≥1, the mean is infinite while every bounded truncation has finite mean log b. The integrable theorem does not apply. In fact the nonnegative iid averages diverge to infinity almost surely: for each fixed integer b, apply the bounded strong law to X1(X≤b); the original average has liminf at least log b. Intersect the countably many probability-one sets and let b grow. This is a separate extended-limit conclusion, not convergence to a finite expectation.

Apply

Set every observation equal to one nonconstant Bernoulli variable. Its sample average never changes, disproving the iid conclusion without independence. For actual data, explain which design assumption makes independence plausible and what dependence would require a different theorem. The law concerns the target expectation of the assumed distribution, not automatic removal of sampling bias.

Continue

The integrable iid strong law is now proved by truncation; it does not give a universal finite-sample rate for every integrable distribution. Next, weak convergence describes distributional limits, including normal approximation under its separate hypotheses. The next lesson supplies Lévy’s continuity theorem by separate tightness and uniqueness arguments; the strong law itself does not prove it.

Continue to Connect distributional limits to integral transforms.