Compress data without losing model information

Try

You receive three iid Bernoulli observations but are told only that their total is one. The possible vectors are 100,010,001. For 0<p<1 each has unconditional probability p(1−p)², so its conditional probability given the total is 1/3. The hidden arrangement has no remaining dependence on p. In this model, recording the total loses the arrangement but loses no information about the parameter in the precise conditional-distribution sense called sufficiency.

Predict

Predict whether the same statement would hold if the pass probability changed with position. If the first observation used p and the other two used a different unknown rate, the three arrangements would no longer have equal conditional probabilities. Sufficiency is a property of a statistic relative to an entire model family. It is not a universal claim that order never matters. A data compression can be sufficient for one scientific question and inadequate after the model or target changes.

Experiment

Work through the conditional ratio rather than simply invoking symmetry. The denominator is the probability of the event S=1, which contains three disjoint vectors.

Vector x	Joint probability	Conditional probability given S=1
100	p(1−p)²	1/3
010	p(1−p)²	1/3
001	p(1−p)²	1/3

Repeat with four trials and total three. Identify exactly where the parameter factors cancel. At p=0, conditioning on S=1 is conditioning on a zero-probability event, so the elementary ratio is undefined. A sufficient conditional version need only agree with actual conditional probabilities on events supported by each parameter law; unsupported events require careful version conventions.

Understand

A statistic T is sufficient when the conditional law of the full data given T can be chosen independently of the parameter. For the dominated families used here, the factorization theorem says the joint density can be written gθ(T(x))h(x), with h parameter-free, exactly when T is sufficient. This is a theorem about densities relative to a common reference measure. General measure-theoretic versions require the corresponding conditional-law framework; our discrete and ordinary Euclidean models satisfy it.

In an iid Poisson sample, the joint mass is e^(−nλ)λ^(Σxᵢ)/Πxᵢ!. The total enters the parameter-dependent factor, while the factorial denominator is parameter-free. Thus the total is sufficient. Parameter-free does not mean constant across observations: h(x) can vary strongly with the data. What matters is that it does not vary with λ. Dropping a parameter-dependent normalizing factor would invalidate the argument.

Apply

For iid normal data with both μ and σ² unknown, expand Σ(xᵢ−μ)²=Σxᵢ²−2μΣxᵢ+nμ². The joint normal density then depends on the sample through the pair(Σxᵢ,Σxᵢ²). Both are needed for this factorization; the sample mean alone does not generally retain the scale information. If σ² is known, the squared-sum term can instead enter h, leaving the sample sum sufficient for μ.

This comparison is a useful learner action: underline the terms involving an unknown parameter in each of the two models. Watch one term move from the parameter-dependent factor to the parameter-free factor when the variance becomes known. Sufficiency changes because the family changed, not because the observed numbers changed.

Continue

Sufficiency does not assert that a statistic is unbiased, low-risk, or computationally convenient. It supplies an information-preserving conditional structure that later allows estimator improvement. Minimal sufficiency is a further compression question and is not proved by merely displaying one factorization. Continue to Exponential families to recognize a common density structure that makes sufficient statistics and moment identities visible. Retain the model dependence and dominated-family assumptions in every application.