Build a null distribution from the assignment mechanism

Try

Four units are assigned exactly two to treatment by a fair random mechanism. Under the sharp null that treatment changes none of their outcomes, the observed outcomes can be held fixed while treatment labels are reassigned according to the original design. Count the allowed assignments before thinking about a p-value.

Predict

Predict whether all 16 binary label strings are allowed. They are not: exactly two treatment assignments were required, leaving C(4,2)=6 allocations. A reference distribution must reflect the assignments the experiment could actually have produced. Ignoring the design changes the question.

Experiment

Use original synthetic fixed outcomes 0,1,3,4 under the sharp null. For each treatment pair, calculate treatment mean minus control mean.

Treated outcomes	Difference
0,1	−3
0,3	−1
0,4	0
1,3	0
1,4	1
3,4	3

If the observed treated pair is 1,4, the statistic is 1. Two of the six equally likely allocations give a statistic at least 1, so the exact upper-tail randomization p-value is 2/6=1/3. Include the observed allocation and ties in the stated tail rule.

Understand

Randomization inference derives its reference from the experimental assignment mechanism. A sharp null of no individual treatment effect specifies enough missing potential outcomes to recompute the statistic under every allowed assignment: each unit would have the same outcome under either treatment. This is stronger than merely asserting that average treatment effect is zero when individual effects can differ.

Choose a statistic and direction before examining the results. Differences of means are one possibility; other statistics can target different features. Enumerating every allocation gives an exact randomization distribution conditional on the fixed units and outcomes under the sharp null. Exact refers to this reference calculation, not to the absence of design, measurement or external-validity limitations.

For larger experiments, a random sample of allowed assignments can approximate the distribution. Report the number of simulated assignments and simulation uncertainty. A conventional Monte Carlo test often uses an appropriate plus-one correction, such as (b+1)/(B+1) for B simulated assignments under the corresponding scheme, rather than reporting a zero p-value after no exceedances. The precise simulation scheme and tie rule must be stated.

Respect restrictions. Blocked assignment requires reassigning within blocks; paired randomization may allow swaps within pairs rather than arbitrary global shuffling. Unequal assignment probabilities require the appropriate weighting or generation mechanism. A generic shuffle button is not sufficient evidence that the reference matches the design.

Observational groups do not automatically support unrestricted permutation. Exchangeability under a suitable model can justify certain permutation procedures, but confounding and selection may violate that assumption. Randomly shuffling labels after data collection does not retroactively randomize treatment or establish causality.

A randomization test conditions on the studied units and assignment. Population generalization requires an additional sampling or transport argument. Keep that distinction visible when translating a rejection into a scientific claim.

Apply

Enumerate the practice dataset’s six allocations independently, checking the sign of each difference. Then describe how the allowable assignments would change if the four units formed two prespecified pairs with one treated per pair. There would be four assignments, and their statistics need a new reference rather than reusing the six-allocation table.

Continue

Finally distinguish randomization under a null from bootstrap resampling for sampling variability. Their mechanisms and targets differ even though both reuse observed information.