Invert tests into coverage statements

Try

A test of each candidate normal mean μ accepts it when the observed sample mean is within a specified standardized distance. Instead of selecting one candidate in advance, collect all candidates accepted by their corresponding tests. The resulting set is a confidence interval. This is not a new probability principle: the event that the true mean belongs to the set is exactly the event that its own test accepts. Valid test calibration transfers directly to coverage.

Predict

Predict whether a discrete test with size at most α yields coverage exactly 1−α. It need not: conservative tests produce coverage greater than the nominal level at some parameters. Randomization can sometimes achieve exact size, but many practical procedures prefer deterministic conservative coverage. Also predict whether two marginal 95% intervals must simultaneously cover both parameters with probability 95%. Marginal guarantees alone do not say this. Multiple statements require a joint probability argument, not repeated use of a marginal percentage.

Experiment

Observe one Bernoulli outcome. Report C(0)=[0,0.8] after a failure and C(1)=[0.2,1] after a success. Compute coverage at each parameter region by checking which possible sets contain p.

True parameter region	Sets containing p	Coverage
0≤p<0.2	C(0) only	1−p
0.2≤p≤0.8	Both	1
0.8<p≤1	C(1) only	p

The coverage is at least 0.8, but is not constantly 0.8. Track the closed endpoints carefully. This deliberately simple rule is a teaching example, not a recommended interval for general analysis. Its purpose is to expose the random-set meaning of coverage without normal approximations or software quantiles.

Understand

For each θ, let Aθ be the acceptance region of a test with rejection probability at most α under that θ. Define C(x) to contain precisely the θ for which x belongs to Aθ. Then Pθ(θ∈C(X))=Pθ(X∈Aθ)≥1−α. The parameter is fixed in this repeated-sampling statement; the set changes with X. Once an interval is realized, a posterior probability for θ requires a prior model. Confidence coverage and Bayesian credible probability can numerically coincide in some models while retaining different constructions and interpretations.

For a known-variance normal upper-tail test, reject when X̄>μ₀+z_(1−α)σ/√n. Under an alternative μ₁, standardizing around μ₁ gives power 1−Φ(z_(1−α)−(μ₁−μ₀)√n/σ). The effect size, variance, sample size and chosen direction all matter. Power is a prospective probability under a specified alternative, not a posterior probability that a rejected null is false.

Apply

Hold α, σ and a positive effect μ₁−μ₀ fixed in the power formula. Increasing n increases the standardized shift and therefore power. Halving the effect requires four times the sample size to preserve that shift. This is a model-based planning relation, not a guarantee that variance estimates or effect assumptions are correct. State the assumed effect in a power calculation rather than replacing it with the observed effect and treating the result as independent confirmation.

For simultaneous inference on m targets, construct intervals with marginal failure probabilities at most α/m. The union bound gives probability of any failure at most their sum, hence at most α. Independence is unnecessary. This Bonferroni construction can be conservative, especially with dependent targets, but its guarantee follows directly from the probability inequality. Selecting only attractive intervals after seeing data changes the reporting procedure and needs separate consideration.

Continue

When reporting an interval, identify the procedure, assumptions, confidence level and whether its guarantee is exact, conservative or asymptotic. For tests, state a meaningful alternative when discussing power. Continue to Bayesian decisions and loss to give parameter probabilities a specified prior-based construction and to connect estimation with the costs of actions. Neither framework should be judged by interpreting it as the other.