Plan the consequences of a decision

Try

A quality test decides whether evidence suggests a defect rate has increased. There are two ways the decision can go wrong: declaring an increase when the null is true, or failing to detect an increase that really exists. Describe the operational consequences before attaching statistical labels. The costs need not be symmetric.

Predict

Predict what happens if the rejection threshold becomes stricter while sample size stays fixed. Fewer null results will cross it, but some real effects will also be harder to detect. A procedure cannot generally reduce both error types simply by changing the cutoff in one direction.

Experiment

Use the decision table below.

Actual state	Reject the null	Fail to reject
Null true	Type I error	Correct nonrejection
Specified alternative true	Correct detection	Type II error

At a particular alternative effect, suppose β=0.30. Power is 1−β=0.70. This is a probability over repetitions under that alternative, not a score attached to one realized result. If two independent tests each have exact Type I rate 0.10 under their true nulls, the chance of at least one false rejection is 1−0.9²=0.19. Independence is part of that calculation.

Understand

Type I error means rejecting a true null. A valid level-α test bounds that probability by α under its null assumptions; common approximate tests aim for approximately that control. Type II error means failing to reject at a particular false-null state. Its probability β varies across alternatives, so power 1−β is a function of the true effect and other design features.

A statement such as “80% power” is incomplete without the effect size, variability, sample size, direction, significance level and design assumptions used. Smaller effects are often harder to detect. More independent observations usually improve power for a fixed nonzero effect under the same model, but do not repair bias or guarantee a scientifically useful result.

Lowering α shrinks a conventional rejection region, reducing false-positive risk but generally increasing false-negative risk at fixed alternatives and sample size. Raising α makes detection easier but tolerates more false alarms. Choose thresholds in advance with the costs and scientific question in mind rather than tuning them until a preferred result appears.

Multiplicity matters when many hypotheses, outcomes or subgroups are searched. Even if each individual test has a valid nominal rate, the chance of at least one false rejection can be much larger. The simple independent example illustrates the mechanism; dependent tests need a different calculation or appropriate multiplicity control. Exploration should be labeled and followed by suitable confirmation.

Power is a planning tool, not proof that a nonsignificant result means no effect. An interval spanning both zero and important effects indicates remaining uncertainty. Recalculating “observed power” from the same result does not add independent information about the effect. Report estimates, intervals and design limitations instead of substituting a post hoc label for evidence.

Apply

Write two short decision stories for the production example, one for each error. State whose costs are involved and what follow-up could reduce consequences. Then specify a minimum effect that would matter for a future study and list the assumptions needed for a power calculation without inventing an unsupported numerical answer.

Continue

Next separate statistical detectability from practical importance. A precise tiny difference and an imprecise large estimate require different decisions.