Specify a test before seeing its answer

Try

A service promises an error probability of 0.10. A team wants evidence that a redesign lowers it. State the parameter p as the error probability in the intended operating conditions. The usual boundary null is H0:p=0.10, with alternative Ha:p<0.10. The alternative reflects the question, not whichever direction later looks favorable.

Predict

Predict whether a surprisingly large observed error rate would support that lower-rate alternative. It would not. A test must define what counts as more extreme in the direction of the question. For a two-sided question, large deviations on either side matter instead.

Experiment

Use a supplied symmetric standard-normal table. Suppose an observed statistic is z=−1.8 and P(Z≤−1.8)=0.0359. A lower-tailed test has p=0.0359. A corresponding two-sided equal-tail test has p=2·0.0359=0.0718. At α=0.05 the first rejects its null boundary and the second does not.

These are different prespecified procedures. Choosing the lower-tailed version only after discovering the observed sign would not preserve the claimed false-positive behavior. Sketch the shaded lower tail and the two shaded tails, using the same observed distance from zero.

Understand

A null hypothesis specifies a population parameter or probability model, not an observed statistic. A test statistic measures discrepancy between data and the null on a relevant scale. Its reference distribution describes behavior if the null and associated assumptions hold. A p-value is the probability of a statistic at least as extreme as the observed one according to the chosen rule, conditional on that reference model.

The p-value is not the probability that the null is true, nor the probability that the effect arose from chance in every possible sense. It evaluates data against one model. A small value can reflect an effect, violated assumptions, selection or another mismatch; interpretation requires more than comparing a decimal to a threshold.

The significance level α is a prespecified rejection threshold controlling Type I error for a valid procedure, exactly or approximately depending on the method. Reject when the chosen rule places the result in its rejection region. If p exceeds α, say “fail to reject,” followed by the relevant alternative. Do not accept the null as proven or claim equivalence from a nonsignificant result.

For common symmetric z and t tests against a two-sided alternative, the two-sided p-value doubles the smaller tail beyond the observed magnitude. That shortcut is tied to the reference distribution and definition of extremeness; it is not a universal rule for every discrete or asymmetric test. Use a stated procedure rather than improvising tails.

Changing outcomes, subgroup definitions or stopping rules after inspecting results can alter false-positive rates. A transparent analysis distinguishes prespecified confirmation from exploratory findings that may motivate new data. Reporting the test’s original direction, statistic and assumptions helps readers assess what was actually evaluated.

Apply

Write a complete hypothesis pair for a claimed improvement in average time, with the sign of improvement defined. Then construct an example of a large deviation in the opposite direction. Explain why it would be important operationally even if it does not reject your prespecified one-sided null in the intended direction.

Continue

Next calculate a one-proportion z statistic. Unlike an estimation interval, its uncertainty comes from the proposed null probability.