Separate detectable differences from useful changes

Try

A large experiment detects a small improvement with a very low p-value. Before deciding to deploy it, ask how large the benefit is and whether it exceeds the cost or minimum useful change. Evidence against an exact zero effect is not the same as evidence that an effect is worth acting on.

Predict

Predict how a fixed tiny difference can become statistically detectable as sample size grows. Standard error can shrink while the effect stays unchanged. Conversely, a potentially important estimate can remain uncertain in a small study. Separate the size of the estimate from the precision with which it was measured.

Experiment

An original synthetic process has a failure risk of 0.12 before a change and 0.09 afterward. The absolute decrease is 0.03, or three percentage points. The relative decrease is 0.03/0.12=0.25, or 25%. Write both measures in a report; saying only “25% improvement” can obscure the baseline and absolute consequences.

For a different example, suppose a justified interval for mean time saving is [0.1,0.5] seconds, but the operational requirement is at least 1 second. Mark zero and the required benefit on a number line. The entire interval is positive yet below the required threshold. A significance label alone would conceal that distinction.

Understand

Effect size describes the magnitude of a comparison. Depending on the question it might be a difference of means, difference of proportions, risk ratio or standardized difference. Units and baseline matter. An absolute risk difference answers a different practical question from a relative decrease, although both can be computed from the same probabilities.

A p-value depends on effect size relative to uncertainty, among other model details. It is not an effect-size measure. With a very large sample, small departures from a point null can be detected. With a noisy or small sample, a large estimate can have a wide interval. Report a suitable estimate and interval rather than reducing every result to significant or nonsignificant.

A practical threshold should be stated before inspecting the outcome where possible. For example, a team may require a minimum time saving after accounting for implementation costs. Compare the entire interval with zero and with that threshold. An interval wholly above zero but below the threshold suggests a small positive effect rather than the required benefit. An interval spanning the threshold leaves the decision uncertain.

A difference test that fails to reject zero does not prove equivalence. Equivalence requires a prespecified band of differences considered acceptably small and an inferential procedure designed for that claim. A wide interval can include both zero and effects far too large to call equivalent. Similarly, a safety claim may require ruling out harmful changes rather than merely failing to detect them.

Average effects can hide variation across people or conditions. A positive mean does not guarantee every individual benefits. Subgroup exploration needs attention to sample size, multiplicity and prespecification. Costs, adverse outcomes and feasibility may affect decisions even when the estimated primary effect is favorable.

Keep causal interpretation tied to design. An observational before-and-after change may reflect other events, regardless of how precisely it is estimated. Practical relevance does not rescue an unidentified causal effect, and a well-identified effect does not automatically satisfy a practical threshold.

Apply

Write a two-sentence decision brief for the time-saving example. The first states the estimated benefit and uncertainty; the second compares it with the stated requirement and names an assumption. In practice, repair an equivalence claim based only on p>0.05 without pretending that a different conclusion follows from the same insufficient information.

Continue

Next compare two proportions. Start with an interval that estimates their difference without imposing equality, then contrast the equality-null test.