Decide what a study must answer before collecting data

Try

A community center wants to decide whether to open one hour earlier. Its manager asks, “Do people like our center?” A favorable answer would not reveal who needs an earlier opening, how often, or whether the people currently absent are the ones most affected. Before collecting any responses, rewrite the question so an answer could inform the proposed decision.

For this worked example, target adults living in the center’s service area during the first four complete weeks of September. An observational unit is one adult resident. Define the outcome as whether that resident wanted to use the center before its current opening time at least once during those weeks. This measures reported unmet demand, not guaranteed future attendance. Record that distinction beside the question. All examples here are synthetic planning exercises, not claims about an actual community.

Predict

Predict how the answer might change if the manager surveys only people already inside at noon. Name two kinds of residents that this method might miss. Then predict whether asking every resident would remove all problems. A complete census removes sampling variation from selecting only some residents for that defined population and date; it does not automatically remove unclear wording, inaccurate recall or coercive collection.

Write the decision, population and outcome before viewing a result. Otherwise it is tempting to redefine “demand” after discovering which response looks favorable. Exploratory questions are useful, but report them as exploratory rather than pretending they were the original aim.

Experiment

Complete the following planning record. Change one row, then explain whether the revised data still answer the original question.

Planning choice	Worked specification	What to check
Decision	Whether to trial an earlier opening	Demand alone does not measure staffing cost
Population and unit	Adult service-area residents; one adult	Existing visitors are a different population
Outcome	Any unmet before-opening use in the named four weeks	Define “wanted to use,” dates and opening times
Comparison	Residents who start work before 9 a.m. versus others	This is a measured grouping, not an assigned treatment
Recruitment	Randomly select residents from a defensible resident frame	Document frame omissions and nonresponse
Analysis	Estimate each group’s proportion and their difference	Keep missing answers separate from “no”

Suppose the frame contains 600 adults. Selecting 120 randomly is a sample. Obtaining the specified measurement from all 600 would be a completed census of that frame. Sending 600 invitations and receiving 420 replies is an attempted census with 180 missing responses, not a completed census. Larger coverage can be valuable, but the missing residents might differ systematically on unmet demand. Replacing missing replies by “no” changes the data rather than repairing that uncertainty.

Understand

Collection timing creates another distinction. A prospective observational study enrolls residents now, records relevant characteristics, and follows their unmet-use reports over the next month. A retrospective study examines existing records or asks about earlier events. Neither design, by itself, randomly assigns opening times. An association between work schedule and demand can have alternative explanations such as distance, childcare responsibilities or transportation access.

A prospective design can establish that a recorded exposure preceded a later outcome, but that ordering does not eliminate confounding. A retrospective attendance archive may be inexpensive yet unable to record visits people wanted to make but could not. Choose the data source for the variable it actually measures. A detailed archive is not automatically the right evidence.

Keep two scope statements separate. Random selection from a good frame can support generalization to the represented population, subject to nonresponse and measurement problems. Random assignment of treatments can support causal comparisons under the experimental protocol. Simply following people longer supplies neither kind of randomization.

Ethics belongs before collection. Obtain informed participation, avoid pressure from service providers, collect only needed information, restrict access and set a deletion schedule. Do not publish named addresses to prove that residents were real. Aggregate categories can still identify people when groups are very small; combine or withhold such summaries where necessary. A consent checkbox does not make every unnecessary disclosure appropriate.

Apply

Try a changed case: a museum wants to know whether visitors understand its new exhibit labels. It proposes counting social-media likes. Write an outcome that actually measures understanding, a sampling frame, a reference period and a comparison before collecting data. State one conclusion that even a perfectly measured observational comparison would not establish.

Compare a possible repair

One plan samples exiting visitors during randomly chosen visit periods and asks a short, prespecified explanation question scored with a stated rubric. The unit is a visitor, the outcome is the response score, and a comparison might distinguish first-time from returning visitors. Likes measure online engagement rather than the proposed understanding outcome. Sampling only exiting visitors does not describe nonvisitors, and first-time versus returning visitors were not randomly assigned those histories. To estimate a causal label effect, a separate randomized label comparison would be needed, with consistent content and a defensible consent procedure.

Continue

Use the practice to write a plan another person could carry out. A good response identifies what will be collected, from whom, when and why, then states what the resulting evidence cannot settle. The next lesson turns selection and treatment allocation into explicit random procedures rather than the instruction “choose people randomly.”