Measurement: test the interpretation before trusting the score

Predict with context

Reliability concerns consistency; validity concerns evidence supporting an intended interpretation and use. A question can produce stable answers while consistently measuring the wrong thing. Validation examines comprehension, recall, judgment and response options.

Predict how the conclusion changes when the missing context or competing account is added. State your reason before reading feedback.

Inspect the authentic source

Census guidance describes cognitive interviews in which people explain their thinking while answering draft questions. Researchers use follow-up probes, revise wording and test again to discover problems in understanding and response.

U.S. Census Bureau — source note. Questionnaire testing standards, consulted October 2026. Authored paraphrase of an authentic official or educational source. This note is not a verbatim primary excerpt. Its description, purpose and limits must stay attached.

Worked investigation

Synthetic question asks ‘Do you regularly use the library?’ One respondent counts online borrowing, another counts building visits, and each answers consistently on repetition. High repeat agreement would not establish a common meaning. Revised questions specify activity and the past 30 days, but require another test rather than automatic acceptance.

Produce an inspectable result

Write a construct definition, two candidate questions and neutral probes asking how the answer was formed. Apply them to the supplied contrasting interpretations. Separate clarity revisions from changes in the construct; record a version identifier so old and revised responses are not silently pooled.

Keep the original record available alongside your analysis. Label invented details, attributed descriptions and your own interpretations separately. For numeric work, show the intermediate arithmetic and units; for interpretive work, point to the specific feature supporting a claim. A reader should be able to identify where another interpretation could enter.

Transfer

A new translation yields the same average score as the original. What remains to test?

Write the answer and its reason before opening feedback. Identify one similarity that makes transfer useful and one difference that limits it. State a specific next observation rather than saying only that more information is needed.

Limits and responsibility

The exercise validates a small operational question, not a clinical instrument. Do not diagnose or rank people from an unvalidated score; evidence of validity is specific to population, context and use.

A source can be authoritative about its own procedure or announcement without establishing every outcome that someone might infer from it. Ask whether the record describes an action, expresses a principle, documents an observation or interprets another source. Those are different evidential roles.

Extend

High school: Complete the specified diagram, record or table and explain one assumption in plain language. All supplied evidence can be examined without collecting personal information.

College: Add a bounded authentic case or methodological record that tests the weakest assumption. Document version, access, selection and interpretation, and present a justified alternative conclusion.

Audit agreement without calling it validity

Imagine two coders tagging ten fictional archive notices: “Does this notice state that an in-person building visit occurred?” Yes/no are nominal categories, not ordered scores. The supplied ledgers are authored demonstrations of possible coding results. No notice text, gold-standard label or actual independent coder study is supplied, so they cannot establish coding accuracy or measured inter-rater reliability in a real corpus.

Predict: could 90% raw agreement have a lower adjusted agreement coefficient than 80%? Could both coders agree perfectly while applying the wrong definition? Record what evidence would be needed to support the intended interpretation.

Initial agreement and validity claim Teaching ledgerBalanced labels with two disagreementsYes dominates with one disagreementEvery record labeled yes by both

Changing the ledger restores its original codes and retains your writing. Edits below are explicit hypothetical recoding, not adjudication against unseen source notices. The report retains the original and edited tables separately.

Record to changeNotice 1Notice 2Notice 3Notice 4Notice 5Notice 6Notice 7Notice 8Notice 9Notice 10 Coder to changeAB Replacement codeYesNoMissingApply hypothetical recodeTest no complete pairs

Notice	Original A/B	Edited A	Edited B	Included?
1	yes/yes	yes	yes	Yes
2	yes/yes	yes	yes	Yes
3	yes/yes	yes	yes	Yes
4	yes/yes	yes	yes	Yes
5	yes/no	yes	no	Yes
6	no/yes	no	yes	Yes
7	no/no	no	no	Yes
8	no/no	no	no	Yes
9	no/no	no	no	Yes
10	no/no	no	no	Yes

Reconstruct the 2×2 table

Both yes=4; A yes/B no=1; A no/B yes=1; both no=4. Complete pairs N=10; excluded incomplete pairs=0 of 10. A missing code is not a disagreement or a no. Complete-pair results can be biased if omissions are selective; do not hide the excluded denominator.

Among complete pairs A says yes 5 times and B says yes 5 times. Observed agreement Pₒ=(both yes+both no)/N=0.8. Under an independent-marginals reference, expected agreement Pₑ=(A-yes fraction×B-yes fraction)+(A-no fraction×B-no fraction)=0.5. This reference preserves each coder's label proportions; it does not show that actual coders guessed independently.

Cohen's κ=(Pₒ−Pₑ)/(1−Pₑ)=0.6. The coefficient is descriptive here, not a test or evidence of valid measurement. No “good enough” cutoff is imposed. Different label proportions can change κ even when raw agreement looks high.

Transfer and protocol revision

Compare the balanced and yes-dominant ledgers. Then inspect the all-yes case: why does 100% agreement leave κ undefined? Return to balanced, mark B's record 10 missing, and explain the changed denominator. Finally propose an independent coding protocol: define a building visit versus online use, retain actual notice context, code independently before reconciliation, record disagreements and inspect criterion/construct evidence. Improving this demonstration's numbers by recoding is not such evidence.

Revised interpretation and validation protocol

Reasoning guide

Balanced: 8/10 agree, both yes fractions 0.5, reference 0.5, κ=0.6. Yes-dominant: 9/10 agree, A yes 0.9 and B yes 1, reference 0.9, κ=0. All yes: observed 1 and reference 1 leave 0/0, not a perfect adjusted coefficient. Balanced with B record 10 missing has 7/9 agreement and reference 41/81; κ=.55. These different margins and missing records alter the arithmetic, not the presence of a valid construct or a correct interpretation.

Shared misunderstanding, copied labels or vague definitions can create apparent consistency. A real agreement study requires independent coding of defined units and appropriate uncertainty; validity needs evidence supporting the intended inference, beyond agreement. No clinical rating or personal disclosure is involved.

Method source and limits

JMP statistical details for the agreement statistic supplies method context for nominal agreement and its independent-rating reference. All ten-row ledgers and wording here are original, not source data or an actual coder study. No sampling confidence interval or universal reliability grade is calculated.

Copy the coding-audit report

Read-only JSON report{ "activity": "MET-03.b coding agreement and validity", "fictional": true, "preset": "balanced", "original": [ { "id": 1, "a": "yes", "b": "yes" }, { "id": 2, "a": "yes", "b": "yes" }, { "id": 3, "a": "yes", "b": "yes" }, { "id": 4, "a": "yes", "b": "yes" }, { "id": 5, "a": "yes", "b": "no" }, { "id": 6, "a": "no", "b": "yes" }, { "id": 7, "a": "no", "b": "no" }, { "id": 8, "a": "no", "b": "no" }, { "id": 9, "a": "no", "b": "no" }, { "id": 10, "a": "no", "b": "no" } ], "edited": [ { "id": 1, "a": "yes", "b": "yes" }, { "id": 2, "a": "yes", "b": "yes" }, { "id": 3, "a": "yes", "b": "yes" }, { "id": 4, "a": "yes", "b": "yes" }, { "id": 5, "a": "yes", "b": "no" }, { "id": 6, "a": "no", "b": "yes" }, { "id": 7, "a": "no", "b": "no" }, { "id": 8, "a": "no", "b": "no" }, { "id": 9, "a": "no", "b": "no" }, { "id": 10, "a": "no", "b": "no" } ], "audit": { "total": 10, "n": 10, "excluded": 0, "yy": 4, "yn": 1, "ny": 1, "nn": 4, "agree": 8, "aYes": 5, "bYes": 5, "observed": 0.8, "expected": 0.5, "kappa": 0.6, "reason": null }, "worksheet": { "prediction": "", "revision": "" }, "feedbackSeen": false, "limits": "Synthetic author-specified coding ledgers, not real independent raters. Complete-pair descriptive agreement; no sampling uncertainty, gold standard, construct validation or universal quality threshold. Expected agreement is an independent-marginals reference, not observed guessing. Temporary practice." }

Writing is temporary, 3,000 characters per field. Reset restores balanced/original codes and blank writing, retaining guide/report exposure this mount; reload clears it. This ungraded simulation does not establish mastery.

Reset coding agreement