Audit a score before interpreting a trait
Separate consistent measurement, valid interpretation and comparison with a reference group.
A question to investigate
If a test gives consistent results, does it necessarily measure the intended ability?
Calculate a standardized score and explain why reliability alone cannot validate an interpretation.
Mechanism and evidence
Standardization concerns consistent administration and scoring. Norms provide a reference distribution for interpreting a score. Reliability concerns consistency; validity concerns whether evidence supports an intended interpretation and use. A stable score can still reflect irrelevant vocabulary or a poorly chosen reference population. The linked introduction discusses intelligence measurement and its history; the fictional scale here is not a diagnostic instrument.
Let x be the score, m the reference mean and s the supplied positive standard deviation. The standardized distance is z = (x − m)/s. Read it as ‘how far above or below this reference mean, measured in this reference group’s standard deviations.’ For A, (70 − 50)/10 = 2. For B, (70 − 65)/5 = 1. The raw score is unchanged while its relative position changes.
Do not convert z directly to a percentile without a distribution model or observed ranking. Two standard deviations above a mean does not mean 200% correct. If s were zero, division would be undefined; identical reference scores provide no nonzero spread for this transformation. Explain that edge case instead of displaying an infinite ability.
The repeat-test record supports one kind of consistency under those conditions. It does not establish language equivalence, comprehensive coverage of abilities or suitability for consequential decisions. A useful validation proposal compares task content with the intended construct, examines relevant outcomes and investigates whether procedures function appropriately across the intended population.
For the artifact, write a revised report using only the supported interpretation: ‘70 is two supplied standard deviations above reference A’s mean on this invented puzzle scale.’ Add what is needed before using it to guide instruction. The history of psychological testing includes harmful overreach; the practical lesson is to question the interpretation and purpose, not to rank classmates or communities.
Inspect the relationships
| Reference | Score | Mean | Standard deviation | Standardized distance |
|---|
| A | 70 | 50 | 10 | 2 |
| B | 70 | 65 | 5 | 1 |
Interpretation task: Explain how the last column can change while the score does not.
Worked example
The same score yields z = 2 against A and z = 1 against B. Consistent retest results do not prove the author’s universal language or general-ability claims.
Independent application
A scale always adds five points to every result. Which conclusion is justified? Explain the denominator, process or rule before checking the response. Record a revision after feedback.
Reading
OpenStax Psychology 2e, measures of intelligence. The numerical records and tasks are original fictional teaching examples; they are not findings reported by this source.
Investigation: does a repeatable score justify this use?
A fictional museum wants to find out whether its diagram instructions need improvement. Its worksheet asks visitors to match a pattern after rotation. The proposed interpretation is performance on these spatial matching tasks with accessible instructions. A reviewer instead writes, “This is a stable measure of general intelligence, and the lower-scoring group has less ability.” Your job is to audit that leap, not to assess anyone’s intelligence.
A construct is the attribute an assessment is intended to represent. Construct-irrelevant variation comes from something outside that intended attribute, such as specialized wording when vocabulary is not the goal. Construct underrepresentation means important parts of the intended attribute are missing: one rotation worksheet cannot sample every aspect of general intelligence. Neither concern is fixed merely by repeating the same test.
The jointly published 2014 Standards for Educational and Psychological Testing, Standards 1.0–1.1, require evidence for a specified score interpretation and use, with an identified population and construct. Chapter 3 addresses barriers unrelated to that construct. This is professional guidance, not a finding from the fictional museum. AERA, APA and NCME Standards, PDF pages 33 and 65–74.
First audit: what exactly repeated?
Twenty fictional visitors repeat one worksheet. Rows show the first classification; columns show the second. “Meets rule” means at least four of five items correct, a classroom rule with no established external validity.
| First occasion | Second: meets rule | Second: does not meet rule | Row total |
|---|
| Meets rule | 8 | 1 | 9 |
| Does not meet rule | 1 | 10 | 11 |
| Column total | 9 | 11 | 20 |
Try: Calculate exact classification agreement. Then count how many people changed category even though both occasions had nine people meeting the rule. Explain why identical group totals are insufficient evidence of individual stability. Write a sentence identifying what this table does not reveal about item scores or a general trait.
Check the classification agreement
Agreement is (8 + 10)/20 = 90%. Two people, or 10%, changed classifications in opposite directions. The marginal count of nine hides those changes. This is raw agreement for a binary classroom classification, not a test–retest correlation of 0.90, a measure corrected for chance, or proof of validity. Raw item scores, time interval, practice effects and the rule’s rationale remain unknown.
Second audit: does the intended construct change your interpretation?
A separate fictional pilot has 20 visitors familiar with geometry vocabulary and 20 without that course experience. These are prior-experience groups, not demographic identities or ability categories. Everyone attempts two five-item forms; form order is balanced within each group, with ten seeing each form first. The forms use different, intended-to-be-comparable diagrams and the same four-of-five rule. No participant is missing. All counts are invented.
- Technical form: “Select the configuration congruent under rotation.”
- Plain form: “Choose the matching arrangement after turning it, without stretching or flipping it.”
The wording intends to express the same operation. Whether the diagrams and instructions actually have equal difficulty requires review; “intended comparable” is not evidence of equivalence.
| Prior vocabulary experience | Technical form: meets rule | Plain form: meets rule |
|---|
| Familiar | 16/20 | 15/20 |
| Unfamiliar | 8/20 | 14/20 |
Predict before feedback: Which comparison matters if the purpose is spatial matching? Would the same language demand be irrelevant if the purpose were understanding geometry terminology? Copy the table, convert all four fractions to percentages, and calculate familiar-minus-unfamiliar differences separately for each form. Do not average away the interaction.
Check the wording contrasts
Technical-form rates are 80% and 40%, a 40-percentage-point difference. Plain-form rates are 75% and 70%, a 5-point difference. The group gap is 35 points smaller on the plain form. Within the unfamiliar group, the rate rises 30 points; within the familiar group it falls 5 points. These are descriptive contrasts, not standardized effect sizes or estimates with supplied uncertainty. Because the same people take both forms, the observations are paired; the margins alone do not show which people changed, just as the earlier retest margins did not.
For spatial matching, unfamiliar terminology is a plausible irrelevant barrier. For a test of geometry terminology, understanding that language is part of the target. Changing the stated purpose changes the evidence needed; it does not change the observed counts. A group gap alone does not prove item bias, and removing a gap alone does not prove fairness. Prior experience was not randomly assigned. Different diagrams, order effects and other group differences remain possible explanations. These aggregate counts do not establish differential item functioning, which requires investigating item performance while accounting for the relevant proficiency.
Build a validity argument
Replace the reviewer’s claim with a four-row evidence ledger. For each proposed inference, mark supported by these records, plausible but untested, or unsupported, then supply a reason and the next evidence needed.
| Proposed inference | Evidence to examine |
|---|
| The binary rule usually gives the same result on two occasions | Paired classification table; distinguish agreement from score reliability |
| Technical wording contributes to the observed gap | Item/form equivalence, response explanations, counterbalanced order and matched response records |
| The worksheet measures general intelligence | Breadth of tasks, theory and evidence relating the proposed construct to scores; the supplied worksheet is insufficient |
| The worksheet can improve museum instructions | Evidence that revisions improve comprehension and usable task performance in the intended visitor population |
High-school work may use this ledger directly. AP work should name the construct and explain two specific threats. College work should propose matched item-level analysis and a validation design, including uncertainty and how the intended use constrains interpretation. Do not turn a small pilot into a rule for ranking or excluding visitors.
Unfamiliar transfer: what did the checklist measure?
A fictional archive checklist aims to measure whether volunteers can distinguish a primary document from a later commentary. One version requires typing a paragraph in 30 seconds. A revised version permits an untimed spoken explanation. In different groups of 15, success changes from 6 to 12. The evaluator claims that speaking doubles historical understanding.
Write a response before opening feedback. Calculate the rates, separate relative change from percentage-point change, identify the changed task demands, and propose a design that tests the intended historical reasoning rather than speed or typing. If the target instead included rapid written reporting, explain why the redesign might change the construct itself.
Check the archive-checklist transfer
The rates are 40% and 80%: a 40-percentage-point difference and a 100% relative increase. That is not a measured doubling of understanding. Timing, response mode and participants all change, so the contrast cannot isolate speaking. Use comparable source cases, a common reasoning rubric and a design separating timing from response mode. Examine who was sampled, provide accessible instructions and keep intended interpretation explicit. No format is automatically valid for every use.
Save your classification calculation, four-cell contrasts, evidence ledger and archive response in the Experiment stage. This source-analysis activity does not administer an intelligence test or award independently reviewed mastery.
