Does the treatment work, or did the groups differ?
Can an apparent benefit reverse while the treatment itself stays unchanged?
Start with a concrete question
Original synthetic study: a fictional practice activity adds the same number of score points to two groups with different starting scores.
Try the initial inputs, then write a prediction before changing one setting. All calculations run locally on your CPU.
What do you expect?
Predict whether the headline difference will equal the true effect when 80% of treated participants start in the higher-baseline group.
Change one thing. Inspect what happens.
- Keep effect at 5 and record both means at 80% composition.
- Change composition to 20%. Explain why the unadjusted difference changes sign.
- Find the composition that recovers the programmed effect. Then set effect to zero and construct a misleading positive headline.
- Write a comparison plan that balances baseline groups and states one additional real-world confounder you would need to measure.
Your settingsMechanism
Calculation belowEvidence
Compare outcomes
Compare the current results
Every bar starts at zero and uses the same numeric scale in this chart. The upper bound automatically adjusts to the largest current value (at least 1), so compare the numbers when changing inputs. Units and quantities come from each label; different kinds of quantity are not interchangeable. Displayed values are rounded to at most six significant digits.Follow the calculation
- High-baseline participants score 80; low-baseline participants score 40 before treatment.
- Treated: 0.80 × 85 + 0.20 × 45 = 77.00.
- Comparison: 0.20 × 80 + 0.80 × 40 = 48.00.
- Within either group the change is 5. Weight both groups equally: ((85 - 80) + (45 - 40)) / 2 = 5.
Group composition changes the headline difference even though the within-group effect is unchanged.
Captures keep your inputs, prediction and observation. Results are recalculated from those inputs. Resetting inputs keeps your captures and challenge history.
Explain the mechanism
What the experiment explains
Original synthetic study: a fictional practice activity adds the same number of score points to two groups with different starting scores.
A confounder influences both which group receives an intervention and the measured outcome. Here starting score is a deliberately visible confounder.
At the defaults, treated mean = 0.8 × 85 + 0.2 × 45 = 77. Comparison mean = 0.2 × 80 + 0.8 × 40 = 48. The observed 29-point gap is not the programmed 5-point effect.
Comparing within each starting group and using equal weights removes this specific imbalance. In a real study, unmeasured differences can remain; random assignment helps address them but is not simulated here.
Read the relationship
Mean = high-group share × high-group score + low-group share × low-group score. Difference = treated mean − comparison mean; all scores are points.
Use the trace as evidence
The trace recomputes from your current controls. Change one control at a time, record what changed, and compare the result with your prediction. The bars use the same quantities as the numerical trace; a larger bar is not automatically a better outcome.
Boundaries of the model
- Deterministic group averages, no sampling variation, equal group sizes, and a constant additive effect.
- Equal weighting fixes only the modeled composition difference, not every threat to causal inference.
Continue with Why more measurements can still be wrong to investigate a different mechanism.
Try a fresh case
New study: high baseline 70, low baseline 30, effect +4. Treated group is 75% high baseline; comparison is 25% high baseline. What is the unadjusted mean difference in points?
This is a public practice check. Retries support learning; matching a number does not certify independent proficiency.
Take the investigation further
Choose any extension. These are project prompts for you to carry out; they are not automatically assessed.
School investigation
Design a fair study of two revision methods; identify what you would balance before seeing results.
College investigation
Implement stratified and pooled estimates, then add unequal group sizes and sampling variability.
Workplace investigation
Draft a rollout experiment with assignment rules, a primary metric, and a plan for baseline imbalance.
Keep the evidence, choose your next step
Completion records your own decision that you have worked through this investigation. Practice checks and independent project evidence are separate.