Card 14

📊 🔄 🤯

Drug A beats Drug B in treating men. Drug A beats Drug B in treating women. How can Drug B be better overall?

💭 How to Think About This

This sounds impossible! If A beats B for men AND A beats B for women, A must beat B overall... right? WRONG. This counterintuitive phenomenon is called SIMPSON'S PARADOX, and it reveals how aggregating data can completely reverse conclusions. It's not a trick—it's real statistics.

Can the overall trend really reverse when you combine groups?

🔒 Write your thinking above to unlock hints

This reversal IS mathematically possible! The key is unequal group sizes. If Drug A treats mostly easy patients and Drug B treats mostly hard patients, A can win in each subgroup but lose overall. The groups aren't comparable!

Drug A: Men 80/100 (80%), Women 15/20 (75%) = 95/120 (79%). Drug B: Men 75/80 (94%!), Women 200/300 (67%) = 275/380 (72%). B beats A in BOTH groups but loses overall! B treated mostly women (harder cases), dragging down its average.

A LURKING VARIABLE (like disease severity) affects both: (1) Which treatment you get, and (2) Your outcome. If harder cases go to Drug B, B's results look worse even if it's actually better. Aggregation hides this crucial factor!

UC Berkeley was sued for gender bias: men admitted at higher rates. But by department, women were admitted at EQUAL or HIGHER rates! Women just applied to more competitive departments. The "discrimination" was Simpson's Paradox, not bias!

Yes, it's possible! This is Simpson's Paradox—trends can reverse when data is aggregated!

It seems impossible, but the math is real.

How it works:

• Drug B treats more patients from the harder-to-treat group

• This drags down B's overall average

• Even though B wins in each subgroup!

The mechanism:

• A lurking variable affects both treatment assignment AND outcomes

• Aggregating data hides this variable

• The aggregate reverses the subgroup trends

Real examples:

• UC Berkeley admissions "bias"

• Baseball batting averages

• Medical treatment effectiveness

Your protection: When comparing groups, always ask: "Is there a lurking variable that makes these groups not actually comparable?"

You spotted the key insight! Unequal group sizes can cause this reversal. If Drug A treats mostly easy patients and Drug B treats mostly hard patients, B's overall average gets dragged down even if it's better within each group.

Drug A: Easy cases 90/100 (90%), Hard cases 10/20 (50%) = 100/120 (83%). Drug B: Easy cases 18/20 (90%), Hard cases 160/200 (80%) = 178/220 (81%). A wins overall, but B beats or ties A in both subgroups! The paradox is real.

Simpson's Paradox appears constantly: • School comparisons (different student populations) • Hospital quality metrics (different patient severity) • Hiring/admission statistics. Simple aggregated comparisons can tell completely wrong stories!

To avoid being fooled: (1) Always ask "Are these groups comparable?"; (2) Look for lurking variables; (3) Disaggregate by important factors; (4) Compare apples to apples. The truth is usually in the stratified data, not the aggregate!

Correct! Unequal group sizes can reverse the trend—this is Simpson's Paradox!

You showed excellent statistical thinking by recognizing this counterintuitive possibility.

The mechanism:

• Treatment B takes on harder cases

• B wins within each difficulty level

• But B's average is dragged down by harder cases

• Aggregate comparison is misleading

Famous real example (UC Berkeley):

• Overall: Men admitted at higher rate

• By department: Women admitted at equal/higher rates

• Why? Women applied to more competitive departments

Your detection skill:

• "Are the groups being compared actually comparable?"

• "Would disaggregating tell a different story?"

• "Is there a lurking variable affecting selection?"

This skill protects you from misleading statistics everywhere.

You're right that a hidden factor is involved! It's called a LURKING VARIABLE—something that affects both which treatment you get AND your outcome. If harder cases go to Drug B, B's overall results look worse even if B is actually better!

The hidden factor creates unequal group sizes: • Drug A: 90% easy cases, 10% hard cases • Drug B: 10% easy cases, 90% hard cases. Even if B beats A in BOTH categories, A's overall percentage is higher because A mostly treats easy cases!

UC Berkeley was accused of discriminating against women (lower admission rate). But by department, women were admitted at equal or higher rates! The lurking variable: women applied to more competitive departments. Aggregation created a false appearance of bias.

Your instinct to look for hidden factors is exactly right! Always ask: "Is there something that affects both who's in each group AND the outcome?" If yes, the aggregate comparison may be completely misleading. Disaggregate to find truth!

Yes! The hidden factor (lurking variable) is exactly what causes Simpson's Paradox!

Your instinct to look for hidden factors is statistically sophisticated.

How it works:

• A lurking variable affects treatment assignment

• The same variable affects outcomes

• This creates unequal group compositions

• Aggregation hides the lurking variable → reversal!

Examples of lurking variables:

• Disease severity in drug trials

• Department competitiveness in admissions

• Socioeconomic status in school comparisons

Your protection:

• "What hidden factor might affect both selection and outcome?"

• "Are these groups actually comparable?"

• "What does disaggregated data show?"

This detective work reveals the true story behind misleading aggregates.

🤔 Which thinking lens(es) did you use?

Select all the lenses you used:

Learn more about the 7 Thinking Lenses →

👨‍👩‍👧 For Parents & Teachers

🌱 A Small Everyday Story

School A has higher average scores than School B.
"School A is better!" parents conclude.
But School A has mostly kids from rich families.
Look closer: among rich kids, B beats A.
Among poor kids, B beats A.
A only "wins" because it has easier students.
B is actually the better school.

See more guidance →

🧠 Thinking habits this builds:

Questioning aggregate statistics by looking for lurking variables
Understanding that trends can reverse when disaggregated
Asking "are these groups actually comparable?"
Recognizing that simple comparisons can be deeply misleading

🌿 Behaviors you may notice (and reinforce):

"But what if we break it down by...?" questions
Skepticism about simple comparisons between unequal groups
Looking for lurking variables that affect selection
Understanding that "fair" comparisons require comparable groups

How to reinforce: When encountering comparisons, ask together: "Are we comparing apples to apples? What might be different about these groups besides the thing we're measuring?"

🔄 When ideas are still forming:

Some learners may become paralyzed, thinking you can never trust any comparison. Help them see that the solution is to ASK about lurking variables and disaggregate when needed—not to distrust all data.

Helpful response: "Good comparisons control for lurking variables. When we break down by the right factors, the truth emerges. What factors should we check?"

🔬 If you want to go deeper:

Study the original Berkeley admissions case in detail
Explore how clinical trials control for lurking variables
Look at Simpson's Paradox in baseball statistics

Key concepts (for adults): Simpson's Paradox, lurking variable, confounding variable, disaggregation, ecological fallacy, stratification, controlling for variables.

Drug A beats Drug B in treating men. Drug A beats Drug B in treating women. How can Drug B be better overall?

🔄 Compare All Perspectives

❌ "Mathematically impossible"

✅ "Yes, if sizes differ"

🤔 "Hidden factor involved"

🤔 Which thinking lens(es) did you use?