Comparing groups

Research Skills — Week 7

Recap

Recap

• Why not just do lots of t-tests?
• How many tests?
• ANOVA
• Run an ANOVA
• Effect sizes
• HolmesCo’s report card
• Wrap-up

Where we are

Last week: t-tests for two groups. The base rate fallacy.

This week: what happens when you have more than two groups — and more than one test.

w01-ppdac-ring.R

• [11:05–11:10] Quick recap. Any lingering questions from last week’s base rate discussion?
• Point at the ring: where this week sits in PPDAC (Week 1, “The cycle”).

Questions?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

Why not just do lots of t-tests?

• Recap

Why not just do lots of t-tests?

🎓 Concept block 1

• How many tests?
• ANOVA
• Run an ANOVA
• Effect sizes
• HolmesCo’s report card
• Wrap-up

The problem

Three groups (e.g., wind farms in three regions).

You could run three pairwise t-tests: A vs B, A vs C, B vs C.

Each test has a 5% false positive rate.

Three tests: P(at least one false positive) = 1 − 0.95³ ≈ 14%

Ten tests: 40%

Twenty tests: 64%

• [11:10–11:30] Multiple comparisons. Build to the HolmesCo mineral survey.

The escalation

Number of tests	P(≥1 false positive)
1	5%
3	14%
10	40%
20	64%
100	99.4%

Run enough tests and you’re guaranteed to “find” something.

HolmesCo’s mineral survey

HolmesCo

“Geological Solutions Since 2019”

Press Release — Multi-Element Anomaly Discovered!

“We tested soil samples from 20 sites for enrichment in 20 elements. We found statistically significant enrichment (p < 0.05) for three elements at one site!”

If there’s truly nothing there, how many significant results would you expect by chance?

• 20 elements × 20 sites = 400 tests × 0.05 = 20 expected false positives. Three hits is fewer than expected by chance. HolmesCo’s “discovery” actually underperforms random noise.

The answer

20 × 20 = 400 tests at α = 0.05

Expected false positives: 20

HolmesCo found 3.

That’s fewer than expected by chance. The “discovery” is noise.

• Let this land. The subversion — that the result is actually less than chance — makes the multiple comparisons problem viscerally clear.

How many tests?

• Recap
• Why not just do lots of t-tests?

How many tests?

💬 Exercise 1

• ANOVA
• Run an ANOVA
• Effect sizes
• HolmesCo’s report card
• Wrap-up

Your project

How many comparisons are you planning to make?

If you run all of them at α = 0.05, what’s your family-wise false positive rate?

Formula: 1 − (1 − α)^k where k = number of tests

• [11:30–11:37] Students calculate for their own projects. Then discuss: “If you’re testing more than one thing, you need to account for it.” (Shortened from 10 to make room for “Same data, four ways” below.)

ANOVA

• Recap
• Why not just do lots of t-tests?
• How many tests?

ANOVA

🎓💻 Concept block 2

• Run an ANOVA
• Effect sizes
• HolmesCo’s report card
• Wrap-up

One test for multiple groups

ANOVA (Analysis of Variance) asks:

“Does any group differ from the others?”

Without specifying which.

• [11:37–12:02] “Same data, four ways” (~5 min), ANOVA mechanics, live demo, post-hoc tests.

The logic

1. Total variation = between groups + within groups
2. If between-group variation is large relative to within-group → at least one group is different
3. The F-statistic captures this ratio
4. The p-value tells you how surprising this F is under the null

Same data, four ways

Before any test: how you show a comparison changes what you see.

Four regions, same 100 sites, same order (low mean to high) — shown four different ways.

• [11:37–11:42, inside concept block 2] A quick sequence, no discussion prompts needed on each panel — let the four builds speak for themselves, then land the takeaway on the last one.
• Idea adapted from Andrew Heiss, Data Visualization with R, session 6 (“Uncertainty”) and session 8 (“Comparisons”) — After Heiss (2026).
• Timing note: this concept block grew from 20 to 25 minutes to fit the sequence; the preceding “How many tests?” exercise and the “Run an ANOVA” exercise after it were each trimmed to compensate. See sessions/weeks-5-10.md.

Same data, four ways: the bar

w07-groups-bar.R

• The “dynamite plot.” Ask: would you trust this? What would change your mind?
• What it hides: the spread within each region, the sample size, and whether the gap between any two bars is bigger than the noise.
• Region order here is by group mean. The exercise page’s Exercise 1 orders its violins by median, which swaps Wales and Southern England — worth a one-line heads-up if a student notices the mismatch.

Same data, four ways: the box plot

w07-groups-box.R

• Better: the spread is visible, and the two-cluster pattern starts to show. Still hides the individual sites and the sample size per box.
• If someone asks “why not just use this” — flag Nick Desbarats’ Nightingale piece, linked on the exercise page: box plots take longer to teach than most students realize, and a good fraction of any room cannot read one confidently.

Same data, four ways: the raw points

w07-groups-points.R

Takeaway: a bar of means can hide a bad idea; the points behind it usually don’t.

• This is the picture ANOVA reasons about: a mean, its uncertainty, and the scatter behind it. Compare it to the F-ratio figure two slides later — same encoding, cyan points and an ink summary.
• Point out that individual sites overlap heavily between regions even though the means and CIs are cleanly separated. Both things are true at once.

Same data, four ways: the raincloud

w07-groups-raincloud.R · after Heiss (2026)

• Optional coda, after the takeaway on the previous slide: shape, sites and interval on one axis. If the block is running long, cut this slide and move straight to “Relative to within.”
• Ask what this shows that panel 3 didn’t. Northern England’s cloud has two bumps: perhaps two kinds of site. No mean, box or interval would reveal that.
• The cloud is drawn with ggdist; the interval is the same hand-computed mean ± 95% CI as the previous slide.
• Further reading (Nick Desbarats, “I’ve Stopped Using Box Plots. Should You?”, Nightingale, and Claus Wilke’s “Visualizing uncertainty”) is linked on the exercise page — don’t read it out.

“Relative to within”

w07-f-ratio.R

• Check the mean bars against each other across the panels: same height. The between-group part of the ratio is held fixed by construction — the script scales one set of residuals rather than drawing twice.
• Both F values come from aov() on the panel’s own points.
• Ask which panel you would be willing to call a real regional difference, before showing any p-value.

Live demo

If significant → which pairs differ?

• Type this live, don’t paste it. Students have wind_output.csv on their own exercise page — tell them to race you to the F value.

• Anticipated code:

  wind <- read.csv("data/wind_output.csv")
  
  model <- aov(capacity_pct ~ region, data = wind)
  summary(model)
  
  TukeyHSD(model)

• The slide previously showed aov(output ~ region, data = wind_df) — neither the object nor the column exists. The real data is wind$capacity_pct by wind$region, matching exercises/week7-content.qmd.

• Numbers: F(3, 96) = 14.21, p = 9.76e-08. Tukey’s adjusted p-values give two clusters — Scotland vs Northern England is the only non-significant pair (p = 0.994) and Wales vs Southern England the only other one above 0.05 (p = 0.673); the four north-vs-south comparisons are all significant.

• Read the pairwise comparisons out loud. Point out the adjusted p-values: that adjustment is the whole reason we are not running six separate t-tests.

Corrections for multiple comparisons

Method	How	When
Tukey’s HSD	All pairwise, designed for ANOVA	After a significant ANOVA
Bonferroni	Divide α by number of tests	Simple, conservative, any context

Both reduce the false positive rate by being stricter about what counts as significant.

• Don’t go deep into the math. The key idea: when you test multiple things, you raise the bar for each individual test.

Run an ANOVA

• Recap
• Why not just do lots of t-tests?
• How many tests?
• ANOVA

Run an ANOVA

✏️💻 Exercise 2

• Effect sizes
• HolmesCo’s report card
• Wrap-up

In WebR

Provided dataset: wind farm output in four regions.

1. Visualize with violins
2. Run aov()
3. Is it significant?
4. Run TukeyHSD() — which pairs differ?
5. Write a one-sentence conclusion

• [12:02–12:15] Hands-on. The WebR exercise page has scaffolding. (Shortened from 15 to make room for “Same data, four ways” above.)

Attendance

Show the QR from https://attendance.durham.ac.uk/mytimetable, then ← back to the task.

Effect sizes

• Recap
• Why not just do lots of t-tests?
• How many tests?
• ANOVA
• Run an ANOVA

Effect sizes

🎓 Concept block 3

• HolmesCo’s report card
• Wrap-up

“Significant” ≠ “important”

A p-value tells you whether an effect is detectable.

It does not tell you whether it’s big enough to matter.

• [12:15–12:35] Effect sizes — the statistical formalization of “Is that a big number?”

The HolmesCo temperature problem

HolmesCo

“Geological Solutions Since 2019”

Two boreholes, 1,000 readings each.

Mean difference: 0.02°C. p-value: 0.001.

“Highly significant temperature anomaly detected!”

Is 0.02°C meaningful for any practical purpose?

• With enough data, trivially small differences become “significant.” The p-value says “detectable,” not “important.”

Cohen’s d

A standardized measure of how far apart two groups are:

d	Interpretation
0.2	Small
0.5	Medium
0.8	Large

effectsize::cohens_d(y ~ group, data = df)

• d = (mean₁ − mean₂) / pooled SD. It answers “how many standard deviations apart are the groups?”

What those numbers look like

w07-cohens-d.R

• The point is the right-hand panel. “Large” still leaves two groups overlapping across most of their range — so a significant result with d = 0.8 does not mean you can tell an individual’s group from their measurement.
• The shared percentage is integrated from the curves, not asserted.

For ANOVA: η² (eta-squared)

Proportion of variance explained by the grouping variable.

η²	Interpretation
0.01	Small
0.06	Medium
0.14	Large

Always report both the p-value and an effect size.

A policy-maker needs to know “how big?” — not just “is it real?”

• Link to “Is that a big number?” from Week 2. Same habit, formalized.

HolmesCo’s report card

• Recap
• Why not just do lots of t-tests?
• How many tests?
• ANOVA
• Run an ANOVA
• Effect sizes

HolmesCo’s report card

✏️💬 Integrative exercise

• Wrap-up

Three test results

HolmesCo

“Geological Solutions Since 2019”

Technical Summary

1. Test A: p = 0.002, Cohen’s d = 1.2
2. Test B: p = 0.04, Cohen’s d = 0.08
3. Test C: p = 0.08, 95% CI = [−0.3, 6.1]

For each: should we act on this? Why or why not?

• [12:35–12:50] A is significant with a large effect — act. B is significant but trivially small — maybe not. C is non-significant but the CI almost excludes zero — worth investigating further.

Wrap-up

• Recap
• Why not just do lots of t-tests?
• How many tests?
• ANOVA
• Run an ANOVA
• Effect sizes
• HolmesCo’s report card

Wrap-up

Key points

1. Multiple tests inflate false positives — correct for them
2. ANOVA compares multiple groups in one test; TukeyHSD finds which differ
3. Significance ≠ importance — always report effect sizes
4. “Is that a big number?” now has a formal answer

Any questions we missed?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Continuation of the same Poll Everywhere Q&A activity opened at the start of the session — this is not a new poll, so don’t reset/archive it yet.
• Surface any questions that have come in since the early Questions slide, especially ones raised during today’s content.
• Once you’ve finished with this slide, archive (reset) the activity at the end of the session so questions don’t carry over.

Next time

Application session: “Deepening your analysis”

Apply ANOVA or refine your t-test with effect sizes and assumption checks.

• Check the Q&A — surface any top-voted questions before finishing.
• Not all groups need ANOVA. Groups with two-group comparisons should deepen their t-test analysis with CIs and effect sizes.