Testing hypotheses

Research Skills — Week 6

Recap

Recap

• The null world
• Interpret a t-test
• Assumptions
• Check your assumptions
• The base rate fallacy
• Plan your test
• Wrap-up

Where we are

• Week 3: the logic of hypothesis testing (conceptual)
• Week 5: experimental design, controls, confounders

This week: the formal machinery. How do you tell a real difference from a fluke?

And then you’ll learn why the answer can mislead you.

w01-ppdac-ring.R

• [11:05–11:10] Frame the session: first half is mechanics, second half is the most important lesson of the course.
• Point at the ring: where this week sits in PPDAC (Week 1, “The cycle”).

Questions?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

The null world

• Recap

The null world

🎓💻 Concept block 1

• Interpret a t-test
• Assumptions
• Check your assumptions
• The base rate fallacy
• Plan your test
• Wrap-up

Is the difference real?

Boreholes in the Stainmore Formation run hotter, on average, than boreholes in the Whin Sill.

“Is this difference larger than we’d expect by chance?”

The Whin Sill at Holwick Scars, Teesdale.
Gordon Hatton / Geograph, CC BY-SA 2.0

• [11:10–11:42] The null world by simulation, then the t-test as its shortcut, each with a live demo.
• Don’t say “t-test” yet. The whole block builds one idea, the null world, and only names the formula once students own the idea. The reveal is “This is a t-test”, after the null-world demo.

Five steps, every test

1. Measure δ — the one number you care about. Here: Stainmore’s mean temperature minus Whin Sill’s.
2. Build a null world — a world where δ really is zero. Simulate it: shuffle the formation labels, recompute δ, repeat.
3. Drop your δ into it. Does it look at home?
4. Count how often the null world makes a δ at least as extreme. That fraction is the p-value.
5. Decide, against a standard you set before you looked.

• Steps 2–4 are the heart of it. Today we do them by brute force, by shuffling; the named tests come later and only replace those steps with a formula.
• The idea that every test is this one test is Allen Downey’s (“There is only one test”); the five-step framing and the interactive version are Andrew Heiss’s Null worlds, https://nullworlds.andrewheiss.com/. Point students there — the sliders make the intuition stick.
• The whole method rests on one question: how surprising would this be if nothing were going on? Plant a flag here. In block 3 we will meet its partner — how plausible was this before we tested? — and the gap between the two is the base-rate fallacy. A null world only ever answers the first question.

A world where formation doesn’t matter

w06-null-world.R · After Heiss (2026), Null worlds

• Left is step 1: the data as sampled. δ = 8.4 °C.
• Right is step 2: keep every temperature exactly as measured, but deal the formation labels out again at random, 1,000 times. In every one of those worlds formation cannot matter — the labels are noise — so the spread of δ is what chance alone produces with these boreholes.
• Step 3: the dashed lines are our real δ, both signs, because the question is “this extreme”, not “this large”.
• Step 4: count the red. 26 of 1,000, so p = 0.026.
• Hold on to this picture: after the reveal, “What ‘how often’ means” shows a smooth curve with shaded tails. That curve is the formula’s version of this histogram.

Build the null world

• Type this, don’t paste it. Students have the same data; invite them to beat you to the p-value.

• Anticipated code:

  borehole <- read.csv("data/borehole_temp.csv")
  temp <- borehole$temperature_c
  
  delta <- mean(temp[borehole$formation == "Stainmore"]) -
    mean(temp[borehole$formation == "Whin Sill"])
  delta
  
  shuffled <- sample(borehole$formation)
  mean(temp[shuffled == "Stainmore"]) - mean(temp[shuffled == "Whin Sill"])
  
  set.seed(6)
  null_deltas <- replicate(1000, {
    shuffled <- sample(borehole$formation)
    mean(temp[shuffled == "Stainmore"]) - mean(temp[shuffled == "Whin Sill"])
  })
  
  ggplot(data.frame(null_deltas), aes(x = null_deltas)) +
    geom_histogram(bins = 30) +
    geom_vline(xintercept = c(-delta, delta), colour = "red")
  
  mean(abs(null_deltas) >= abs(delta))

• Numbers from a real run: δ = 8.417; p = 0.026 (26 of 1,000 shuffles). The null deltas run from about −11.4 to +11.1.

• Step 3 is unseeded on purpose: run it two or three times so the room sees each shuffle give a different δ, near zero. Then set.seed(6) immediately before replicate() — that is what the figure on the previous slide used, so the p-value matches it exactly. Put the seed anywhere else and the third decimal moves; that wobble is itself worth a sentence: a simulated p-value is an estimate, and 1,000 shuffles is plenty to tell 0.03 from 0.3.

• abs() twice makes it two-sided: “this extreme”, either way.

What would the null world be?

For each claim, how would you build a world where it is false?

1. Wind farms in Scotland run at a higher capacity factor than wind farms in Wales.
2. Boreholes drilled after 2000 read warmer than older ones at the same depth.
3. Monitoring wells downstream of a quarry have more sulphate than wells upstream.

• [3 min, within 11:10–11:42] Desks, one minute per claim. Ask for the shuffle, not a test name.
• 1: deal the region labels out again at random across the farms (wind_output.csv has 25 per region). Same move as the boreholes.
• 2: shuffle the drilling-date labels. Then ask: is date the only thing that changed? Newer probes, deeper holes, different sites — the shuffle answers “chance?”, not “why?” (Week 5’s confounders).
• 3: shuffle upstream/downstream. Wells near each other are not independent, which a shuffle quietly assumes. Good place to plant the word “assumption” for block 2.
• The point to drill: every one of these is the same five steps. Only δ and the labels change.

This is a t-test

A t-test asks the same question with a formula, instead of a shuffle:

1. Assume no difference: the null world (the null hypothesis)
2. Calculate how far apart the means are, relative to the variability (→ t-statistic)
3. Ask: if there were really no difference, how often would we see a t this extreme? (→ p-value)

• The reveal. Map each line back to the five steps: line 1 is step 2, lines 2–3 are steps 3–4.
• The payoff line: “Everything t.test() does is a shortcut for what we just did by hand.” Shuffling needs no normality assumption — but it still compares means, and Stainmore’s long tail drags its mean. Block 2 comes back to that, and it is not the test’s fault.
• The formula buys speed and a smooth curve; it costs assumptions about the shape of the data. Next demo: does it agree with our 0.026?

Live demo

• Type this, don’t paste it. Say out loud that the students have the same dataset open; invite them to get to the p-value first, and start typing a minute in.

• borehole is already loaded from the null-world demo (the cells on this deck share one R session), so step 1 is a one-liner or a skip.

• Anticipated code:

  borehole <- read.csv("data/borehole_temp.csv")
  
  ggplot(borehole, aes(x = formation, y = temperature_c)) +
  # geom_violin() +
    geom_point() 
  
  t.test(temperature_c ~ formation, data = borehole)

• ggplot2 is already attached on this slide. Points (with violins, if you like) are what exercises/week6-app.qmd Exercise 1 asks for.

• Show the output: t-statistic, df, p-value, confidence interval.

• Set it beside the shuffle: 0.030 from the formula, 0.026 from 1,000 shuffles. Same question, same answer, two routes.

• Interpret aloud: “The mean difference is 8.4°C, with a 95% CI of [0.9, 16.0]. The p-value is 0.03, meaning we’d see a difference this large about 3% of the time if there were truly no difference.”

• Exact values, so the next slide’s table matches: t = 2.2419, df = 47.179, p = 0.0297, CI [0.865, 15.969], means 35.30 (Stainmore) and 26.88 (Whin Sill).

Reading the output

t = 2.24, df = 47.2, p-value = 0.030
95% CI: [0.86, 15.97]

Element	Meaning
t-statistic	How many SEs apart the means are
p-value	How surprising, in the null world (where H₀ is true)
95% CI	Plausible range for the true difference

• Emphasize: the CI is often more useful than the p-value. It tells you the size of the effect, not just whether it exists.
• These are the real numbers from borehole_temp.csv, so they should match the live demo exactly. If they don’t, the data changed.

What “how often” means

w06-t-distribution.R

• Step 3 of “This is a t-test” was: if there were really no difference, how often would we see a t this extreme? That “how often” is an area, and this is it.
• Both tails, because the question was “this extreme”, not “this large”.
• t, df and the shaded fraction all come from the same t.test() the demo just ran.

Interpret a t-test

• Recap
• The null world

Interpret a t-test

💬✏️ Exercise 1

• Assumptions
• Check your assumptions
• The base rate fallacy
• Plan your test
• Wrap-up

Write one sentence

I’m showing you R output from a t-test.

Write one sentence interpreting this result for a policy audience.

Then we’ll compare.

• [11:42–11:50] Show pre-prepared output on screen. Students write, then share.
• Common mistake to watch for: “the p-value means there’s a 3% chance the null is true.” Flag this — it’s exactly the confusion block 3 will address.

A common mistake

“p = 0.03 means there’s a 3% chance the null is true.”

Wrong. But extremely common. We’ll come back to this.

• Plant the seed now. The base rate fallacy in block 3 will explain why this interpretation is wrong.

Assumptions

• Recap
• The null world
• Interpret a t-test

Assumptions

🎓 Concept block 2

• Check your assumptions
• The base rate fallacy
• Plan your test
• Wrap-up

Every test rests on assumptions

For the t-test:

1. Independence — observations don’t influence each other
2. Normality — data within each group are roughly normal
3. Equal variance — groups have similar spread (Student’s t; Welch’s t relaxes this)

• [11:50–12:05] Assumptions, how to check, what to do when violated. Keep “How to check” to a quick pass: name Levene’s test but don’t demo it. The skew figure is where the time should go.

How to check

Assumption	Check
Normality	Histogram, QQ plot, shapiro.test()
Equal variance	Side-by-side points or violins, Levene’s test
Independence	Think about the study design

When assumptions are violated

A skewed variable can produce misleading p-values.

w06-skew-and-logs.R

• This is the dataset from the live demo. The test they just ran is the one that breaks: p = 0.030 raw, p = 0.22 logged. Both p-values are computed by the script.
• Nothing was removed and no outlier was excluded. The only change is the scale the test was run on.
• Someone may ask: the shuffle test assumed nothing about normality and still gave 0.03 — so was the t-test really broken? Shuffle the log temperatures and you get p ≈ 0.22, the same as the logged t-test. Both machines agree on whichever δ you hand them; the transform changes the question (typical ratio, not mean difference in °C), and with a tail like Stainmore’s the mean is the wrong question.

HolmesCo made the same mistake

HolmesCo

“Geological Solutions Since 2019”

Statistical Analysis — Soil Permeability

t-test on raw values: p = 0.04 → “Significant difference!”

t-test on log-transformed values: p = 0.23 → “No significant difference.”

The “significant” result was an artefact of skewness.

• This is a vivid example. HolmesCo tested permeability (always log-normal) without transforming. The p-value changed from 0.04 to 0.23 — the “discovery” vanished.

Check your assumptions

• Recap
• The null world
• Interpret a t-test
• Assumptions

Check your assumptions

✏️💻 Exercise 2

• The base rate fallacy
• Plan your test
• Wrap-up

In WebR

Given a dataset:

1. Make histograms of each group
2. Decide whether to transform
3. Run t.test() before and after transformation
4. Compare: did the conclusion change?

• [12:05–12:15] Quick hands-on exercise. The WebR page has the scaffolded version with hints.

Attendance

Show the QR from https://attendance.durham.ac.uk/mytimetable, then ← back to the task.

The base rate fallacy

• Recap
• The null world
• Interpret a t-test
• Assumptions
• Check your assumptions

The base rate fallacy

🎓💬 Concept block 3

• Plan your test
• Wrap-up

The most important 25 minutes of the course

I’m going to show you why a statistically significant result can still be wrong most of the time.

• [12:15–12:40] Take your time. This is the intellectual centrepiece of the module. Let the numbers sink in.

Warm-up: Monty Hall

Some of you simulated the Monty Hall problem in first-year Python.

Switching wins 2/3 of the time. But your gut says 50/50.

Why? Because your gut ignores the information the host gave you when they opened a door.

That confusion — between P(win) and P(win | what you now know) — is exactly what this block is about.

• Brief connection. Don’t re-derive Monty Hall. The point: conditional probability is unintuitive, and this has serious consequences.

The medical test

A disease affects 1 in 1,000 people.

A test is 99% accurate (99% sensitivity, 99% specificity).

You test positive.

What’s the probability you have the disease?

Cast your vote

PollEv.com/geol

text geol to 07480 781235

• Activate the poll before showing this slide. Use a numeric open-ended response so answers appear as a histogram.
• Most students will cluster around 95–99%. A few may guess lower. The spread makes the reveal (“it’s 9%”) land much harder — they can see on screen that almost everyone got it wrong.
• Wait until responses plateau, then advance to the arithmetic.

The arithmetic

Out of 100,000 people:

	Has disease	No disease	Total
Test positive	99	999	1,098
Test negative	1	98,901	98,902
Total	100	99,900	100,000

P(disease | positive) = 99 / 1,098 ≈ 9%

Not 99%. Nine percent.

• Let this land. Pause. “Why is it so low?”

Why is it so low?

Because the disease is rare.

The base rate matters enormously.

w06-base-rate.R

Even a very accurate test produces many false positives when the condition is uncommon.

• The 9% from the previous slide is the marked point. Same test, same 99% accuracy, everywhere on this curve — only the population changes.
• Ask where they would have to move along the axis to get the 99% they guessed in the poll. It is a condition affecting most of the room.

HolmesCo strikes gold

HolmesCo

“Geological Solutions Since 2019”

Press Release — Major Gold Discovery in County Durham!

“Our geochemical assay (95% accuracy) detected gold in 50 out of 10,000 soil samples. Confirmed multi-element anomaly!”

• Work through the numbers on the board or next slide.

The reality

Suppose 5 in 10,000 samples are actually gold-bearing.

	Gold present	No gold	Total
Assay positive	~5	~500	~505
Assay negative	~0	~9,495	~9,495

HolmesCo’s 50 positives? Almost certainly all false.

The press release is nonsense.

• The 50 “hits” is actually fewer than the ~500 expected false positives. If anything, the assay is underperforming.
• This is the Week 6 equivalent of the traitor reveal.

Connection to p-values

A p-value tells you: P(data | H₀)

How surprising is this data if the null is true?

What you actually want: P(H₀ | data)

How likely is the null to be true given this data?

These are not the same thing. The difference depends on the base rate.

• Call back to the null world: every p-value today answered “how surprising would this be if nothing were going on?” The refrain on the next slide is the question the null world cannot answer.
• Key sentence: “If you’re testing an implausible hypothesis, a significant p-value is probably a false positive. If you’re testing something well-supported, a significant result is more convincing.”

The new refrain

“How plausible was this before we tested?”

• This is the Bayesian insight without the formula. The habit of asking about prior plausibility is what matters.
• This also sets up Week 7: if you run 20 tests, the base rate for any individual hypothesis being true is low → false positives accumulate.

Plan your test

• Recap
• The null world
• Interpret a t-test
• Assumptions
• Check your assumptions
• The base rate fallacy

Plan your test

✏️ Integrative exercise

• Wrap-up

For your project

Discuss with your group:

1. What comparison will you test?
2. What’s your null hypothesis?
3. What’s your prior expectation — plausible or a long shot?
4. What assumptions should you check?

Write 3–4 sentences.

• [12:40–12:50] This feeds directly into the application session.

Wrap-up

• Recap
• The null world
• Interpret a t-test
• Assumptions
• Check your assumptions
• The base rate fallacy
• Plan your test

Wrap-up

Key points

1. A t-test compares two groups — but check the assumptions first
2. A p-value is P(data | H₀), not P(H₀ | data)
3. The base rate determines whether a “significant” result is real
4. Always ask: “How plausible was this before I tested?”

Exit ticket

A t-test comparing two groups returns p = 0.04. Which statement is correct?

PollEv.com/geol

text geol to 07480 781235

• Correct answer: B — P(data | H₀), not P(H₀ | data).
• A is the classic confusion of P(data | H₀) with P(H₀ | data). Very common, even among experienced researchers.
• C conflates statistical significance with practical or scientific significance — and ignores the base rate argument from this session.
• D inverts the conditional probability (like A, but from the other direction).
• If many students answer A or D, revisit the medical test example: “If the disease has a 1-in-1000 base rate, a positive test is probably still a false positive — even though p < 0.05.”

Any questions we missed?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Continuation of the same Poll Everywhere Q&A activity opened at the start of the session — this is not a new poll, so don’t reset/archive it yet.
• Surface any questions that have come in since the early Questions slide, especially ones raised during today’s content.
• Once you’ve finished with this slide, archive (reset) the activity at the end of the session so questions don’t carry over.

Next time

Application session: “First tests”

You’ll run your first real tests on your project data.

Remember: a significant p-value is the start of the conversation, not the end.

• Check the Q&A — surface any top-voted questions before finishing.
• Students should come ready to work on their project data. Groups may need to do data cleaning before testing.