Models and their limits

Research Skills — Week 8

Recap

Recap

• Bias bingo
• Linear regression
• Fit and interpret
• Assumptions and diagnostics
• HolmesCo’s regression
• The three failures of models
• Quick reflection
• Wrap-up

Where we are

You can now test hypotheses, compare groups, and quantify effect sizes.

This week: models.

A model is a simplified story about how the world works. The question is always: how much does it capture, and what does it leave out?

w01-ppdac-ring.R

• [5 min; clock time TBC: Week 8 plan not yet rebalanced] Frame the session: models as simplifications. Every equation in science is a model. Every test contains an implicit model.
• Point at the ring: where this week sits in PPDAC (Week 1, “The cycle”).

Questions?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

Bias bingo

• Recap

Bias bingo

💬 Homework recap

• Linear regression
• Fit and interpret
• Assumptions and diagnostics
• HolmesCo’s regression
• The three failures of models
• Quick reflection
• Wrap-up

How did you do?

The Economist, 23 September 2026: “Many surgical interventions are little better than placebo”.

Hands up: a row? Two rows? Full house?

Which square was hardest to fill?

• [20 min; clock time TBC: Week 8 plan not yet rebalanced] Recap of Weeks 3–7 through the bias-bingo homework (materials/bias-bingo.md). Answer key with paragraph numbers: staff/bias-bingo-key.qmd.
• Quick show of hands, then ask for the hardest square. Expect base rate neglect or multiple comparisons. Multiple comparisons has no clean example; reward anyone who argued “not here”.
• Don’t read the article out; students have it. Quote only the phrases on the slides.

Why did surgeons think it worked?

Patients felt better after surgery. Three stories fit that:

1. The operation fixed the problem
2. The body healed on its own
3. Being cared for made them feel better (placebo)

Pooling 100 surgical trials: about two-thirds of the improvement was stories 2 and 3.

• Squares: confounding, placebo, effect size.
• Recovery and ritual always travel with the operation, so an uncontrolled before/after comparison credits the scalpel with all three. That is confounding in its purest form.
• “Is that a big number?” One-third of the benefit is still a benefit; the question is whether it justifies the risk and cost.
• Ask: which square did you put this under? Several answers are right.

How do you separate them?

Compare	Isolates
Surgery vs sham surgery	the operation itself
Sham surgery vs no treatment	the placebo
No treatment, before vs after	natural recovery

One 2002 trial splashed water into a dish so the sham sounded right.

• Squares: control group, randomization, blinding, “Compared to what?”.
• Each row is a control group answering a different question. Most of the benefit in the sham arm turned out to be natural recovery, not placebo.
• Moseley’s knee trial (165 patients): the team matched even sounds a patient under general anaesthetic might register. Blinding taken seriously.
• A 2020 review: only about 1% of trials for chronic musculoskeletal pain compared surgery with doing nothing at all.

Why were surgeons so sure?

• Patients who come back for check-ups are mostly the ones who got better
• Those who come back don’t want to disappoint their surgeon
• Surgeons’ predictions of who will improve: no better than a coin toss

• Squares: survivorship bias, reporting bias, confirmation bias.
• The surgeon’s evidence is their own follow-up clinic. The failures have gone elsewhere; the successes are polite. Years of experience build confidence on a biased sample.
• “Confidently be unconfident” (Blazeby) is a good line to leave on the screen verbally.

The square most people missed

Scans find tears and “abnormalities” in knees, backs and shoulders.

The same abnormalities are common in people with no pain at all.

How plausible was this before we tested?

• Square: base rate neglect (Week 6).
• We want P(abnormality causes pain | abnormality seen). If most pain-free people of the same age have it too, finding it in a patient tells you very little. Same structure as HolmesCo’s gold assay.
• Also worth a minute: the paid-per-operation vs salaried contrast (UK shoulder operations fell by about four-fifths after the trials; US numbers did not). Is that a fair comparison? Countries differ in far more than how surgeons are paid: confounding again.

Now make it geological

Surgery	Geology
Only recovered patients return	Dry holes and failed slopes never get written up
Patients heal anyway	Groundwater recovers once pumping stops. Did the remediation work?
Abnormal scans in painless knees	Faults are everywhere. Is this one why the ground subsided?
Surgeons have no uncertainty	Experts disagree on the same seismic section

In geology nobody says the word “placebo”. Would you spot these?

• Ask for the geological versions students wrote at home first; use the table as backup.
• Seismic interpretation: Bond et al. (2007, GSA Today) gave the same synthetic section to hundreds of geoscientists; most did not reach the right interpretation.
• Remediation is rarely compared with an untreated control site. The answer is the Week 5 before–after–control–impact design.
• Link forward: a model is a claim about causes. Today’s question, “What is the model not capturing?”, is another way of asking which of these we’ve missed.

Linear regression

• Recap
• Bias bingo

Linear regression

🎓💻 Concept block 1

• Fit and interpret
• Assumptions and diagnostics
• HolmesCo’s regression
• The three failures of models
• Quick reflection
• Wrap-up

What is a model?

A deliberate simplification that captures some features of reality and ignores others.

Every statistical test you’ve run contains an implicit model.

The t-test models data as two normal distributions with equal variance.

Now we make the model explicit.

• [20 min block; clock time TBC: Week 8 plan not yet rebalanced] Linear regression — concept and live demo.

Linear regression

\[y = \beta_0 + \beta_1 x + \varepsilon\]

“y increases by β₁ for every one-unit increase in x, plus noise.”

That’s the whole model. It’s powerful — and limited.

Live demo

• Type this, don’t paste it. The students have elec on their own page; tell them to beat you to the R² and start typing a minute in.

• Anticipated code:

  elec <- read.csv("data/uk_electricity.csv")
  
  ggplot(elec, aes(x = year, y = bioenergy_twh)) +
    geom_point() +
    geom_smooth(method = "lm")
  
  model <- lm(bioenergy_twh ~ year, data = elec)
  summary(model)

• The slide previously showed lm(output ~ windspeed, data = turbine_df) and claimed R² ≈ 0.6. No such dataset exists in this repo, so no one could ever have run it. Biomass over time is real data the students already know from Week 1.

• Numbers: slope 1.693 TWh per year, R² = 0.926, p = 4.4e-15, residual SE 3.73 on 24 df. “Biomass generation has risen about 1.7 TWh a year, and a straight line accounts for 93% of the variation.”

• Deliberately not the solar data — that is Exercise 1 in a few slides’ time, and its negative slope should surprise them.

Reading the output

Element	Meaning
Intercept (β₀)	Predicted y when x = 0
Slope (β₁)	Change in y per unit change in x
R²	Proportion of variance explained
p-value (slope)	Is the slope distinguishable from zero?

• Emphasize R² interpretation: “R² = 0.6 means 60% of the variation is explained. 40% is something else.”

Fit and interpret

• Recap
• Bias bingo
• Linear regression

Fit and interpret

✏️💻 Exercise 1

• Assumptions and diagnostics
• HolmesCo’s regression
• The three failures of models
• Quick reflection
• Wrap-up

Solar panels and temperature

Solar panel output vs temperature — counterintuitive.

1. Make a scatter plot
2. Fit lm()
3. Interpret the slope and R²
4. Plot the fit

Is the relationship linear? Does the model capture the pattern?

• [15 min; clock time TBC: Week 8 plan not yet rebalanced] Output decreases at high temperatures for crystalline silicon panels. Students discover a negative slope where they expected positive. Good moment to ask “Did the data surprise you?”

Attendance

Show the QR from https://attendance.durham.ac.uk/mytimetable, then ← back to the task.

Assumptions and diagnostics

• Recap
• Bias bingo
• Linear regression
• Fit and interpret

Assumptions and diagnostics

🎓 Concept block 2

• HolmesCo’s regression
• The three failures of models
• Quick reflection
• Wrap-up

The four assumptions

1. Linearity — the relationship is actually linear
2. Independence — residuals don’t influence each other
3. Homoscedasticity — constant variance of residuals
4. Normality of residuals (for inference)

• [20 min block; clock time TBC: Week 8 plan not yet rebalanced] Assumptions, diagnostic plots, what to do when violated.

Diagnostic plots

Four plots, two that matter most:

Plot	What to look for
Residuals vs Fitted	Pattern = non-linearity or heteroscedasticity
QQ plot	Departures from the line = non-normal residuals

• Anticipated code — one line, and model is still in the session from the live demo a few slides back:

  plot(model)

• If the session has been reset, re-run the demo cell first.

• Walk through each plot. “No pattern in residuals vs fitted = good. Points on the QQ line = good.”

• Biomass over time is a useful case here: R² is 0.92 and the residuals-vs-fitted panel still curves, because growth was flat then steep. High R², wrong shape — which is the point of the whole session.

HolmesCo’s regression

• Recap
• Bias bingo
• Linear regression
• Fit and interpret
• Assumptions and diagnostics

HolmesCo’s regression

✏️💬 Exercise 2

• The three failures of models
• Quick reflection
• Wrap-up

The scenario

HolmesCo

“Geological Solutions Since 2019”

Groundwater Analysis Report

Linear model: groundwater level ~ rainfall. R² = 0.49.

“Rainfall is the dominant control on groundwater levels.”

Now look at the diagnostic plots…

• [15 min; clock time TBC: Week 8 plan not yet rebalanced] Show diagnostic plots with a clear seasonal pattern in the residuals. There’s a time-lag and seasonal cycle the model isn’t capturing.

What’s wrong?

w08-residuals.R

• The residuals show a seasonal pattern — the model is missing a time-dependent structure
• R² = 0.49 means 51% of the variation is unexplained
• “Dominant” is a stretch when your model misses more than it captures

• Key question: what is the model not capturing? (Seasonal recharge, time lag between rainfall and groundwater response, temperature effects on evapotranspiration.)

The three failures of models

• Recap
• Bias bingo
• Linear regression
• Fit and interpret
• Assumptions and diagnostics
• HolmesCo’s regression

The three failures of models

🎓💬 Concept block 3

• Quick reflection
• Wrap-up

Failure 1: Overfitting

A model that fits the training data perfectly but fails on new data.

It’s memorized the noise.

• [25 min block; clock time TBC: Week 8 plan not yet rebalanced] Three failures — each a different form of hubris. Demo the polynomial overfitting live.

Demo: polynomial degree 15

🖥️ Live demo

Fit a degree-15 polynomial to 20 data points. It goes through every point but oscillates wildly between them.

Compare to the straight line: fits worse, predicts better.

More complex ≠ more useful.

• Live-code: geom_smooth(method = "lm", formula = y ~ poly(x, 15)) vs y ~ x on the same plot. The visual contrast is striking.
• The next slide is the same contrast, pre-rendered — use it as the fallback if WebR misbehaves, and leave it up while they discuss.

What that looks like

w08-overfitting.R

• Both R² values are read off the fitted models. “Fits worse, predicts better” is arithmetic here, not rhetoric.
• Ask for a prediction at x = 11 from each model. One of them has an answer; the other has left the building.

Failure 2: Extrapolation

A model that works within the observed range but breaks down outside it.

Example: UK solar capacity has grown roughly exponentially since 2010. Extrapolate to 2050 → solar exceeds the entire UK grid.

w08-extrapolation.R

What’s the model not capturing?

Planning constraints, grid limits, diminishing suitable sites.

HolmesCo’s extrapolation

HolmesCo

“Geological Solutions Since 2019”

“Our linear trend in groundwater decline predicts the aquifer will be empty by 2028.”

They didn’t account for seasonal recharge or the fact that the decline was caused by temporary pumping.

Failure 3: Blind spots

Things the model structurally cannot represent.

• A single-turbine power curve has no term for wake interference → overpredicts wind farm output by 10–20%
• A carbon accounting model that counts biomass as zero structurally cannot see the emissions it ignores

You can only fix a blind spot if you know it’s there.

The new refrain

“What is the model not capturing?”

Overfitting: too much faith in your data.

Extrapolation: too much faith in your model.

Blind spots: too much faith in your framework.

• These are three forms of the same mistake: confusing the map for the territory. Each is a different flavour of hubris.

Quick reflection

• Recap
• Bias bingo
• Linear regression
• Fit and interpret
• Assumptions and diagnostics
• HolmesCo’s regression
• The three failures of models

Quick reflection

✏️ One sentence

• Wrap-up

Write one sentence

“In my project, the biggest thing my analysis might be missing is…”

• [5 min; clock time TBC: Week 8 plan not yet rebalanced] Quick written reflection. Students keep this for their reports.

Wrap-up

• Recap
• Bias bingo
• Linear regression
• Fit and interpret
• Assumptions and diagnostics
• HolmesCo’s regression
• The three failures of models
• Quick reflection

Wrap-up

Key points

1. A model is a simplification — powerful, but limited
2. Always check the diagnostics: residuals don’t lie
3. Three failures: overfitting, extrapolation, blind spots
4. Always ask: “What is my model not capturing?”

Any questions we missed?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Continuation of the same Poll Everywhere Q&A activity opened at the start of the session — this is not a new poll, so don’t reset/archive it yet.
• Surface any questions that have come in since the early Questions slide, especially ones raised during today’s content.
• Once you’ve finished with this slide, archive (reset) the activity at the end of the session so questions don’t carry over.

Next time

Application session: “Fitting and breaking models”

You’ll fit models to your data — and deliberately try to break them.

• Check the Q&A — surface any top-voted questions before finishing.
• The application session includes both project work and a structured extrapolation exercise that everyone does.