How confident should we be?

Research Skills — Week 3

Recap

Recap

• Error and uncertainty
• What could go wrong?
• The logic of hypothesis testing
• How much evidence is enough?
• Sampling and representativeness
• Assumption audit
• Wrap-up

Where we are

• Week 1: What makes a testable question
• Week 2: Summarizing data, honest visualization, ggplot2

You’ve produced figures showing biomass trends and emissions comparisons.

This week: how much should we trust those numbers?

w01-ppdac-ring.R

• [11:05–11:10] Recap. Ask: “Did anyone look at their figures again after last session? Notice anything new?”
• Point at the ring: where this week sits in PPDAC (Week 1, “The cycle”).

Questions?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

Error and uncertainty

• Recap

Error and uncertainty

🎓 Concept block 1

• What could go wrong?
• The logic of hypothesis testing
• How much evidence is enough?
• Sampling and representativeness
• Assumption audit
• Wrap-up

All measurements have uncertainty

The question is not whether there’s uncertainty — it’s how much and what kind.

• [11:10–11:30] Two types of error, then precision vs accuracy.

Two kinds of error

Systematic error (bias)

Your instrument is consistently wrong.

Example: emission factors that exclude the supply chain always underestimate true emissions.

Random error (noise)

Measurements vary each time.

Example: annual electricity generation fluctuates with weather, demand, and plant outages.

• Systematic error is the sneaky one — you can’t detect it by repeating the measurement. You need an independent check.
• Random error averages out with enough data. Systematic error doesn’t.

Precision vs accuracy

Precise but inaccurate

Measurements cluster tightly — but around the wrong value.

e.g., a miscalibrated thermometer

Accurate but imprecise

Measurements centre on the right value — but scatter widely.

e.g., noisy field readings

Always on target, or always into your own goal?
Dominic Nelson, CC BY-SA 4.0

Which is worse? It depends on whether you can correct the bias.

• Memorable example: Footballer hoofing in a long-range shot.
  • Precise, accurate: always on target
  • Equally precise, inaccurate: Always hoofs ball into own goal
  • Imprecise, accurate: Always gets it to right end of the pitch.
  • Imprecise, inaccurate [me]: Could end up anywhere
• Geoscience examples: radiometric dating is precise but relies on assumptions about initial conditions (accuracy). Temperature records are accurate on average but have coverage gaps (precision).

All four combinations

w03-precision-accuracy.R

• Point at the precise-but-inaccurate panel: repeating the measurement makes it look better and does nothing to the error. That is the whole danger of systematic bias in one picture.
• The dashed line is a real formation mean from borehole_temp.csv, not a round number chosen to look tidy.

What could go wrong?

• Recap
• Error and uncertainty

What could go wrong?

💬✏️ Exercise 1

• The logic of hypothesis testing
• How much evidence is enough?
• Sampling and representativeness
• Assumption audit
• Wrap-up

The coal emission factor

w03-emission-factors.R

“What are the sources of uncertainty in this number?”

Discuss as a desk. 5 minutes. Then share.

• The number in question is coal’s 910. Coal is chosen deliberately: it is a measured quantity with no accounting conventions to untangle. Biomass’s zero waits for the assumption audit at the end of the session.

• [11:30–11:45] Expected answers: coal rank and carbon content vary between batches; calorific value; plant efficiency; measured at the stack or calculated from fuel burned? Combustion only, or mining, methane and transport too (910 rises to 990 with the supply chain)? Averaged over which plants, and which years?

• Classify each as systematic or random.

• Keep the list: the assumption audit asks which of these apply to biomass, and what biomass adds.

• UNFCCC = United Nations Framework Convention on Climate Change DESNZ = UK Department for Energy Security and Net Zero

The logic of hypothesis testing

• Recap
• Error and uncertainty
• What could go wrong?

The logic of hypothesis testing

🎓 Concept block 2

• How much evidence is enough?
• Sampling and representativeness
• Assumption audit
• Wrap-up

The courtroom analogy

In a trial

• Start with “innocent”
• Look at the evidence
• Decide: enough to convict?

In statistics

• Start with “no effect” (null hypothesis)
• Look at the data
• Decide: enough to reject the null?

The Jury (1861), John Morgan — Buckinghamshire County Museum, public domain

• [11:45–12:05] This is conceptual — no formulae yet. The goal is to build intuition for why we need formal testing.

Two types of mistake

	Null is true	Null is false
Reject null	✘ Type I error (false positive)	✔ Correct
Don’t reject	✔ Correct	✘ Type II error (false negative)

Type I: Convicting an innocent person.

Claiming one ridgetop site is windier than the next when they’re the same.

Type II: Acquitting a guilty one.

Missing a real difference because your sample was too small.

• Type I: Null = innocence; rejection = convict
• Type II: Null = innocence; acceptance = release
• The significance level (α, usually 0.05) is our threshold for Type I error — how much surprise we require.
• Type II error depends on sample size, effect size, and variability. We’ll formalize this later; for now, the intuition is enough.

What does “overlap” look like?

w03-overlap.R

At-chimney emissions: CO₂ measured as it leaves the plant. Same means in each panel (Coal: 910; Biomass: 1000 kg CO₂/MWh).

The question: is this difference bigger than we’d expect by chance?

• What the figure shows. Both centres are at-chimney emission factors read from emission_factors.csv: biomass combustion_only and coal official, which DESNZ defines as combustion only. That is what a stack monitor would read at the point of combustion — the CO₂ physically leaving the plant per MWh generated.
• It is deliberately not lifecycle or net accounting. No supply chain, no regrowth credit, no UNFCCC zero. Say the term out loud — “at-chimney emissions” — because the rest of the module turns on the difference between this and the net figures we meet next session.
• Expect the “that looks wrong” reaction. Biomass is higher than coal at the chimney, and that is correct: wood holds less energy per unit of carbon than coal, so a MWh from pellets sends modestly more CO₂ up the stack. Nothing about the chimney makes biomass cleaner — the entire case for biomass rests on the regrowth argument, which is exactly why it deserves scrutiny. (materials/carbon-accounting-summary.md has the derivation if someone asks.)
• Only the spread is invented. The question is what a fixed difference looks like at two levels of variability.
• “Would you bet these are different?” Ask it of each panel in turn. The gap between the centres is identical — only the spread changes.
• The shaded band is where a single measurement tells you nothing: it could have come from either kind of plant. Verbal example: it’s not unlikely that I’d get a measurement of X from biomass and Y from coal, reversing the true order of the means.
• Just visual intuition for now: we’ll formalize this mathematically in Week 6.

How much evidence is enough?

• Recap
• Error and uncertainty
• What could go wrong?
• The logic of hypothesis testing

How much evidence is enough?

💬 Exercise 2

• Sampling and representativeness
• Assumption audit
• Wrap-up

The scenario

You measure at-chimney emissions from 5 biomass plants and 5 coal plants.

The biomass mean is higher. But there’s overlap.

Are you convinced?

• [12:05–12:15] Let students discuss in small groups, then collect responses.

Attendance

Show the QR from https://attendance.durham.ac.uk/mytimetable, then ← back to the task.

What if…

• What if you measured 50 of each?
• What if the difference were twice as large?
• What if one measurement were an extreme outlier?

• Couch this in “precision vs. accuracy” - what would we gain here?
• Ability to discriminate; confounders?

Sample size, effect size, and variability all matter.

We’ll formalize this in Week 6 — for now, trust your intuition that eyeballing isn’t good enough.

• The point: students should leave feeling “I can see why we need formal tests” — which motivates Week 6’s t-tests.
• Don’t resolve the disagreement neatly. The tension is productive.

Sampling and representativeness

• Recap
• Error and uncertainty
• What could go wrong?
• The logic of hypothesis testing
• How much evidence is enough?

Sampling and representativeness

🎓 Concept block 3

• Assumption audit
• Wrap-up

Where did your data come from?

And does it represent what you think it represents?

• [12:15–12:35] Population vs sample, selection bias, survivorship bias, confounding.

Population vs sample

You have data from Drax — one power station.

Can you generalize to “biomass electricity”?

Drax produces ~86% of UK biomass electricity. Does that help or hurt?

One power station, standing in for a whole fuel. © Drax

• It helps (the sample is most of the population) but it also means Drax is unusual — it’s massive, purpose-built, and uses a specific supply chain. Other biomass plants may behave very differently.

Selection bias

If you only measure the biggest, best-known facility, your results may not generalize.

Survivorship bias: if failing biomass plants shut down and disappear from the data, the remaining ones look better than average.

Confounding: biomass plants might be newer than coal plants. Any efficiency difference might reflect age, not fuel.

Every boat in the harbour survived the storm.
Royal Navy / MOD, OGL v1.0

Older, or worse built? The fuel is not the only thing that differs.
AI-generated (Gemini/Nano Banana 2)

• Geological parallel: sampling only exposed rock faces — weathered, accessible, atypical of the subsurface.
• These are the same issues HolmesCo will demonstrate in Week 5 (valley boreholes, convenience sampling).

Assumption audit

• Recap
• Error and uncertainty
• What could go wrong?
• The logic of hypothesis testing
• How much evidence is enough?
• Sampling and representativeness

Assumption audit

✏️💬 Integrative exercise

• Wrap-up

The carbon accounting rule

The standard framework: CO₂ from burning biomass is counted as zero at the point of combustion.

Discuss as a desk:

1. List the assumptions this framework makes
2. For each: is it testable? What evidence would challenge it?
3. Which assumptions, if wrong, would most change the conclusion?

• [12:35–12:50] This directly feeds Week 3 app (scenario analysis) and Week 4 (the briefing).
• Bridge from Exercise 1: which of the coal uncertainties also apply to biomass? What does biomass add (regrowth, and over what period)?
• Expected assumptions: trees regrow (timescale?), supply chain emissions are negligible, land use doesn’t change, the alternative fuel would have been burned anyway.
• The “if wrong, most changes the conclusion” question is the key one. It teaches students to prioritize assumptions by impact.

What did you find?

• Type 3–4 assumptions in as students call them out. Students will need them for the application session (scenario analysis) and the Week 4 briefing.

“What assumptions are we making?”

This is the question that separates good analysis from bad.

Wrap-up

• Recap
• Error and uncertainty
• What could go wrong?
• The logic of hypothesis testing
• How much evidence is enough?
• Sampling and representativeness
• Assumption audit

Wrap-up

Key points

1. All measurements have uncertainty — the question is how much
2. Systematic error is more dangerous than random error
3. The biomass “zero” is an assumption, not a measurement
4. Next step: what happens when we change the assumptions?

Exit ticket

A lifecycle assessment reports that UK biomass electricity produces 0 gCO₂/kWh. What is the most important thing to check before accepting this number?

PollEv.com/geol

text geol to 07480 781235

• Correct answer: B — What system boundary and timescale the assessment used.
• The “zero” is not a measurement — it is a policy assumption that depends entirely on what is included (combustion only? transport? processing? regrowth?) and over what time horizon.
• D (peer review) is a reasonable instinct but wrong as the most important thing — a peer-reviewed paper can still use a narrow system boundary.
• This question directly tests whether students have internalized “What assumptions are we making?” as a reflexive question.

Any questions we missed?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Continuation of the same Poll Everywhere Q&A activity opened at the start of the session — this is not a new poll, so don’t reset/archive it yet.
• Surface any questions that have come in since the early Questions slide, especially ones raised during today’s content.
• Once you’ve finished with this slide, archive (reset) the activity at the end of the session so questions don’t carry over.

Next time

Application session: “Changing the assumptions”

You’ll take the data and ask: does the answer change when we change the inputs?

• Check the Q&A — surface any top-voted questions before finishing.
• Students should arrive ready to do arithmetic in R — multiplying emission factors by generation volumes under different scenarios.