Designing an investigation

Research Skills — Week 5

Welcome to Phase 2

Welcome to Phase 2

• What makes a good experiment?
• HolmesCo’s site investigation
• Confounding variables
• Spot the confounder
• Sampling strategies
• Design sketch
• Wrap-up

The second phase

Phase 1 (done ✓)

Scaffolded formative mini-project.

Phase 2 (starts now)

Your own group’s question and dataset.

Summative — 30% of module grade.

w01-ppdac-ring.R

• [11:05–11:10] Set the tone. Phase 2 is where students take ownership. The skills are the same; the stakes are higher.
• Briefly outline the assessment: individual 2,000-word policy report based on group project work.
• Point at the ring: where this week sits in PPDAC (Week 1, “The cycle”).

Questions?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

The assessment

Individual policy report (1,500–2,000 words).

• Based on your group’s investigation
• Same structure as Phase 1: summary → evidence → caveats → conclusions
• Due after Week 10

Your commit history matters — it shows your individual contribution.

• The report is individual even though the project is group-based. Students need to understand: the group collects and analyses data together; each person writes their own report.

What makes a good experiment?

• Welcome to Phase 2

What makes a good experiment?

🎓 Concept block 1

• HolmesCo’s site investigation
• Confounding variables
• Spot the confounder
• Sampling strategies
• Design sketch
• Wrap-up

The logic of comparison

You can’t know if something is big unless you measure something else.

Tim Harford: “Is that a big number?”
PopTech, CC BY-SA 2.0

• [11:10–11:30] Experimental design fundamentals.

Controls

What you hold constant so you can isolate the variable of interest.

Example: How does latitude affect wind farm output?

What confounding variables might we need to account for before we compare wind farms at different latitudes?

Control for	Why
Turbine model	Different models have different capacities
Terrain	Ridgetop/valley/offshore: affects wind speed
Grid curtailment	Grid operator tells certain farms to stop generating

What can’t you control? Weather. Maintenance schedules.

• Geoscience angle: in field sciences we rarely get true experiments. We observe natural variation and try to isolate factors. This makes design more important, not less.
• Maintenance schedules: more efficient after a service? Lack the data to be able to control for this - operators won’t tell you.

Treatment vs control groups

Lab sciences

Treatment group gets the intervention.

Control group doesn’t.

Random assignment.

Field sciences

“Treatment” is often a natural condition.

“Control” is a comparison group.

You can’t randomize geology.

• Because we can’t randomize, we need to think harder about what might confound our comparisons. That’s next.

HolmesCo’s site investigation

• Welcome to Phase 2
• What makes a good experiment?

HolmesCo’s site investigation

💬✏️ Exercise 1

• Confounding variables
• Spot the confounder
• Sampling strategies
• Design sketch
• Wrap-up

The scenario

HolmesCo

“Geological Solutions Since 2019”

Ground Investigation Report Summary

Esh Spinning is a proposed twelve-turbine wind farm in County Durham.

We drilled 3 boreholes alongside access roads in the Deerness valley. All three encountered competent sandstone.

We conclude that bedrock across the site is suitable for turbine foundations.

How far do you trust the conclusion?

Work with the rest of your desk for 5 minutes.

• The students have the full report in the “exercises” folder.

• [11:30–11:45] Expected problems: (a) boreholes are in the valley, turbines are on ridgetops; (b) 3 is tiny; (c) convenience sampling; (d) no context for whether “competent sandstone” = “suitable for foundation”. Requires comparison to safe & unsafe wind farm geology; presence of mine workings?

Attendance

Show the QR from https://attendance.durham.ac.uk/mytimetable, then ← back to the task.

The problems

• Tiny sample: 3 boreholes for a multi-km² site
• Wrong location: boreholes in the valley, turbines on the ridgetops
• Selection bias: sampled where it was convenient, not representative
• No comparison: how do we know this is unusually good (or bad)?
• Is this the right question? e.g. presence of mine workings?

Confounding variables

• Welcome to Phase 2
• What makes a good experiment?
• HolmesCo’s site investigation

Confounding variables

🎓 Concept block 2

• Spot the confounder
• Sampling strategies
• Design sketch
• Wrap-up

What is a confounder?

A variable that is correlated with both the treatment and the outcome.

You can’t tell which caused the effect.

• [11:45–12:05] Confounders are the hidden trap in observational studies.

Confounding examples

Observation	Apparent cause	Real cause (confounder)
Ice cream sales ↑, drowning ↑	Ice cream causes drowning?	Hot weather
More solar panels → higher GDP	Solar causes wealth?	Latitude, governance, investment
Wind farms → fewer bird species	Turbines harm birds?	???

Confounders, contra causation, can cause correlation.

• The third row is real data, and the next four slides unpack it. Leave the “real cause” column for now; come back to it.

Wind farms and birds

w05-wind-map.R

• Real data, and a real question: do wind farms harm birds? Every square of Great Britain that is mostly land. Wind farm = 8 or more turbines, from the government’s Renewable Energy Planning Database.
• Birds: every species recorded in each square in 2006–15, from the British Trust for Ornithology’s volunteer records (Garden BirdWatch, BirdTrack and surveys), via the NBN Atlas.
• Ask where the wind farms are: exposed, windy, away from towns. Park that.

Wind farm squares have fewer species

w05-birds-raw.R

Turbines harm birds?

• Let the room agree that turbines harm birds. It is the story they expect, and the confidence intervals do not overlap.
• Then ask: where do birdwatchers live? Who submits these records?

Allowing for recording effort

w05-birds-effort.R

Turbines help birds?

• A square with ten thousand garden-feeder records will list more species than a moor that three people visited. Records stand in for effort.
• The method is stratification: put squares into ten bands by number of records; compare each square with its own band.
• The answer has reversed. A wind farm developer would print this one.
• The implicit message is that wind farms are responsible for the increase. Are we reading correlation the right way?

Before the turbines

w05-birds-before.R

The “benefit” was there before any turbine. Something else chose those squares.

• These squares had no turbines while the birds were being recorded, so turbines cannot have caused the gap. Developers pick sites that were already better for birds: windy, open, unenclosed, away from people.
• “Later” includes refused applications: they still show where developers chose to go.
• Correcting for one confounder changed the answer; it did not make the answer right. Back to the table: the “real cause” for row three is both recording effort and site selection.

What would a controlled experiment look like?

You fund wildlife research. Design the study that would tell you what wind farms do to birds.

• [5 min, desks, within 11:45–12:05] Hoped-for answers: count birds at the same site before and after; compare with similar sites where nothing was built; count the same way every time.
• You cannot randomize where wind farms go — so this is the best that field science can do.

Same sites, before and after

Pearce-Higgins et al. (2012) J. Appl. Ecol. 49: 386–394

Design

• 18 upland wind farms, UK
• Breeding birds counted before, during and after construction
• Same counts on nearby reference sites with no turbines

Findings

• Snipe −53% during construction
• Curlew about −40%, with no recovery
• No such declines on reference sites

• Each wind farm is compared with itself, which removes site selection; the reference sites catch changes that would have happened anyway. This is a before–after–control–impact design.
• Skylark and stonechat increased: “may be spurious” (differences between wind farm and reference sites). But is this confirmation bias? Even a good design has to watch for confounders.
• Most of the damage was during construction, not operation. A snapshot map lacks that granularity.
• doi:10.1111/j.1365-2664.2012.02110.x

How to handle confounders

1. Randomization — when possible (rare in geoscience)
2. Stratification — compare within subgroups
3. Matching — pair similar observations
4. Acknowledgement — when all else fails, be honest about what you can’t control

• 1: e.g. “Choose at random which requested turbine sites we’ll build on”
• 2: e.g. stratify by n bird observations, earlier
• 3: Compare sites where confounders happen to have the same values
• 4: most common in geoscience and policy analysis. Honesty about confounders is itself a research skill.

Spot the confounder

• Welcome to Phase 2
• What makes a good experiment?
• HolmesCo’s site investigation
• Confounding variables

Spot the confounder

💬✏️ Exercise 2

• Sampling strategies
• Design sketch
• Wrap-up

Three scenarios

For each: identify the confounder and suggest how to address it.

1. “Communities near wind farms report more headaches than communities without wind farms.”

Confounders: rural/urban, age, awareness/nocebo effect, reporting bias.

Scenario 2

2. “Countries with higher nuclear capacity have lower carbon emissions per capita.”

Confounders: GDP, industrialization stage, energy mix decisions driven by geography and politics.

Scenario 3

3. “Students who use AI assistants score higher on coding assignments.”

Confounders: prior coding experience, motivation, time spent on assignments.

• [12:05–12:20] Quick discussion for each. Don’t linger — the point is pattern recognition, not exhaustive analysis.

Sampling strategies

• Welcome to Phase 2
• What makes a good experiment?
• HolmesCo’s site investigation
• Confounding variables
• Spot the confounder

Sampling strategies

🎓 Concept block 3

• Design sketch
• Wrap-up

Our objective is to understand a target “population”. In biology, that might be every ant in a colony. In Earth Sciences, we might want to characterize every crystal in a magma body. In Esh Spinning, we want complete knowledge of the subsurface geology across the site.

We’re seldom going to characterize the complete population, so we need a subsample. But how do we obtain a sample that has the same characteristics as the whole population?

How to choose your sample

Strategy	How	When
Random	Every unit has equal probability	Gold standard; often impractical
Stratified	Sample within each subgroup	Ensures representation
Systematic	Every nth unit	Regular grids, monitoring stations
Convenience	Whatever’s easiest	HolmesCo’s default

Shap Granite. Sample a grain: which one?
jtweedie1976 / Flickr, CC BY 2.0

• [12:20–12:40] Practical sampling. The key message: your sampling strategy is a decision you must justify.

• Characterizing a magma body. Could: sample grains at random. Have to be careful: if we just point at a sample, we’re more likley to hit a big grain. That might be good if we want to characterize the composition - or bad if we want to characterize grain size!

• Stratified: Sample all plag grains, all qz, all micas…

• Systematic: A grid?

The same site, four ways

w05-sampling-strategies.R

High Hedley Hope wind farm, County Durham.
Where on this site would you sample?
Oliver Dixon / Geograph, CC BY-SA 2.0

• Random and systematic are the pair the table cannot separate: both are “unbiased”, and they look nothing alike. Note the gaps in the random panel — that is what stratification exists to prevent.
• Systematic: Regular spacing is a shorthand for “pick the samples that are most different from one another” – R package Coreset. Difference may not be “distance in XY”.
• The convenience panel is the next slide. Ask what question that sample can answer, and about what area.

The HolmesCo default

“We sampled these because they were there.”

Honest — but weak.

Your sampling strategy is a decision you must justify in your report.

• This connects directly to Phase 2: students will need to explain why they chose their data sources and whether their sample is representative.

Design sketch

• Welcome to Phase 2
• What makes a good experiment?
• HolmesCo’s site investigation
• Confounding variables
• Spot the confounder
• Sampling strategies

Design sketch

✏️💬 Integrative exercise

• Wrap-up

Choose your topic

Browse the available project topics.

Form provisional groups (3–4 people).

Pick a candidate topic. Then sketch:

1. Question — what are you investigating?
2. Data — what would you need?
3. Comparison — what’s your control?
4. Confounders — what should you worry about?

• [12:40–12:50] Groups form and discuss. Each group shares one sentence. Topics don’t need to be final — that happens in the app session.

Wrap-up

• Welcome to Phase 2
• What makes a good experiment?
• HolmesCo’s site investigation
• Confounding variables
• Spot the confounder
• Sampling strategies
• Design sketch

Wrap-up

Key points

1. Good design = clear comparison + controlled confounders
2. Sampling is a choice you must justify
3. HolmesCo’s mistakes are easy to make — watch for them in your own work

Any questions we missed?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Continuation of the same Poll Everywhere Q&A activity opened at the start of the session — this is not a new poll, so don’t reset/archive it yet.
• Surface any questions that have come in since the early Questions slide, especially ones raised during today’s content.
• Once you’ve finished with this slide, archive (reset) the activity at the end of the session so questions don’t carry over.

Next time

Application session: “Planning your investigation”

• Finalize groups and topics
• Set up your group GitHub repo
• Write a one-page research plan

• Check the Q&A — surface any top-voted questions before finishing.
• Students should come to the application session with a topic preference and at least a tentative group.