Week 5: Designing Your Investigation
Controls, confounders, and sampling
Welcome to Phase 2
In Weeks 1–4, you learned to question, explore, analyse, and communicate — all using the biomass mini-project. Now you’ll design your own investigation. Same skills, your question.
This page accompanies the content session. These exercises are discussion-based: no code today, and no marking. Discuss as a desk, and argue before you open a reveal — they are ordered so the first gives you a way in and the last gives you the answer.
Each exercise has an extension underneath it. If your desk finishes early, that is where to go.
Exercise 1: Critique HolmesCo’s site investigation
Read the HolmesCo ground investigation report, then answer the questions as a desk.
See the full report: HolmesCo Site Investigation
Summary: HolmesCo was hired to assess whether a proposed wind farm site in County Durham has suitable ground conditions. They drilled 3 boreholes — all in the valley bottom where access was easy — and concluded that “the bedrock is competent sandstone throughout the site.”
Discuss as a desk:
- Where are the boreholes? Where will the turbines actually go?
- Is 3 boreholes enough to characterize a whole site?
- What kind of sampling bias is this?
- What is missing that would make this a proper investigation?
Do not start with the statistics. Start by drawing the site: the ridge, the valley below it, the roads, and three crosses where the rig went.
Then ask the only question that matters about any sample — what would have to be true for these three points to represent the whole site? — and ask who chose where to drill, and what they were optimizing for.
- No boreholes on the ridge, which is exactly where the turbines are planned. The boreholes are in the valley: convenient, and unrepresentative in a direction you can predict. Here the difference is written into HolmesCo’s own report: the valley is Pennine Lower Coal Measures, while the ridge is capped by Pennine Middle Coal Measures. “Throughout the site” stretches one formation’s result over another.
- Roads are not a random sample of ground. Much of the ridge is made ground — this is old coalfield country — and roads are built on bedrock for good engineering reasons. So even a road-side borehole on the ridge would be biased: the places you can reach are the places where the ground is good. And in a coalfield, “is there competent sandstone?” may not be the right question at all — what about old mine workings?
- Three is a very small sample. You cannot characterize geological variability across a site from three points, and nothing in the report tells you how variable the site is, because with three points clustered together they could not have found out.
- Selection bias — convenience sampling. They sampled where a drill rig could get to, not where the data were needed. Note that this is not dishonesty. It is what happens when the cost of sampling varies across the thing being sampled, which it almost always does.
- No comparison. “Competent” compared to what? Without a reference the word is an opinion wearing a technical coat.
A better investigation would start with a desk study (geological map, extent of made ground, mining records), then place boreholes on the ridge where the turbines go — off the roads as well as beside them — stratify across the geological units and the made ground, and include enough of them to say something about variability rather than only about the mean.
The transferable point: a sample is not representative because it was collected carefully. It is representative because of where it was collected — and convenience and representativeness usually pull in opposite directions.
Extension: make it worse, then make it honest
Optional, for desks that finish early.
- HolmesCo’s conclusion is not necessarily false — the valley bedrock really may be competent sandstone. Rewrite their one-sentence conclusion so that it is fully supported by the three boreholes they actually drilled. How much of the original claim survives?
- Suppose the budget allows exactly five boreholes, not three. Mark on your sketch where you would put them, and write the sentence justifying each. Then say which one you would sacrifice if the budget fell to four, and why that one.
- Name a real constraint that would make your own ideal design impossible — access, cost, permission, time. Every project in this room will hit one. The skill being tested is not avoiding the constraint; it is stating in the write-up which direction it pushed your results.
Exercise 2: Spot the confounder
For each scenario, identify the most likely confounder and suggest how you would address it.
Scenario A: > “Communities near wind farms report higher rates of headaches than > communities without wind farms.”
Scenario B: > “Countries with more solar panels have higher GDP per capita.”
Scenario C: > “HolmesCo found that deeper boreholes in County Durham have higher > temperatures. They conclude that drilling deeper always finds hotter > rock.”
A confounder is not just “something else going on”. It has to be linked to both sides: to who ended up in which group, and to the outcome you measured. If it touches only one of the two, it is noise, not confounding.
So ask two questions, in this order. What decided which communities, countries or boreholes went into which group? And could that same thing have moved the outcome on its own?
In one of these three the answer to the first question is “the researchers did” — and that is the one to be most suspicious of.
A — Wind farms and headaches
Candidates: awareness (people who know they live near a wind farm report more symptoms; the nocebo effect is well documented here), rural versus urban baselines, age structure of rural populations, and reporting bias in communities already campaigning.
Addressing it: compare communities matched on demographics and rurality that differ only in proximity. Better still, survey the same community before and after a wind farm is built — that removes every confounder which does not change over the window.
B — Solar panels and GDP
Candidates: latitude and governance (stable, wealthy, temperate countries both install solar and have high GDP), investment climate, and outright reverse causation — rich countries can afford solar, rather than solar making countries rich.
Addressing it: look within countries over time and ask whether solar growth precedes GDP growth. Between-country comparisons of this kind are nearly uninterpretable, because countries differ in thousands of ways at once.
C — Borehole depth and temperature
The confounder is where they drilled. HolmesCo drilled deep in the geothermally interesting area and shallow elsewhere, so depth and location are entangled and cannot be separated after the fact.
This is the worst of the three, and the reason is not obvious. Rock genuinely does get hotter with depth, so HolmesCo’s conclusion is true. Their data simply do not demonstrate it. A correct conclusion drawn from evidence that could not have supported it is still bad science — and far harder to catch, because nobody checks the working on an answer they already believe.
Addressing it: standardize depth across locations, or record location and include it in the analysis. Neither is possible retrospectively if nobody wrote down where the holes were.
Extension: confound your own project
Optional.
- Take your group’s provisional question and write down the one confounder that would most embarrass you if a reviewer raised it. Then write what you would do about it. If the answer is “nothing”, say so plainly in the plan — a named, unaddressed confounder is a caveat; an unnamed one is a flaw.
- Scenario C is the dangerous pattern: right answer, wrong evidence. Find another instance of it, in this course or outside it. The Week 2 biomass emission factors are good hunting ground.
- For Scenario A, design the cheapest study that would actually separate the nocebo explanation from a physical one. What does each explanation predict that the other does not? If you cannot name such a prediction, the hypotheses are not yet testable — which is the Week 1 problem wearing new clothes.
Integrative exercise: design sketch
Browse the list of summative project topics for 5 minutes. Then, with your provisional group, pick a candidate topic and sketch answers to:
- What is our question? (One sentence.)
- What data would we need?
- What is our control or comparison?
- What confounders should we worry about?
Each group shares one sentence with the class.
Say your question out loud, then say what you would conclude if the result came out the opposite way to your hypothesis. If you cannot finish that second sentence — or if the honest answer is “we would go looking for a different analysis” — it is not yet a research question.
Key points
- You cannot know whether something is big unless you measure something else. Controls are how you isolate what matters.
- A confounder is linked to both the grouping and the outcome. Ignore it and your result may be an artefact.
- Your sampling strategy is a decision you have to justify. “We sampled these because they were there” is honest but weak — and it is the most common design in real geoscience, so learn to state which way it biases you.
- Everything HolmesCo does wrong this week, you should do right in your project — and where you cannot, say so in writing.