AI, reproducibility, and integrity

Research Skills — Week 9

Recap

Recap

• What AI does well — and badly
• AI audit
• Reproducibility
• Reproducibility check
• Scientific integrity
• Open discussion
• Wrap-up

Where we are

Your projects are nearly done. You’ve designed investigations, run tests, fitted models, and identified limitations.

This week: two forces that shape modern research —

AI and the pressure to produce results.

w01-ppdac-ring.R

• [11:05–11:10] This session is more discursive than technical — more debate and reflection, less coding.
• Point at the ring: where this week sits in PPDAC (Week 1, “The cycle”).

Questions?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

What AI does well — and badly

• Recap

What AI does well — and badly

🎓💬 Concept block 1

• AI audit
• Reproducibility
• Reproducibility check
• Scientific integrity
• Open discussion
• Wrap-up

AI is good at…

• Generating first-draft code
• Summarizing text
• Suggesting structure
• Spotting syntax errors
• Explaining error messages

• [11:10–11:30] Honest assessment of AI capabilities and limits.

AI is bad at…

• Knowing whether its answer is correct
• Understanding your specific data
• Making judgement calls about methodology
• Citing sources reliably
• Distinguishing meaningful from spurious results

Live demo (if possible)

🖥️ Give an AI the HolmesCo gold assay scenario

“We tested 10,000 soil samples with a 95%-accurate assay. 50 tested positive. What’s the probability of a true gold deposit?”

Watch what it does with the base rate.

• The AI will likely either get it wrong (ignoring the base rate) or give a generically correct but vague answer without applying it to the specific numbers.
• The point: fluent ≠ thoughtful.
• This is the Week 6 press release, word for word. They have already worked it out by hand — the test is whether the AI can.
• Whatever number it offers, the next slide is the answer, and it is not a number.

The discovery is smaller than the error rate

w09-assay-floor.R

• Week 6 did this arithmetic in a 2×2 table. Same numbers, same press release — keep the two in step if either is ever edited.
• A 95%-specific assay calls 5% of barren ground positive: 500 samples out of 10,000, before any gold is involved. HolmesCo found 50.
• So the honest answer is not a percentage at all. It is “you have fewer hits than your own assay produces from nothing, and you never told me how common deposits are here, so a probability was never available.”
• That is what to check the AI’s answer against. A fluent one will give a confident number — which means it invented the base rate.

The key distinction

AI is fluent but not thoughtful.

It can write a convincing paragraph about any result — including a wrong one.

AI audit

• Recap
• What AI does well — and badly

AI audit

💬 Exercise 1

• Reproducibility
• Reproducibility check
• Scientific integrity
• Open discussion
• Wrap-up

Your experience

Think about a time in this module when you used AI (or were tempted to). What did it help with? Where did you have to override it?

Common patterns:

• Useful: debugging code, plot templates, explaining errors
• Unreliable: interpreting results, choosing methods

• [11:30–11:45] Open discussion, or think, discuss as a desk, then share. Collect themes on the board. Students are generally honest about this — the goal is reflection, not confession.

Reproducibility

• Recap
• What AI does well — and badly
• AI audit

Reproducibility

🎓 Concept block 2

• Reproducibility check
• Scientific integrity
• Open discussion
• Wrap-up

The replication crisis

Many published findings don’t hold up when others try to reproduce them.

w09-replication-rates.R

• [11:45–12:10] Reproducibility as a practical skill.
• “Many” is exactly the kind of word this course spends ten weeks training them to challenge, so it should not survive on our own slide.
• The three definitions of success differ, and that is the second lesson here: “replicated” is not one thing. Psychology counted a significant replication; economics required the same direction too; cancer biology used three of five criteria. Comparing the bars directly is the mistake to point at.
• All three numbers come from the papers’ own abstracts, cited under the chart.

Why?

• p-hacking — trying many analyses, reporting the one that “works”
• Underpowered studies — too few observations to detect real effects
• Selective reporting — filing away null results
• Data not shared — no one can check your work

Your commit history IS your reproducibility record

If someone cloned your repo and ran your code, would they get the same results?

One number, or the shape it came from

A bootstrap interval — like the one you’ll build this afternoon in exercise w9_1, from uk_electricity.csv — is not a fact about the world. It is a range read off a distribution of 1,000 resampled means.

w09-bootstrap-distribution.R

• Idea adapted from Andrew Heiss, Data Visualization with R, session 6 “Uncertainty” — show uncertainty as a distribution, not just an interval.
• Same statistic as exercise w9_1: the mean of late (UK wind generation, 2015-2025), bootstrapped 1,000 times.
• The top panel is what goes in a report sentence: “65.6 TWh [55.2, 75.1]”. The bottom panel is what that sentence compresses — the shaded band is the same interval, cut out of the same shape.
• Point to ask: the histogram is not symmetric around 65.6 in the way a textbook normal curve would be. That is real information the single-number interval throws away.

Rerun it without the seed

This afternoon’s bootstrap draws random resamples. Run the same code twice with no set.seed() and the 95% interval moves both times:

Run	95% interval (TWh)
No seed, run 1	55.8 to 74.8
No seed, run 2	55.7 to 76.1
set.seed(2847), any run	55.2 to 75.1, always

Which of these three rows would you want printed in your report?

• Numbers are from three actual runs of this afternoon’s replicate() code against uk_electricity.csv (two unseeded, one with set.seed(2847)), not estimated.
• The point is not that the seeded number is “more correct” — all three intervals are equally valid draws from the same procedure. It is the only one a reader, a marker, or you in six months can check against what you wrote down.
• This is the same distinction as reproducibility generally: a seed does not fix an answer, it fixes which answer you get, so it can be checked.

Practical reproducibility habits

1. Script everything — no “I typed this in the console”
2. Document your decisions — why this test, why you excluded those data points
3. Record software versions — sessionInfo() in R
4. Seed anything random — set.seed() before sample() or replicate(), once, not inside the loop
5. Share data and code — within ethical limits

• Connection to the policy report: a policy-maker should be able to trace your conclusion back to your data.
• Item 4 connects straight to this afternoon’s exercise w9_1, whose fourth fault is exactly a missing seed.

Reproducibility check

• Recap
• What AI does well — and badly
• AI audit
• Reproducibility

Reproducibility check

💬✏️ Exercise 2

• Scientific integrity
• Open discussion
• Wrap-up

Can you re-run your analysis?

Open your project repo. Start a fresh R session.

Run your analysis script from scratch.

Does it work? What breaks?

Common problems:

• Hardcoded file paths
• Missing libraries
• Steps done in the console but not saved

Fix what you can. 10 minutes.

• [12:10–12:20] This is surprisingly powerful. Most students will find at least one thing that breaks. That experience is more persuasive than any lecture.

Attendance

Show the QR from https://attendance.durham.ac.uk/mytimetable, then ← back to the task.

Scientific integrity

• Recap
• What AI does well — and badly
• AI audit
• Reproducibility
• Reproducibility check

Scientific integrity

🎓💬 Concept block 3

• Open discussion
• Wrap-up

The spectrum

• [12:20–12:40] Brief and frank. Not preachy.

Three levels

Level	What	Example
Innocent errors	Rounding, wrong column, misread output	Your peer reviewer catches these
Questionable practices	p-hacking, HARKing, selective reporting	Often unintentional — this is what base rate and multiple comparisons sessions were about
Fabrication	Making up or altering data	Rare, career-ending, easier to detect than you’d think

Where does AI fit?

If AI writes your analysis and you don’t check it → you are responsible for the errors.

If AI generates data or figures that don’t reflect reality → that’s fabrication, even if unintentional.

The HolmesCo connection

HolmesCo doesn’t fabricate data.

But their corner-cutting on design, analysis, and interpretation produces conclusions that are functionally no better than fabrication.

Integrity isn’t just “don’t lie.”

It’s “do the work properly.”

Open discussion

• Recap
• What AI does well — and badly
• AI audit
• Reproducibility
• Reproducibility check
• Scientific integrity

Open discussion

💬 Q&A

• Wrap-up

Your questions

This is a looser session. What’s on your mind?

• Questions about your reports?
• AI use policies?
• What counts as plagiarism when AI is involved?

• [12:40–12:50] Address what comes up. Students often have genuine questions about the AI boundary. The honest answer: use AI as a tool, check everything, and document what you used.

Wrap-up

• Recap
• What AI does well — and badly
• AI audit
• Reproducibility
• Reproducibility check
• Scientific integrity
• Open discussion

Wrap-up

Key points

1. AI is useful for mechanics; unreliable for judgement
2. Reproducibility = scripted, documented, version-controlled
3. Integrity = doing the work properly, not just not lying

Any questions we missed?

Submit questions:

PollEv.com/geol

text geol to 07480 781235

• Continuation of the same Poll Everywhere Q&A activity opened at the start of the session — this is not a new poll, so don’t reset/archive it yet.
• Surface any questions that have come in since the early Questions slide, especially ones raised during today’s content.
• Once you’ve finished with this slide, archive (reset) the activity at the end of the session so questions don’t carry over.

Next time

Application session: “AI as critic”

You’ll feed your analysis to an AI and evaluate its feedback.

The question: how much should you trust it?

• Check the Q&A — surface any top-voted questions before finishing.
• Students should bring their analysis code and a draft results section to the application session.