Week 7: Deepening Your Analysis

Effect sizes and what they mean

Introduction

The content session established that region matters, with a p-value of 0.0000001. That is a statement about how confident you can be, not about how much it matters. A p-value that small mostly tells you the sample was big enough.

Today is about the other question — how big is the difference? — and about turning the answer into a sentence a wind developer could act on.

Today’s goals:

Compute η² and say what fraction of the variation region explains
Compute Cohen’s d for one pair, and read it honestly
Turn an effect size into units a non-statistician can use
Apply all of it to your own project

How these exercises work

The code boxes start empty, except for comments to guide you through the code that’s expected.

Work one comment at a time. If you get stuck:

Hint 1 restates what you are trying to do.
Hint 2 sketches the shape the code might take.
Hint 3 gives you the code with the key pieces blanked out.
Solution gives you the lot.

Take a real attempt before opening a hint: you will learn by doing, not by reading. The demonstrators are here to help.

Three things worth knowing:

The boxes share one R session, exactly like a script. Anything you create in Exercise 1 is still there in Exercise 2.
Variable names are up to you. The last thing your block prints is what gets checked, so end each block with the answer.
Exercises have extensions underneath them. If you finish early, give them a stab. These are designed to equip you to tackle real coding problems you might face, using the documentation. We can help – but check the manuals first!

Stuck?

Ask on the class board, in Code Q&A. Someone else is likely stuck on the same thing, so asking in public helps them too. Demonstrators check it most weekdays, but they give classmates a chance to answer first.

The board is private to this class, so sign in to GitHub first. If the link shows “404 – page not found”, you’re not signed in, or you haven’t accepted the Classroom 50 invitation yet.

Exercise 1: How much does region explain?

ANOVA splits the total variation into a part explained by the grouping and a part left over. η² (eta-squared) is the first as a fraction of the whole:

\[\eta^2 = \frac{SS_\text{between}}{SS_\text{total}}\]

Both sums of squares are in the ANOVA summary table. Extract them and print η².

Hint 2

The shape of it:

fit <- aov(outcome ~ group, data = df)
tab <- summary(fit)[[1]]
tab

ss_between <- tab["group", "Sum Sq"]
ss_total <- sum(tab[, "Sum Sq"])

ss_between / ss_total

The row name has trailing spaces in some R versions, so tab[1, "Sum Sq"] is the more robust way to get the first row.

Hint 3

wind_aov <- aov(capacity_pct ~ ______, data = wind)
aov_table <- summary(wind_aov)[[1]]
aov_table

ss_between <- aov_table[1, "Sum Sq"]
ss_total <- ______(aov_table[, "Sum Sq"])

ss_between / ______

Solution

wind_aov <- aov(capacity_pct ~ region, data = wind)
aov_table <- summary(wind_aov)[[1]]
aov_table

ss_between <- aov_table[1, "Sum Sq"]
ss_total <- sum(aov_table[, "Sum Sq"])

ss_between / ss_total

η² = 857.6 / 2788.7 = 0.31. Region explains about 31% of the variation in capacity factor: a large effect by Cohen’s benchmarks (0.01 small, 0.06 medium, 0.14 large).

“Where you build matters — geography accounts for nearly a third of the difference in output” is a sentence a planner can use.

But read the other 69% before you write it. Most of the variation in capacity factor is within regions, between one site and the next. Region is the largest single factor you have measured, and it is still the smaller part of the story. An η² of 0.31 is simultaneously “a large effect” and “not most of what is going on”, and both halves belong in the report.

Treat the benchmarks with suspicion too. Cohen proposed them as a stopgap for fields with no established scale, and said so. In wind energy there is a scale — percentage points of capacity factor, which convert directly into money — so use that instead wherever you can.

Extension: build it yourself, then improve it

Optional.

Compute η² from first principles, without aov(). You need the grand mean, each group’s mean and size, and: \(SS_\text{between} = \sum n_i (\bar{x}_i - \bar{x})^2\), \(SS_\text{total} = \sum (x_{ij} - \bar{x})^2\). You should land on 0.31 exactly. Doing this once removes most of the mystery from ANOVA.
η² is biased upward: it can only ever be positive, even when the groups are identical in truth. Demonstrate that — generate four groups from the same distribution with rnorm(), run the ANOVA, and look at η². Repeat a few times. What is the smallest value you can get?
ω² (omega-squared) corrects that bias: \(\omega^2 = \frac{SS_\text{between} - df_\text{between} \times MS_\text{within}}{SS_\text{total} + MS_\text{within}}\). Compute it — you should get 0.28 — and rerun task 2 with it. Which of the two would you report, and would you have said the same before you knew which was larger?

Exercise 2: How big is one difference?

η² describes the whole grouping. Cohen’s d describes one pair: the difference between two means, measured in standard deviations.

\[d = \frac{\bar{x}_1 - \bar{x}_2}{s_\text{pooled}}, \qquad s_\text{pooled} = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}}\]

Compute it for Scotland against Southern England — the largest gap in the data.

Hint 3

scotland <- wind$capacity_pct[wind$region == "Scotland"]
south <- wind$capacity_pct[wind$region == "______"]

n1 <- length(scotland)
n2 <- length(south)
pooled_sd <- sqrt(
  ((n1 - 1) * sd(scotland)^2 + (n2 - 1) * ______^2) /
    (n1 + n2 - 2)
)

(mean(______) - mean(______)) / ______

Solution

scotland <- wind$capacity_pct[wind$region == "Scotland"]
south <- wind$capacity_pct[wind$region == "Southern England"]

n1 <- length(scotland)
n2 <- length(south)
pooled_sd <- sqrt(
  ((n1 - 1) * sd(scotland)^2 + (n2 - 1) * sd(south)^2) /
    (n1 + n2 - 2)
)

(mean(scotland) - mean(south)) / pooled_sd

d = 1.54. Cohen’s benchmarks are 0.2 small, 0.5 medium, 0.8 large, so this is very large — a Scottish site sits about one and a half within-region standard deviations above a Southern English one.

The comparison that makes d worth computing is the other one. Scotland against Northern England gives d = 0.07, which is indistinguishable from nothing. The same variable, region, produces a very large effect for one pair and no effect at all for another — which is precisely what the single η² of 0.31 averaged away.

Report both, or report neither. An η² on its own invites the reader to assume the grouping matters uniformly, and here it does not.

Extension: d for every pair

Optional.

Compute d for all six pairs and put them in a table beside the TukeyHSD p-values from the content session. Do the two orderings agree? They need not — d ignores sample size and p-values do not.
An effect size is an estimate, so it has uncertainty too. Bootstrap a confidence interval for d: resample both groups with sample(x, replace = TRUE) several hundred times, recompute d each time, and take the 2.5th and 97.5th percentiles with quantile(). Is the interval narrow enough to justify calling this effect “very large” rather than merely “large”?
Cohen’s d is biased upward in small samples, like η². Look up Hedges’ g, apply its correction factor, and see how much it moves at n = 25 per group. Then work out at what sample size you would stop caring.

Exercise 3: The number a developer can use

“d = 1.54” will not survive contact with a planning meeting. Neither will “η² = 0.31”. Convert the difference into the unit the decision is actually made in: electricity.

A 50 MW wind farm running flat out for a year would produce 50 × 8760 = 438,000 MWh, or 438 GWh. Its capacity factor is the fraction of that it actually delivers.

Print how much more electricity a 50 MW farm would generate per year in Scotland than in Southern England, in GWh.

Hint 2

The shape of it:

max_gwh <- 50 * 8760 / 1000

diff_pct_points <- mean(group_a) - mean(group_b)

max_gwh * diff_pct_points / 100

Sanity-check as you go. A 50 MW farm at a 30% capacity factor should produce something around 130 GWh a year. If your intermediate numbers are in the thousands, a division by 100 has gone missing.

Solution

max_gwh <- 50 * 8760 / 1000

diff_pct_points <- mean(scotland) - mean(south)

max_gwh * diff_pct_points / 100

27.7 GWh a year. The Scottish farm produces about 129 GWh, the Southern English one about 102.

That is the difference between the two sentences you can write:

Capacity factor differed significantly by region (F(3, 96) = 14.2, p < 0.001, η² = 0.31).

The same 50 MW wind farm would generate roughly 28 GWh more each year in Scotland than in Southern England — enough for about 9,000 homes.

The first is what you did. The second is what it means. Your report needs both, in that order, and most students write only the first.

One caution, since you now have a number that sounds decisive: it assumes the 25 sampled sites in each region are representative, that a new farm would land at its region’s mean rather than somewhere in the 69% of variation within the region, and that nothing else — grid connection, planning, cable length to demand — differs between the two. None of those is safe. The number is a starting point for an argument, not the end of one.

Extension: put uncertainty on the recommendation

Optional.

The 28 GWh figure uses the two sample means as though they were exact. Redo it using the ends of the TukeyHSD confidence interval for that pair instead, and report the range. How much does the recommendation move?
A site in the bottom quartile of Scottish sites against one in the top quartile of Southern English ones: which wins? Compute it. This is the single most important caveat on a regional recommendation, and it takes two lines of code.
At roughly £50 per MWh, what is 28 GWh a year worth over a 25-year operating life? Then find one assumption in that calculation you would not defend in front of an examiner.

Now: your project data

The rest of the session is your group’s.

Run your main test — t-test for two groups, ANOVA for three or more.
Check the assumptions, as in Week 6, and say what you did.
Compute an effect size — Cohen’s d for a pair, η² for a grouping. If you used ANOVA and it was significant, run TukeyHSD() as well, and check whether every pair behaves like the overall test suggests. Often one pair carries the whole thing.
Convert the effect into your project’s own units — tonnes, pounds, degrees, GWh, households. Exercise 3 is the template, and this step is what separates a report from a printout.
Write the results paragraph, with the statistics first and the translation second.

Four questions to answer as you go:

Is the result significant, and at what threshold — chosen when?
Is the effect big enough to matter in your units?
What does the confidence interval include that would change your recommendation?
Which assumption, if violated, would do the most damage to your conclusion — and did you check that one?

If your analysis is not ready

Any of these is a good use of the hour:

Compute the effect size your project will need on someone else’s data, so the code exists before the data do.
Write the results paragraph with the numbers left as blanks. The shape of the sentence is the hard part; filling in three numbers later is not.
Work out what effect size would be too small for your project to care about, and write it down now, before you know the answer. That single sentence is the most credible thing you can put in a report, and it is worthless if written afterwards.

Save your work

Commit your analysis code and your results paragraph to your group repo via a pull request. Your individual report goes in its own repo: accept the report assignment and start writing there. At the end of Week 9 your group’s history is copied into it.