NCEA Level 3 · Year 13 · Statistics · 2 minutes
How opinion polls work (and why they sometimes miss)
Who you ask matters more than how many. And every poll comes with a margin.
Key terms
- Population
- Everyone you want to know about.
- Sample
- The people you actually ask.
- Simple random sample
- A sample where every possible group of that size is equally likely to be chosen, so it tends to look like the population.
- Margin of error
- A measure of the random wobble in a poll; for results near fifty percent it is roughly one divided by the square root of the sample size.
- Sampling bias
- A problem where some groups are more likely to be asked, or to answer, so the sample misleads.
- Confidence interval
- A range, such as thirty-seven to forty-three percent, likely to contain the true value.
Transcript
In nineteen thirty six, an American magazine ran an enormous election poll. More than two million people answered. It predicted Roosevelt would lose. He won forty six of forty eight states. So what went wrong?
The Literary Digest mailed ten million ballots to car owners, telephone users and its own readers. In the Great Depression, those lists leaned towards wealthier people. And only about a quarter sent their ballots back.
Meanwhile, a pollster called George Gallup asked about fifty thousand people, chosen with quotas to mirror the whole country. He called it for Roosevelt. The lesson: a huge biased sample can lose to a smaller representative one.
The population is everyone you want to know about. The sample is the people you actually ask. In a simple random sample, every possible group of that size is equally likely to be chosen, so the sample tends to look like the population.
But even a good random sample wobbles. That's sampling error. For results near fifty percent, a rule of thumb for the margin of error is one divided by the square root of the sample size. Ask a thousand people, and that's about three percentage points. So forty percent in the poll gives a ninety five percent confidence interval of roughly thirty seven to forty three percent.
Margin of error only measures that random wobble. It can't fix sampling bias: when some groups are more likely to be asked, or to answer. That's what sank the Digest. And opinions can shift before election day.
Now, a New Zealand poll. A typical one asks about a thousand voters. Say Party A is on thirty four percent and Party B on thirty one. The gap between them has its own margin of error, nearly twice as big, so the poll can't tell you who's really ahead. And a two point jump since last month? Could just be sampling error.
So, to recap. One: a big sample can't fix a biased one. Two: the margin of error is roughly one divided by the square root of the sample size. Three: a poll is a snapshot with a margin, not a prediction.
Sources
Every fact in the video, and where it comes from.
- 1en.wikipedia.org/wiki/The_Literary_Digest
- In 1936 The Literary Digest polled about ten million people; about 2.38 million (≈2.4 million, about a quarter) responded.
- The Digest's names came from its own readers, registered car owners and telephone users (also club membership lists); in the Depression these lists leaned towards better-off people.
- Final Digest headline, 31 Oct 1936: 'Landon, 1,293,669; Roosevelt, 972,897'; it predicted Landon would win with about 57% of the vote.
- Roosevelt won every state except Maine and Vermont (46 of 48) with about 61% of the popular vote (60.8%; Freedman et al. round to 62%).
- George Gallup surveyed about 50,000 people and correctly predicted Roosevelt's win (Gallup used quota sampling to match the population, not a simple random sample).
- 2facweb.cdm.depaul.edu/sjost/it223/documents/election.htm
- The Digest's names came from its own readers, registered car owners and telephone users (also club membership lists); in the Depression these lists leaned towards better-off people.
- Roosevelt won every state except Maine and Vermont (46 of 48) with about 61% of the popular vote (60.8%; Freedman et al. round to 62%).
- George Gallup surveyed about 50,000 people and correctly predicted Roosevelt's win (Gallup used quota sampling to match the population, not a simple random sample).
- 3historymatters.gmu.edu/d/5168
- Final Digest headline, 31 Oct 1936: 'Landon, 1,293,669; Roosevelt, 972,897'; it predicted Landon would win with about 57% of the vote.
- 4en.wikipedia.org/wiki/Simple_random_sample
- In a simple random sample every member of the population has an equal chance of being selected.
- 5en.wikipedia.org/wiki/Margin_of_error
- 95% margin of error for a proportion near 50% ≈ 1.96 × sqrt(0.25/n) ≈ 0.98/√n ≈ 1/√n; n = 1000 → ≈ ±3.1 percentage points. Quadrupling n halves the margin.
- 6pewresearch.org/short-reads/2016/09/08/understanding-the-margi…
- A simple random sample of 1,067 has a margin of error of ±3 points at 95% confidence; the margin for the difference between two candidates is about twice the individual margin; sampling error is only one of many sources of error.
- 7veriangroup.com/hubfs/1News-Verian-Poll-FAQs.pdf
- 1News Verian polls use a nationally representative sample of around 1,000 eligible NZ voters; maximum sampling error ≈ ±3.1 percentage points at 95% confidence for results around 50%, smaller for results further from 50%.
Image and sound credits
- Campaign rally at Madison Square Garden, 31 October 1936; “The Roosevelt Caravan”: FDR Presidential Library & Museum, CC BY 2.0, via Wikimedia Commons
- Walker Evans, New Orleans street corner, 1936: Farm Security Administration, public domain, via Wikimedia Commons
- Wall-mounted telephone, c. 1930: Wyndham & Districts Historical Society, CC BY, via DigitalNZ
- Voters outside the polling booth, Wellesley Street West, Auckland, 1908 (Auckland Weekly News): Auckland Libraries Heritage Collections, no known copyright restrictions
- Ballot boxes, New Zealand, 2008: SimonLyall, public domain, via Wikimedia Commons
- Literary Digest headline (31 October 1936) re-typeset; all charts drawn in code
- Paper textures: ambientCG (CC0) · Torn paper edges generated with AI
- Sound effects: Freesound contributors (CC0)
- Music: generated with ACE-Step · Voice: Gemini TTS
Teaching this next week?
Kuraplan turns this topic into a lesson plan, slides and a worksheet for your class in about a minute.
Plan a lesson on this topic
