In the autumn of 1936, the Literary Digest was the most trusted election forecaster in America, and it knew it. The New York weekly had run a mail-in straw poll before every presidential election since 1916 and had called the winner every time; its editors liked to boast of the poll's uncanny accuracy. That year it mounted its most colossal effort yet, posting around ten million mock ballots — enough for roughly a fifth of the entire electorate — and receiving nearly 2.4 million back, a response no survey before or since has come close to matching. The verdict, splashed across the magazine days before the vote: Alf Landon, the Republican governor of Kansas, would beat President Franklin D. Roosevelt comfortably, taking about 57 per cent of the vote and carrying most of the states.
Election day delivered one of the great humiliations in the history of measurement. Roosevelt won roughly 61 per cent of the popular vote and 523 of the 531 electoral votes, carrying 46 of the 48 states. Landon won only Maine and Vermont — prompting Roosevelt's campaign manager, James Farley, to twist the old saw 'as Maine goes, so goes the nation' into 'as Maine goes, so goes Vermont'. The Digest had polled nearly two and a half million people and missed the result by around 20 percentage points.
Where 2.4 million answers went wrong
The autopsy became the founding lesson of modern statistics. The Digest had drawn its mailing lists from the sources it could get in bulk: its own subscribers, telephone directories and car registration rolls. In 1936, in the depths of the Depression, a telephone and a motor car were badges of relative prosperity, and the election that year split unusually sharply along economic lines. The magazine had, in effect, surveyed the people most likely to be furious with the New Deal and mistaken them for the country.
That is the version told in textbooks, but the fuller story is subtler. Only about a quarter of the ballots were ever returned, and analysis decades later — notably a 1988 study drawing on a 1937 survey that had asked voters whether they received and returned Digest ballots — concluded that non-response did even more damage than the biased lists. Roosevelt supporters, broadly content with their president, largely binned the ballot; Landon supporters, eager to register a protest, posted theirs back. Two biases stacked in the same direction, and no sample size on earth could cure them: ten times as many ballots would simply have produced the same wrong answer with more decimal places.
The 50,000 who were right
The men who understood this were running much smaller polls that same year. George Gallup, a 34-year-old former advertising researcher, along with rivals Elmo Roper and Archibald Crossley, used quota sampling — deliberately building miniature electorates balanced by region, age, sex and income. Gallup's survey of tens of thousands of voters called the election for Roosevelt. Better still, in July 1936, months before the Digest published anything, Gallup ran a small sample of the same kinds of telephone and car-owner lists and publicly predicted what the Digest's own poll would say — Landon at around 56 per cent — landing within about a point of the magazine's eventual figure. The Digest's editor reacted with public indignation that anyone would presume to forecast its famous poll. Gallup had demonstrated the whole point of the new science: a small sample of the right people beats a mountain of answers from the wrong ones.
The magazine never recovered. Its authority was its product, and both were gone; within two years the Literary Digest had folded, absorbed into Time in 1938. Gallup's name, meanwhile, became a synonym for polling itself — though the story carries a second act worth remembering. In 1948 the quota-sampling pollsters, Gallup included, confidently called the election for Thomas Dewey over Harry Truman, giving the Chicago Daily Tribune its immortal early edition headline, Dewey Defeats Truman, brandished by a grinning, victorious Truman the next morning. Quota sampling, it turned out, had quiet biases of its own — interviewers filling their quotas tended to pick the more approachable and more prosperous respondents — and the debacle pushed the industry towards probability sampling, in which every voter has a known chance of being selected, as statisticians had advocated all along.
The 1936 fiasco remains the standard classroom fable because its moral is so clean and so widely ignored. Every unscientific online poll, every phone-in vote, every survey of whoever happened to answer is the Literary Digest in miniature — a big number wearing the costume of evidence. The Digest drowned in the largest response in polling history, sunk by the people who never wrote back. When someone waves a sample at you, the question is never how many answered. It is who never got asked, and who never bothered to reply.
Quiz nuggets
- The Literary Digest mailed around ten million straw ballots in 1936 and received nearly 2.4 million back — and still called the election wrongly.
- The Digest predicted Alf Landon would win with about 57 per cent; Roosevelt took roughly 61 per cent of the popular vote and 523 of 531 electoral votes.
- Landon carried only Maine and Vermont, prompting James Farley's quip 'as Maine goes, so goes Vermont'.
- In July 1936 George Gallup predicted the Digest's wrong result in advance, using a small sample drawn from similar telephone and car-owner lists.
- The Literary Digest folded in 1938; ten years later the Dewey Defeats Truman headline of 1948 exposed the flaws of quota sampling too.