In 1943, American bombers came home wearing their damage like a map. Ground crews could tell you where a returning aircraft was most likely to be peppered: across the fuselage, along the wings, around the gunners' stations. Military analysts, weighing where to add armour — every plate of which cost speed, range and payload — did the obvious thing and plotted the bullet holes. The pattern seemed clear enough. Reinforce the riddled places, surely, because that is evidently where aircraft get hit.
At the Statistical Research Group in New York, a Columbia University team of mathematicians doing classified war work, a Hungarian émigré named Abraham Wald saw the flaw in the obvious. The data described only the planes that returned. The bombers that took fire in the engines or the cockpit were not in the sample, because they were at the bottom of the sea or scattered across occupied Europe. The unmarked places on the survivors were not evidence of safety; they were evidence of death. A plane could take a burst through the fuselage and limp home; a plane hit in the engine, as often as not, simply vanished from the data. The armour, Wald concluded, belongs where the holes aren't.
The man from Kolozsvár
Wald had reached that Manhattan office the hard way. Born in 1902 in Kolozsvár — then in Hungary, now the Romanian city of Cluj-Napoca — he came from a devout Jewish family, the grandson of a celebrated rabbi, and because he would not attend classes on the Sabbath much of his schooling happened at home. He studied mathematics in Vienna under Karl Menger and produced brilliant work in geometry and econometrics, but as a Jew in 1930s Austria he was barred from any proper university post. When Nazi Germany annexed Austria in 1938 he escaped to the United States; most of the family he left behind were murdered in the Holocaust.
The Statistical Research Group he joined was probably the most formidable statistical team ever assembled, its roster including Milton Friedman and George Stigler — both future Nobel laureates in economics — under the direction of W. Allen Wallis. Wald was its deep theoretician. Besides the aircraft work, he invented sequential analysis, a method that lets an inspector stop a test as soon as the evidence becomes decisive rather than after a fixed number of trials; it saved so much wartime effort in munitions testing that it was kept classified until after the war.
One caution the man himself would have appreciated: the polished anecdote — baffled generals proposing to armour the bullet holes, Wald delivering the devastating punchline — is a later embellishment. What actually survives is a series of dry wartime memoranda, circulated in 1943 and only published openly in 1980, under the title A Method of Estimating Plane Vulnerability Based on Damage of Survivors. In them Wald built a mathematical procedure for reconstructing the fate of the missing aircraft from the damage on the returning ones, estimating how likely a hit to each section was to bring a plane down. The dialogue is embroidered; the insight, and the mathematics, are entirely real.
Seeing the missing planes
That insight now has a name: survivorship bias — the error of drawing conclusions from the survivors of a selection process while forgetting the casualties you never see. Once you have the concept, it appears everywhere. Investment companies quietly close their worst-performing funds, so the average returns of the funds still trading flatter the industry. Old buildings seem better made than new ones — because the shoddy old buildings were demolished generations ago, leaving only the sturdy to represent their era. Bestselling founders who dropped out of university are urged on the young as proof that degrees don't matter, by an audience that never hears from the dropouts who failed.
The First World War offers a companion story, often told alongside Wald's: when steel helmets were introduced, field hospitals recorded more head wounds, not fewer — allegedly because men who would previously have died were now surviving to be counted. It is a perfect illustration, though historians note the documentary trail for the neat version is thin; treat it as a parable rather than a footnoted fact. Wald's memoranda need no such caveat. His methods were used again by the American military in Korea and Vietnam, and survivorship bias is now taught to every statistics undergraduate, usually with a diagram of a bullet-riddled bomber that never quite existed.
Wald did not live to see his idea become folklore. In December 1950 he and his wife Lucille flew to India, where he had been invited to lecture; their Air India flight crashed in the Nilgiri Hills of southern India, killing everyone aboard. He was 48. The statistician who taught the world to reason about the planes that never came back died in one — an irony he, of all people, would have insisted tells us nothing at all about the safety of flying. The lesson he left behind is simple and permanently uncomfortable: before you learn from the evidence in front of you, ask what evidence never made it into the room.
Quiz nuggets
- Abraham Wald was born in 1902 in Kolozsvár, then Hungary — today the Romanian city of Cluj-Napoca — and fled to the United States after the 1938 Anschluss.
- Wald worked in the Statistical Research Group at Columbia University, whose members included future Nobel laureates Milton Friedman and George Stigler.
- His 1943 memoranda, published openly only in 1980, were titled A Method of Estimating Plane Vulnerability Based on Damage of Survivors.
- Wald invented sequential analysis, a wartime quality-control method kept classified until after the Second World War.
- Wald and his wife died in December 1950 when their Air India flight crashed in the Nilgiri Hills; he was 48.