Numéraire
Volume IV · The Premium · The Prices
Plate the Eleventh, of twelve

The Survivor

Survivorship bias, and the failed gamblers we never see: why, observing only those who succeeded — the funds that lived, the founders who struck it rich — and never those who took the identical risks and lost, we systematically mistake luck for skill. The armored planes, the graveyard of vanished funds, the coin-flippers with spotless records; and the hard conclusion that when the unlucky disappear, skill and luck cannot be told apart by their outcomes — so that the reward we hand the survivor as the wage of judgment is, in unknowable part, the prize of having survived.

in print · published

Through plate after plate this volume has trailed a single shadow. The bold investor who staked everything on one holding and grew rich looked, in the diversification plate, like a man of nerve and vision — though his fortune was the prize of a bet he need never have placed. The fund that beat the market looked, in the efficient-market plate, like the work of genius — though chance alone, among the thousands who tried, would have produced it. The founder who struck it rich looked, in the profit plate, like a master of judgment — though his judgment may have been a guess that happened to come good. The seller of catastrophe insurance looked, in the black-swan plate, like a steady earner — until the disaster he had not provided for arrived. In every case the figure we behold is a survivor, and in every case we are tempted to read his survival as the proof of a skill. This plate, the volume's penultimate, drags that shadow into the light and gives it its name: survivorship bias, the systematic error by which, seeing only those who came through and never those who took the identical risks and were destroyed, we credit to skill what may have been only luck. It will show the error in its classic forms — the armored planes, the graveyard of vanished funds, the coin-flippers with spotless records — and will press toward the hard conclusion the whole volume has been approaching: that when the unlucky have disappeared from view, skill and luck can no longer be told apart by their outcomes, so that the reward we hand the survivor as the wage of his judgment is, in a proportion we can never determine, merely the prize of his having survived.

The bias is simple to state and astonishingly hard to feel. We form our picture of what leads to success by examining the successful — and the successful are, by definition, the ones still standing to be examined. The failures, who attempted the same thing and were ruined, have left the stage: their funds are closed, their companies dissolved, their fortunes scattered, their memoirs unwritten. They do not appear in the rankings, the case studies, the admiring profiles; they are absent from the very data on which we build our theories of success. And their absence is not random. It is precisely the unsuccessful who are missing, so that the sample we study has been selected, with perfect and invisible reliability, for survival. From such a sample one can learn a great deal about what the survivors had in common — and almost nothing about whether those commonalities caused the survival, for the failures may have shared every one of them too, and they are not here to be asked. To reason from the survivors to the causes of success, without accounting for the vanished failures, is to commit the error in its pure form; and because the failures are invisible, the error feels not like error at all but like the plainest common sense — learning success from the successful, as who would not?

The classic cure for the error is a story from the Second World War, and it turns the instinct exactly on its head. The military, wishing to armor its bombers against enemy fire, examined the planes returning from their missions and plotted where they had been hit: the wings and the fuselage were riddled, the engines and the cockpit largely clean. The natural proposal was to reinforce the parts that showed the most damage. The statistician Abraham Wald saw that this was precisely backwards. The planes being examined were the ones that had come back — the survivors — and the pattern of their wounds showed not where bombers were most often struck, but where a bomber could be struck and still return. The clean areas on the survivors, the engines and the cockpit, were not spared by the enemy; they were the places where a hit was fatal, so that the planes hit there had gone down and were not in the hangar to be examined. The armor, Wald concluded, belonged exactly where the returning planes were unmarked — for the absence of damage there was not evidence of safety but evidence of death. The whole of survivorship bias lives in that inversion: the marks on the survivors map what can be endured, and it is the unmarked places, the wounds we never see because their bearers did not return, that kill.

The wounds borne by those who returned show only what a body can survive; it is the unmarked places that are mortal, for the ones they struck are not here to bear witness.

The error pervades the ordinary measurement of economic success, quietly inflating every estimate of skill. Consider the records of investment funds. When one reads that the average fund returned such-and-such over the past twenty years, the figure is very often computed from the funds that still exist — and the funds that performed worst have, across those twenty years, been quietly closed or merged into others and struck from the record. The dead are not counted; only the survivors are averaged; and the average is therefore flattered, by a margin careful study has measured in whole percentage points, into reporting a skill the full population never possessed. The same bias rules the literature of enterprise. The shelves groan with studies of the habits of the wildly successful founder — his early rising, his boldness, his scorn of convention, his trick of leaving school — offered as the recipe for success. But these are the habits of the survivors, and no one has tabulated the legions of failed founders who rose just as early, dared just as much, scorned convention just as freely, and vanished without a trace. If the failures shared the very same habits, the habits explain nothing; and the confident recipe, distilled from winners alone, may be worthless, or worse than worthless — a set of instructions for taking precisely the risks that ruined the many we never hear of. The survivors write the history of success, and the dead are not present to dispute it.

Underneath these particular cases lies a hard truth about evidence itself: that once the unlucky are gone, a track record cannot, by itself, distinguish skill from luck. Imagine a great contest of coin-flipping, with a million entrants, the losers dropping away at every round. After ten rounds, by the plain arithmetic of chance, something like a thousand entrants will have flipped ten heads in a row; after twenty rounds, one or two will have flipped twenty. These champions will have spotless, dazzling records — an unbroken string of wins — and will be, every one of them, the beneficiaries of nothing whatever but luck. Yet if we are shown only the champions, and not the million who fell away, their records are indistinguishable from what genuine skill would produce; a single truly skilled flipper, had such a thing existed, would have shown precisely the same unbroken string. This is the crux, and it is unforgiving. A brilliant record is consistent with great skill and consistent with great luck, and when the failures who flipped and lost have been cleared from the room, there is nothing left in the record alone to tell us which of the two we are looking at. The survivor's track record, however glorious, is evidence of survival; whether it is also evidence of skill, the record by itself can never say.

Set a million to flipping coins and some will flip twenty heads running; their records are spotless, their skill is nothing, and from the record alone no eye can tell them from a master.

It would be a serious overcorrection, and a misreading of this plate, to conclude that all success is luck and skill a myth. That is not the claim, and the volume is careful not to make it. Skill is real. Some investors do possess genuine and durable edges — better information, a sounder temperament, the discipline to act when others panic — and earn by them; some founders do read an uncertain future more truly than their rivals; some of the reward that flows to survivors is the honest wage of a real ability, or of a real uncertainty soberly borne. The trouble is not that skill does not exist but that we cannot reliably detect it from the outcome alone, and that survivorship bias pushes our estimate of how much of success is skill systematically, and often grossly, upward. To sort the skilled from the merely lucky takes far more than a record of wins: it takes a sample long enough that luck has had time to wash out, a full accounting of the failures as well as the survivors, and a clear idea of what luck alone would have produced — and even then the separation is partial and contested. The honest position is the uncomfortable middle: that skill and luck both operate; that they are mixed, in every fortune, in proportions the outcome cannot reveal; and that our untutored eye, shown only the winners, will mistake far more of the luck for skill than is truly there.

For the volume's long question — whether the reward for bearing risk and uncertainty is earned or merely fallen into — survivorship bias supplies not an answer but the reason a clean answer cannot be had. Every reward the volume has examined is, in the world, drawn by a survivor: the premium collected by the investor who did not go broke, the profit kept by the founder who did not fail, the fortune held by the family whose bets came good. We see these survivors and read their rewards as the wages of skill and nerve and judgment — as earned, as deserved. But the failures who bore the identical risks and lost are gone, uncounted, and their absence inflates the apparent skill of all who remain; and once they are gone, no inspection of the survivor's record can tell us whether his reward was the wage of a real judgment or the prize of a lucky draw. The confident attribution of a great fortune to merit — which is the whole substance of the earned reading — is therefore an inference the evidence cannot bear, however fervently the fortunate, and the world that admires them, may wish that it could. This does not convict every fortune of luck; it convicts our confidence. The verdict the volume has held in suspense through plate upon plate here takes its final and unwelcome form: not that the reward for risk is never earned, but that whether any particular reward was earned by skill or won by luck is, in the general case, simply undecidable from what we are able to see — and that the whole bias of our seeing runs toward crediting the skill and forgetting the luck.

So the survivor stands as the volume's deepest lesson and its hardest: that the reward we are most sure was earned by skill is, in a measure we cannot fix, the prize of a survival we have mistaken for skill, because the ones who did the same and died are not here to correct us. The price of risk, traced now through eleven plates, has proved at every turn less transparent than the clean theory promised — too large for sober fear, too small for true catastrophe, and drawn, in the end, by survivors whose merit we cannot read from their winnings. One plate remains, and it must do what a final plate should: render, from all of this, what verdict can honestly be rendered — not upon any single fortune, whose alloy of skill and luck is sealed beyond our reading, but upon the great fortune in general, and the question the whole volume was built to ask. How much of any great fortune is the wage of a courage genuinely shown, and how much the mere reward of having borne uncertainty and survived, is the matter of the plate to come, the keystone of the arch. The bridge from here runs to the primitive of choice, where what remains of merit, when outcome is so poor a guide to it, must at last be weighed. As the table says of every price, first to last: the argument remains open.

A note on the plate's sources

  1. Abraham Wald, “A Method of Estimating Plane Vulnerability Based on Damage of Survivors” (1943) — the armor belongs where the survivors are unhit.
  2. Francis Bacon, Novum Organum (1620) — the drowned worshippers whose votive tablets were never hung.
  3. Nassim Nicholas Taleb, Fooled by Randomness (2001) — luck in the mask of skill.
  4. Stephen J. Brown, William N. Goetzmann, Roger G. Ibbotson & Stephen A. Ross, “Survivorship Bias in Performance Studies” (1992) — the graveyard of vanished funds.
  5. Daniel Kahneman, Thinking, Fast and Slow (2011) — the illusion of skill, and what we see being all there is.
  6. Phil Rosenzweig, The Halo Effect (2007) — the false narratives built upon the winners.
  7. Michael J. Mauboussin, The Success Equation: Untangling Skill and Luck (2012) — the modern reckoning of the two.