Back during World War II, the RAF lost a lot of planes to German anti-aircraft fire. So they decided to armor them up. But where to put the armor? The obvious answer was to look at planes that returned from missions, count up all the bullet holes in various places, and then put extra armor in the areas that attracted the most fire.
Obvious but wrong. As Hungarian-born mathematician Abraham Wald explained at the time, if a plane makes it back safely even though it has, say, a bunch of bullet holes in its wings, it means that bullet holes in the wings aren’t very dangerous. What you really want to do is armor up the areas that, on average, don’t have any bullet holes. 
Why? Because planes with bullet holes in those places never made it back. That’s why you don’t see any bullet holes there on the ones that do return. Clever!
On a related note — related because it involves both an air force and some counterintuitive statistical reasoning — Henry Farrell writes about a passage from Justin Fox’s Myth of the Rational Market involving the great behavioral economist Daniel Kahneman. Here is Kahneman reacting to an Israeli Air Force flight instructor who told him that whenever he chewed out a student he got better performance the next time out, but whenever he praised a student he got worse performance:
As a man trained in statistics, Kahneman saw that of course a student who had just brilliantly executed a maneuver (and was thus praised for it) was less likely to perform better the next time around than a student who had just screwed up. Abnormally good or bad performance is just that — abnormal, which means it is unlikely to be immediately repeated. But Kahneman could also see how the instructor had come to his conclusion that punishment worked. “Because we tend to reward others when they do well and punish them when they do badly, and because there is regression to the mean,” he later lamented, “it is part of the human condition that we are statistically punished for rewarding others and rewarded for punishing them.”
Likewise, we tend to go to the doctor only when an illness has become really bad. Sometimes, though, that just means the illness is at its lowest point and has nowhere to go but up. And sure enough, after we see the doctor we get better. But it’s not always because of something the doctor did. Sometimes it’s just a natural consequence of when humans choose to see them.
Finally, on yet another related note — related this time solely because it involves counterintuitive statistical reasoning, not because it involves an air force — check out Nate Silver’s brief seminar on how weak most of us are at calculating how much variance there is in the things we do every day. There’s a lot of natural variance in everyday life, and there’s not always a reason for it. But we like to make up reasons anyway. Statistics is a profoundly unnatural enterprise for most of us.
