I cannot remember when I first heard the story about armoring WWII bombers against enemy fire, but it was so counterintuitive that it captured my imagination. I think about Abraham Wald and his analysis of those aircraft whenever I have to consider lots of data and what seems like apparent answers. The story of how bombers were eventually armored has led to counterintuitive conclusions in my own life.
The airplane story took place during World War II when a young mathematician named Abraham Wald was working for the war department’s Statistical Research Group. The problem he was working on was a statistical analysis of the distribution of damage to bombers coming back from their missions. The broken planes would land, and someone would map out the various holes left from enemy fire scattered over the aircraft. Eventually, a master map was created for each type of airplane, showing the concentration of damage laid out on a diagram of the plane. The Statistical Research Group’s task was to decide where were the best spots to add reinforcing armor to the airplanes. The obvious answer was to look at the diagram, find the places that had the most holes and then apply more shielding to those areas. Abraham Wald had other ideas.
Abraham’s genius was to recognize that all the collected data only came from aircraft that had successfully returned. He assumed the damage across all airplanes (those that landed and those that crashed) was uniformly distributed. He then went on to think about where the holes might be on the aircraft that went down. Logically, he concluded, the missing aircraft’s damage was in spots other than the ones the team could examine. The damage maps they had collected were, in fact, diagrams showing where aircraft could take a hit and still make it back. Instead of having a chart of where to add extra armor, they had a picture showing them where they did not need it. The “blank spaces” in the diagrams were the airplane’s most vulnerable areas.
Perhaps humans are not naturally built for statistics. We can see the damage, count up holes, create pretty diagrams, but our intuition to tell a story about the data in front of us inevitably leads us astray. We forget about the airplanes that did not come back because they are not present, even though they still have data necessary to complete the analysis. Those holes, the ones we cannot see, fill a gap in our knowledge. The empty spaces complete the picture.
Looking for the empty spaces to complete the picture turns out to be a very useful life skill. My favorite trick, when confronted with a problem is to look at the blank spaces and see if they tell a different story. The technique starts with a willingness to suspend judgment, and then ask for information that may have been overlooked. Sometimes the right answer is not obvious at all, no matter how many smart people have achieved consensus.