Deja view - counting evidence twice

Wait a minute! Didn’t that horse already cross the finish line?

In citing research to support an argument, it’s not very hard to accidentally count something twice - particularly when we don’t get all the way back to primary sources.

For example in David J. Dempsey’s 2010 book Present Your Way to the Top, at the bottom of page 3 he says that:

“....In a survey that appeared in the Book of Lists, 3,000 people were asked ‘What are you most afraid of?’ and the number-one response was ‘speaking before a group.’ It ranked higher than the fear of heights, insects, and even death (Ref. 2). In another study, 2,500 people were asked to list their greatest fear, and the largest percentage of respondents listed public speaking (Ref. 3).”

His reference 3 was to the 7th edition of Stephen E. Lucas’s book The Art of Public Speaking. I’ve previously blogged about how Lucas referred to the 1973 Bruskin survey, where that data in the Book of Lists originally came from. So, references 2 and 3 actually are to the exact same data.

Stephen J. Senn has described how this same pesky problem occurs in medical science. His article on Overstating the evidence - double counting in meta-analysis and related problems discusses ways to keep from fooling ourselves.

Some criminals deliberately miscount bills when giving change. A folded bill can be put in the front of a stack and counted twice. It then is unfolded as it’s handed back. If you don’t recheck the count you will be shortchanged. Other short change artists fold and hide one bill they got from you, in the back of a stack. Then they say that you made a mistake, and get you to add an extra.   

In war (hot or cold) it’s common to confuse the enemy about your unit strength. Back in high school I remember reading Robert Lee Scott’s book God Is My Co-Pilot about his World War II experiences flying a P-40 from near Kunming, China. He had his propellor spinner repainted when refueling and rearming after each solo sortie, and produced the impression that one plane was a whole squadron.  

In 1955 the Soviet Union flew their ten new Bison bombers past the reviewing stand on Aviation Day. When they got over the horizon they circled back. They made a total of six passes - to give the illusion of sixty bombers and create western fears of a Bomber Gap.