Obstacles to Good Reasoning 3B: Misleading Comparisons


The way numerical and graphical information is presented to us can impede our ability to reason well. We're going to look at three basic classes of ways we can be misled (or mislead!). The first is absolute vs relative quantity. Absolute numbers don't give us any context. To evaluate the importance of a statistic we need to evaluate it relative to similar events or in terms of rates or percentages. Another way we can be mislead is by failing to distinguish between average and median. A third way is according to how graphs are drawn.

Misleading by Numbers: Absolute vs Relative Quantity, Rates, and Percentages

Presenting information as an absolute number is (in most cases) akin to committing the fallacy of confirming evidence. For example, suppose I tell you hospitals average 156 emergencies per full moon. OMG!!! The full moon causes people to go loco! Well, hold on a tick. before we jump to conclusions we're missing some very important information. If you think back to the last lesson on confirmation bias you should be able to figure it out. We need to know the average number of visits on non-full moon nights. Guess what? There's no difference.1

The lesson here is that in order to know if a quantity is significant we need to compare it to similar events or things; that is, we need to put it in context. Of course, this isn't always the case. Sometimes absolute numbers matter, for example when we're talking about preventable deaths. But even then we have to be careful of the context. Suppose we're evaluating a medical treatment and we learn that over 500 people die from complications. This is bad but we need more information. We need to know how many would have died if there had been no treatment at all. Suppose it is 4000. Well, if that's the case then you're better off doing the treatment than not.

Another way we can avoid being mislead by absolute numbers is to express them as a percentage of the total phenomena we're investigating. For example, the US spends 158 billion on welfare. That's a whole lotta enchiladas. If, however, we put that as a percentage of the annual federal budget, it turns out to be about 10%.2 Still a fair amount, but probably not as much as you might have thought.

Another example might be some people's outrage that there will be or have been cuts military spending. Although it's a matter of controversy just how much the US spends on the military/defense, most people put it at just above 20%.3 Here we need to make some decisions about the appropriate context for comparison. Since military is something you use against other militaries one useful comparison class would be the military spending of other countries, particularly adversarial countries. During the hight of the British Empire, their government had a rule to spend double whatever the nearest competitor spent. That is, if France went from 90 to 100 ships, Britain would go from 180 to 200. Doubling your next rival seems to be a reasonable way to ensure dominance. If you don't think doubling is enough, how many times bigger than your competitor do you think a military should be? By what multiple do you think the US military exceeds its nearest competitors? It turns out that the US military is currently as large as the next seven countries combined—3 of which are close allies.


Another way absolute numbers can mislead is if the events or categories being compared are of different sizes. For example, one common trop in the anti-vaccine literature is to say that more vaccinated people get measles than unvaccinated. Even though this is false, we can show how it could be true without undermining the reasonableness of getting vaccinated. After a single dose, the MMR vaccine is 95% effective against vaccines (i.e., 5% failure rate). Suppose we live in a village with 100 children. That means, out of every 100 vaccinated children 5 could catch measles. About 91% of children are vaccinated against MMR. That means out of a 100 children 91 are vaccinatedx.05 failure rate=5 vaccinated children will get measles. The infection rate for measles for unvaccinated people is almost 100%.4 Let's be beyond charitable and say, in our village of 100 children only 60% of unvaccinated children catch measles (thanks to herd immunity). That is only 3 unvaccinated children The anti-vaxxer can say, Ah ha! More vaccinated children (5) than unvaccinated (3) caught measles. It's better to be unvaccinated!!111!!!!1!

What's the mistake here? It goes back to absolute numbers. The vaccinated population is many times larger than the unvaccinated population so even if their infection rate is low their absolute numbers will be greater than the unvaccinated group whose numbers are small. At the end of the day, which would you rather? A 5% chance of catching a disease or a 60% chance? The correct context for comparison is rate of infection in each respective population group, not absolute numbers.

Caveat about Using Percentages
When we're dealing with extremely low numbers, percentages can mislead. This occurs a lot in claims from the supplement industry. Suppose a certain supplement increases a biomarker from 0.1 ppm to 0.2 ppm. The advertisement will read something like: Super Gainz 2000 increases [insert biomarker] levels by 100%!!11!!!!11! This is an instant where absolute quantity might matter if that biomarker is something that can fluctuate without any major effects.

Central Tendency and Average vs Median

Suppose you want to figure out which teacher to take next semester for math. Both teachers give difficult tests but you want to know which gives the more difficult ones. In teacher A's class there are seven students. They receive the following scores on a test: 2, 2, 2, 2, 20, 20, 20. In this case the average score (or the mean) is the sum of all the scores divided by seven. This works out to 70/7 = 10. Note that even though 10 is the average, it is distorted by the high scores of 20 compared to other scores. The majority of the students' scores are below the average.

In teacher B's class each of the 7 students scores 10. The average score is 10 (70/7). Looking at the averages we can't tell which class to take, so how do we decide?

The median is the value which is such that half the scores are above it and half the scores below. For teacher A the median is 2 and for teacher B it's 10. Which teacher would you take?

The moral of the story is that averages distort the picture because they don't track the central tendency of the data. Outliers get erased when we only look at averages. When you hear averages cited, be weary and see if you can track down the median.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License