Principles of Logic 4: Hypothesis Testing


Now that we know what conditionals are we can take a baby step towards learning the scientific method. In it's most basic form, the scientific method uses modus ponens and conditionals to test causal claims. Let's see how:

Suppose someone says echinacea cures the common cold. I can take this claim and convert it into a conditional hypothesis:

Conditional hypothesis: If someone with a cold takes echinacea then their cold will be cured.

In order to test the conditional hypothesis I'm going to get out my Bunsen burner, my beakers, my test tubes, safety goggles, and my white lab coat. But most importantly I'm going to need logic. Specifically, I'm going to need the modus ponens rule. Let me explain:

I have a conditional hypothesis:

(P1). If someone takes echinacea then their cold will be cured.

In order to test it I'm going to make the antecedent true. In other words, I'm going invite people to my lab when they get a cold and give them echinacea. Hence, the second premise of our modus ponens:

(P2). People with colds take echinacea.

Finally, I'm going to see if the conditional is true. If I give people with a cold echinacea it it true that

(C). it will cure their cold (?)

To recap, all I'm doing is making a modus ponens argument:

(P1). If someone with a cold takes echinacea then their cold will be cured.
(P2). People with colds take echinacea.
(C). Are their colds cured?


The Details: Control Groups, Natural Prevalence Rates, and Placebos

Suppose I gave a 1000 people echinacea and 7 days later their colds are completely better. Conditional hypothesis confirmed!!!! If I give people with colds echinacea they get better!!!11 I gave it to them and they got better!!!!11!! Hold the press!!!! Echinacea cures colds in just 7 days!!!!

At this point you might begin to suspect the result isn't that impressive. Why not? Because in order to establish causation we also need to know the average amount of time it takes for someone to get rid of their cold without using echinacea. Let's call it the natural rate of remission (or recovery). Then we'll compare that duration to the group that gets the treatment. If there's a difference, then we have reason to believe the causal connection.

So, what this means in terms of our experiment is that we need two groups. We need one group as the control group: that is, the group that doesn't receive treatment. The control group will give us the average rate of recovery without intervention. The treatment group will get the treatment (duh!). Once we've given treatments to both the control group and the treatment group, we'll take the average recovery time of each group. If the recovery times are substantially different then we can say there's some evidence in favor of the conditional hypothesis.

Quick Aside on What This Lesson Doesn't Include: Before moving forward, I need to make a few remarks about scientific reasoning concepts that we won't cover in this particular unit: measurement errors, blinding, dosing groups, selection bias, sample bias, placebo, nocebo, degrees of latitude, statistical power, statistical significance, and many more. We're just at the beginning stages of learning about scientific reasoning. We have a long way to go. If you're familiar with the above concepts, hang on, we'll get there. If you aren't, please continue on in blissful ignorance (for now!). All this to say, for this unit, we're just keeping things simple so we can get a grasp on the foundations of scientific reasoning. The fancy details will come later in the course.

Now, let's finish up with the above example. Many real labs have done the above experiment. Guess what the results are? People buy echinacea all the time so it must work, amIright? AmIright? It turns out that all the highest quality studies (larger sample sizes, better controls) all report either no difference between groups or a statistically negligible effect size.1 So why do people keep buying it? Return to our lessons on confirmation bias!

The best way to learn how to use conditional hypothesis for basic scientific thinking is to run through examples…

Example 2: Thimerosal and MMR Vaccines
If you waste a few dollars on echinacea, no big deal. But sometimes failing to engage in critical thinking doesn't just lead to false beliefs and an lighter wallet; it leads to people dying or getting sick unnecessarily. In 1998, Andrew Wakefield, a British doctor published a study purporting to show that thimerosal in the MMR (measles, mumps, rubella) vaccine causes autism. The study was problematic for a variety of reason and was eventually retracted.2 But not before the damage was done.

The media, who either forgot their college critical thinking class or were on their cell phones when they took it, reacted unskeptically to the Wakefield study. Soon Jenny McCarthy and other celebrity "experts" took up the cause, infecting the internet with false beliefs. Thousands of parents, scared by the prospect of inadvertently giving their child autism, refused to vaccinate. Over time, pockets of unvaccinated populations emerged in turn leading to outbreaks of what had previously been an eradicated disease.3

Rant over. Let's pull out our test tubes and modus ponens and see how to test the claim.

Claim: Thimerosal in vaccines causes autism.
Conditional hypothesis: If a child receives a vaccine with thimerosal then they are (more) likely to get autism.

Notice my application of the principle of charity in formulating the conditional hypothesis. An uncharitable interpretation would be: If a child receives a vaccine with thimerosal they will get autism. That's too strong. The charitable reading is that it increases the likelihood not that it guarantees the effect. Besides, if it were true the vaccines with thimerosal always cause autism pretty much everyone vaccinated before 2001 would have autism. I haven't googled it yet, but I'm pretty sure that's false.

How are we going to set up the study? If I only measure the autism rate of children who have been vaccinated with the MRR vaccine I can't establish causation. I need to compare the autism rate of unvaccinated to vaccinated and see if there's a difference. There are a couple of ways to do this.

The simplest way is to give one large group of children the MMR vaccine and to have another group as my control (i.e., they don't get the vaccine). The control group establishes the natural prevalence rate. It tells me the natural prevalence of autism in a given population. Then I follow the two populations for several years. (Some of the first behavioral indications of autism are at around 4 years old but usually it's at around 5 or 6 years old). If the vaccinated group has a statistically significant higher rate, then I have evidence in favor of the hypothesis.

Aside: Sometimes, for ethical reasons you can't have a control group so you need to do either retroactive studies or look for what are called "natural experiments." Vaccines are one such case. Don't worry about this for now. We'll cover these concepts in the unit on scientific reasoning.

So what really happened with the relation between thimerosal and vaccines? Here's the really interesting thing. Even though many labs tried to replicate Wakefield's results, none could. But since so many parents got scared and stopped vaccinating their children, a new MMR vaccine was developed without thimerosal. By 2001 all vaccines (except the flu vaccine) were thimerosal-free.

If thimerosal causes autism what should have happened to the autism rate after thimerosal was removed from vaccines in 2001? It should have fallen, right? Guess what happened? Autism rates rose (significantly). In the unit on scientific reasoning we'll discuss why in more detail. The short answer is the inclusion criteria changed. The diagnosis "autistic" used to be reserved for only severe cases but it changed to "autism spectrum" and now includes even mild cases.

The removal of thimerosal is an excellent example of a natural experiment. We had a population who had vaccines with thimerosal and we had a population with vaccines without it (anyone vaccinated after 2001). If we control for the changes in inclusion criteria, we can see that autism rates remained unchanged, thereby falsifying the conditional hypothesis that thimerosal causes autism.

Your Turn:

Here are some examples you can try for yourself:

Hypothesis/Claim 1: Sour sop cures cancer!!11!!11!!
Conditional Hypothesis: ?
Construct the Experiment: Suggest how you'd test the hypothesis by using the concept of control group, treatment group, and natural rate of remission (i.e., what percentage of the population will survive the cancer without any treatment).

A note on testing treatments. Generally, treatments aren't tested against a natural remission rate because it would be unethical not to give a patient a treatment. Instead, new treatments are tested compared to the current standard treatment. If the new treatment leads to a higher rate of remission than the current standard, then we adopt it. But even if it has some effect relative to the natural non-treatment rate we don't care unless it's better than the current standard. Keeping the above in mind, construct your experiment.

Hypothesis/Claim 2: Eating/drinking supplement X prevents me from getting sick.
Conditional Hypothesis: ?
Construct the Experiment: Suggest how you'd test the hypothesis using the concepts we've discussed.

Hypothesis/Claim 3: Weight-loss supplement "Super Quantum Lipo-Fat Burner Extra Mega Shred" causes weight loss.
Conditional Hypothesis: ?
Construct the Experiment: Suggest how you'd test the hypothesis.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License